Merging Performance Sessions

As of Legacy orchestrator 24.09, orchestrator supports merging sessions from separate instances (referred to as source Legacy orchestrator) into a single instance (referred to as the target Legacy orchestrator).

This article explains how to merge all performance sessions from a source to a target.

About Merging

The components that support the sessions will be merged, including:

• The Assurance Sensor Control
• The Reflectors
• The Credential
• Control template
• Stream template

The Export template of the session in the source Legacy orchestrator is not merged. Instead, all sessions following the merge process will utilize the Export template from the target Legacy orchestrator as the default. This also applies to other device types.

What Is Not Supported

This feature does not support merging sessions under the following conditions:

• Deployment Profile: Not supported if the source's Deployment profile is larger than the target's profile
• Session Profile: Not supported if the source's Session profile is larger than the target's profile
• User Profile: Not supported if the source's User profile is larger than the target's profile
• Session duplication: Not supported if there are duplicate sessions on both systems
• Assurance Sensor Control duplication: Not supported if there are duplicate products on both systems
• Hostname of the Assurance Sensor Control duplication: Not supported if there are duplicate hostnames for Assurance Sensor Control on both systems
• Release Version: Releases prior to 24.09 are not supported

Preparation

Prepare minimum two Legacy orchestrators: one for the source Legacy orchestrator, one for the target Legacy orchestrator.

For the source Legacy orchestrator:

• Select the Legacy orchestrator that is used less frequently or is intended to be shut down or removed after merging.
• It must have an equal or smaller Deployment, Session, and User profile than the target Legacy orchestrator (see Deployment Show Configuration).
• It must be equal to or smaller than the available instance capacity on the destination orchestrator. This includes session, reflector, Assurance Sensor Control and number of devices.
• The release version is the same release version as the destination orchestrator.

For the target Legacy orchestrator:

• Select the primary Legacy orchestrator or one that is used more frequently.
• The target Legacy orchestrator must have sufficient capacity for instances from both the source and target Legacy orchestrators, which includes:
  • The number of Sessions (maximum supported depend on the session profile: 52K or 125K).
  • The number of Reflectors (maximum supported depend on the session profile: 52K or 125K).
  • The number of Assurance Sensor Controls (maximum supported is 100 Assurance Sensor Controls).
  • The number of devices (maximum supported depend on the Legacy orchestrator deployment profiles: 5K, 15K, 30K, 60K).

Requirements

• The fresh system must start services at least once so that Legacy orchestrator can generate the default configuration. Refer to the Services Start section Docker CLI Reference.
• If the source Legacy orchestrator is running an Ubuntu operating system and must be merged with a target Legacy orchestrator running a Red Hat operating system, you must remove the write permission from the group for the skylight user on the source Legacy orchestrator. In other cases, this step is not necessary.

1. Log into the source Legacy orchestrator on port 22 as the skylight user.

2. Remove the write permission from the group for the skylight user:

chmod g-w /home/skylight

• Between the source and the target Legacy orchestrator, ensure there is no duplication of Session names, Assurance Sensor Control and the hostname of the Assurance Sensor Control.

Merge Procedures

Check for Duplicate Session Names

To ensure Provider Connectivity Assurance will continue to graph all sessions after merging, the merge process does not allow duplicate session names.

This implies that you must verify if the session name on the source Legacy orchestrator matches the session name on the target Legacy orchestrator. If they are the same, you must change the session name on either the source or the target Legacy orchestrator. To do this, execute the supported orchestrator script by following the steps below:

1. Log into the source Legacy orchestrator and select Session tab and export the Session CSV file by clicking the Export button.
2. Log into the target Legacy orchestrator and select Session tab and export the Session CSV file by clicking the Export button.
3. Transfer these CSV files from your computer to the target Legacy orchestrator:

scp <file_name_of_source_session>.csv skylight@x.x.x.x:/<directory>

Where:

• x.x.x.x: The IP address of the target Legacy orchestrator

• < directory >: The directory files to be archived; archiving the file in the /tmp folder is recommended.

4. SSH to the target Legacy orchestrator on port 22 as the skylight user.

5. Execute the validate_duplicated_session_name.sh script by entering the following command:

~/so/bin/validate_duplicated_session_name.sh <directory>/<file_name_of_source_session>.csv <directory>/<file_name_of_target_session>.csv

Backup the Data Store

To avoid unexpected interruptions, such as outages or manual cancellations during the merge process, you must back up the database on both the source and target Legacy orchestrators.

On the source Legacy orchestrator

1. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

2. Enter the following command:

database backup

This command allows you to backup the MySQL database. The backup is generated as a tar.gz file in the 'backups' folder of the so-mysql volume.

3. Exit the SOCLI by entering:

exit

On the target Legacy orchestrator

1. Log in as the skylight user to SOCLI by SSH to the target Legacy orchestrator on port 2200.

2. Enter the following command:

database backup

This command allows you to backup the MySQL database. The backup is generated as a tar.gz file in the 'backups' folder of the so-mysql volume.

3. Exit the SOCLI by entering:

exit

• The backup must be taken from the active site.

• The backup file is stored at /home/skylight/so/mysql-ha/backups folder.

On the Non-Hot Standby system:

• The backup is generated as a tar.gz file in the ${volume-location so-mysql}/backups folder.

By default: /home/skylight/so/mysql/backups

Stop the Legacy orchestrator Services

On the source Legacy orchestrator

Stopping Legacy orchestrator Services for Hot Standby Configuration

Active Site

1. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

2. Stop the Hot Standby process by entering:

redundancy control stop

3. Exit the SOCLI by entering:

exit

Stopping Legacy orchestrator Services for Non-Hot Standby Configuration

1. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

2. Stop service by entering:

service stop

3. Exit the SOCLI by entering:

exit

On the target Legacy orchestrator

Stopping Hot Standby Process and Legacy orchestrator Services for Hot Standby Configuration

Active Site

1. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

2. Stop the Hot Standby process by entering:

redundancy control stop

3. Exit the SOCLI by entering:

exit

Stopping Legacy orchestrator Services for Non-Hot Standby Configuration

1. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

2. Stop service by entering:

service stop

3. Exit the SOCLI by entering:

exit

Execute Merge Process on Target Legacy orchestrator

To execute the merge process

1. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

2. Enter the merge process with the following command:

database merge-session host x.x.x.x port <port>

Where:

• x.x.x.x: The IP address of the source Legacy orchestrator

• < port >: The customized source MySQL port. This port is used to connect the database from the target to the target Legacy orchestrator. Ensure that the port entered is unused; the port is optional and not required to be input.

Note that if you do not input the port, the default source MySQL port (3306) will be used. You can enter the following command without specifying the port:

database merge-session host x.x.x.x

After entering the command for the merge process, you will be re-asked to start the process for merging the sessions.

Start the process of merging session data from the source host 10.220.0.41 into the database
Proceed ? (y/N)

Enter y if you are sure. To cancel the process, enter any other character.

3. Enter the skylight user password for the source Legacy orchestrator. This is necessary to create the connection between the source and the target orchestrators in order to merge the database.

Start the process of merging session data from the source host 10.220.0.41 into the database
Proceed ? (y/N)
y
Setup authentication to the source host with ip 10.220.0.41
skylight@10.220.0.41's password:

Once the password is entered, the database merge process will begin. Wait for the process to complete before proceeding to the next steps.

The output of the process upon completion will be as follows:

Check the deployment profile configuration on the source and destination
Check Skylight orchestrator services status on the source host
Run so-mysql-merge container on the source host
Begin session merging process
  Run so-mysql container on the destination host
  Merge source session data into the database
  Stop so-mysql container on the destination host
Remove existing so-mysql-merge container on the source host
Successfully merge the session

Start the Legacy orchestrator Services

After the merge process is complete, you must start the services only on the target Legacy orchestrator.

Starting the application for Hot Standby configuration

Active site

1. Log in as the skylight user to the target Legacy orchestrator designated as the preferred active site on port 2200.

2. Start the Hot Standby process by entering:

redundancy control start

3. Verify redundancy status by entering:

redundancy show status 

Starting the Legacy orchestrator services for Non-Hot Standby configuration

1. Log in as the skylight user to the target Legacy orchestrator on port 2200.

2. Start services by entering:

services start

Tracking Process

When merging sessions from the source Legacy orchestrator to the target Legacy orchestrator, logs will be created to track the process and will be archived in the /home/skylight directory.

The following is the log name format:

session-merge_< yyyy-mm-dd >< hh-mm-ss >.log

Where:

• < yyyy-mm-dd > represents the date.

• < hh-mm-ss > represents the time.

Example: The tracking process log: session-merge_2024-07-25T13-44-28.log

The logs capture the process of merging sessions, including the initial steps of connecting SSH keys, checking available resources on the target Legacy orchestrator, handling duplicate instances, and updating the database.

How Legacy orchestrator Handles Duplicate Instances

If Reflector, Stream template, Control template, or Credential instances are duplicated between the source Legacy orchestrator and the target Legacy orchestrator, the merge process will handle them as follows:

• The instances available on the target Legacy orchestrator will be retained.
• Instances from the source Legacy orchestrator will have an incremental suffix added until there are no duplications after merging, in the format:

<name>_<increment suffix>

Example with Stream template instance on the source Legacy orchestrator before merging:

twamp

Example with Stream template instances on the target Legacy orchestrator before merging:

twamp

twamp_1

twamp_2

Example Stream template instances after merging to the target Legacy orchestrator:

twamp

twamp_1

twamp_2

twamp_3

Note that twamp_3 refers to the added increment suffix for the instance that merged from the source Legacy orchestrator; instances on the target Legacy orchestrator remain unchanged.

Rollback Procedure

The Rollback Procedure is designed to ensure no changes occur in the database when unexpected interruptions, such as outages or manual cancellations, occur throughout the merge process.

During the merge process, a new container called so-mysql-merge will be created on the source Legacy orchestrator to merge the database to the target Legacy orchestrator. If you manually cancel the process, you must remove the so-mysql-merge container on the source Legacy orchestrator and restore the database on the target Legacy orchestrator; this also applies to outage scenarios.

On the source Legacy orchestrator

Removing the merge container and starting the application for Hot Standby configuration

Active Site

1. SSH to the source orchestrator on port 22 as either the skylight or visionems user.

2. Remove the so-mysql-merge container by entering:

docker ps -a --filter "name=so-mysql-merge" --format "{{.Names}}" | xargs -r docker rm -f

This command locates and removes containers with the name so-mysql-merge .

3. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

4. Start the Hot Standby process by entering:

redundancy control start

5. Verify redundancy status by entering:

redundancy show status 

6. Exit the SOCLI by entering:

exit

Removing the merge container and starting Legacy orchestrator Services for Non-Hot Standby Configuration

1. SSH to the source orchestrator on port 22 as either the skylight or visionems user.

2. Remove the so-mysql-merge container by entering:

docker ps -a --filter "name=so-mysql-merge" --format "{{.Names}}" | xargs -r docker rm -f

This command locates and removes containers with the name so-mysql-merge .

3. Log in as the skylight user to SOCLI by SSH to the source Legacy orchestrator on port 2200.

4. Start services by entering:

services start

5. Exit the SOCLI by entering:

exit

On the target Legacy orchestrator

Restoring the database and starting the application for Hot Standby configuration

Active Site

1. Log in to the target Legacy orchestrator, designated as the preferred active site, on port 2200 as the skylight user.

2. Restore the database that was backed up in the previous step (refer to Backup the Data Store) by entering:

 database restore <filename_of_the_backup_archived>

3. Ensure the skylight user has read access to the archive that was copied in the /tmp folder.

4. Start the Hot Standby process by entering:

redundancy control start

5. Verify redundancy status by entering:

redundancy show status 

6. Exit the SOCLI by entering:

exit

Restoring the database and starting Legacy orchestrator services for Non-Hot Standby configuration

1. SSH to the source orchestrator on port 22 as either the skylight or visionems user.

2. Restore the database that was backed up in the previous step (refer to Backup the Data Store) by entering:

 database restore <filename_of_the_backup_archived>

3. Ensure the skylight user has read access to the archive that was copied in the /tmp folder.

4. Start services by entering:

services start