Upgrading Skylight orchestrator Docker 23.04.1

This Skylight orchestrator Docker Upgrade article includes procedures that will help the personnel of mobile network operators, Cable Multi-Service Operators (MSOs), business service providers, and cloud and data center services upgrade and maintain their performance assurance solutions.

Specifically, this article explains how to upgrade the Skylight orchestrator Docker to version 23.04.1 from any of these previously-installed versions:

• Skylight orchestrator Docker running 22.10 5K profile
• Skylight orchestrator Docker running 22.10 15K profile
• Skylight orchestrator Docker running 22.10 30K profile
• Skylight orchestrator Docker running 22.10 60K profile
• Skylight orchestrator Docker running 23.04 5K profile
• Skylight orchestrator Docker running 23.04 15K profile
• Skylight orchestrator Docker running 23.04 30K profile
• Skylight orchestrator Docker running 23.04 60K profile

Prior installation data and configuration information is retained during the upgrade process.

Pre-Upgrade Requirements and Procedures

This section describes the requirements and procedures that must be followed before performing an upgrade.

The following sections cover the procedures for upgrading a Docker host running Skylight orchestrator 22.10 or Skylight orchestrator 23.04 software on the Docker host. During the upgrade, the system will be unavailable for approximately 90 minutes.

Before performing the upgrade procedures, you must read through the Release Notes for Skylight orchestrator 23.04.1.

---

WARNING: If you are using Skylight Performance Analytics, the Roadrunner component must be upgraded to the latest release before upgrading to Skylight orchestrator Docker 23.04.1. Please contact the Skylight Analytics support team to verify that your Roadrunner version is compatible with Skylight orchestrator Docker 23.04.1.

---

Requirements

You must ensure you have the following items before starting the upgrade procedure. If you are missing anything, contact Accedian Technical Support.

1. SSH client (such as PuTTY)
2. SCP client (such as WinSCP)
3. The Docker host must have over 25 GiB of free space for Docker images.
4. The Docker host must have 15 GiB of free space for the temporary folder used during the installation.
5. One user must have sudo privileges or root access on the Docker host.
6. The multipath tool (multipath-tools package) must have been installed on the host.
7. To upgrade to version 23.04.1, you must have access to the following package file:

   docker-dist-23.04.1-GA-1.sh

8. Check and ensure the current deployment profile matches with resource requirements (vCPU, RAM, disk). Please refer to the Docker Deployment Profile Configuration and Docker Disk Configuration articles for additional information

Pre-Upgrade Procedures

This section explains the procedures that must be performed before upgrading to version 23.04.1.

The pre-upgrade procedures are:

1. Backing up the Data Store
2. Transferring file to the Docker host that deployed Skylight orchestrator Docker
3. If operating in a Hot Standby configuration, stop the redundancy feature

Backing up the Data Store

A database backup of the system is needed.

Requirement:

Skylight orchestrator services must be started.

To back up the mysql database

1. Open an SSH terminal session to the Skylight orchestrator CLI on port 2200.
2. Log in as the skylight user.
3. Create a backup of the data store by entering:

database backup

---

Notes:

On the Non-Hot Standby configuration system, the backup is generated as a tar.gz file in the folder ‘backups’ of the ‘so-mysql’ volume.

On the Hot Standby configuration system:

• The backup must be taken from the active site.
• The backup is generated as a tar.gz file in the folder '/home/skylight/so/mysql-ha/backups'

---

Transferring File to the Docker host which deployed the Skylight orchestrator Docker

This procedure explains how to install the required upgrade script to the appropriate location.

The following upgrade file must be available:

• docker-dist-23.04.1-GA-1.sh

To transfer file

The procedure must be executed by the skylight user.

1. From your computer, use an SCP client (such as WinSCP) and the Skylight user to transfer the required file to the /home/skylight folder.

---

CAUTION: The location of the required file must be /home/skylight.

---

If operating in a Hot Standby configuration, stop the redundancy feature

1. Open an SSH terminal session to the Skylight orchestrator CLI on port 2200.
2. Log in as the skylight user.
3. Stop the redundancy feature process by entering:

redundancy control stop

---

Note: Redundancy feature can be stopped on both sites by running above command on a site only.

---

Upgrading Skylight orchestrator Docker

This procedure contains instructions on how to upgrade to Skylight orchestrator Docker 23.04.1. Upgrades are supported from the following releases:

• Skylight orchestrator Docker running 22.10 5K profile
• Skylight orchestrator Docker running 22.10 15K profile
• Skylight orchestrator Docker running 22.10 30K profile
• Skylight orchestrator Docker running 22.10 60K profile
• Skylight orchestrator Docker running 23.04 5K profile
• Skylight orchestrator Docker running 23.04 15K profile
• Skylight orchestrator Docker running 23.04 30K profile
• Skylight orchestrator Docker running 23.04 60K profile

---

Note: If you are running a redundant configuration, this procedure must be executed on each virtual appliance of the redundant pair.

---

Removing the Older Version Before Upgrading

---

Notes:

Skip this step if the current Skylight orchestrator Docker is NOT upgraded from any previous version.

The older version must be removed before upgrading to free up disk space on the Docker host.

---

To remove the older version before upgrading

1. Open an SSH terminal session to port 22 of the appliance, log in to the appliance with one user having sudo privileges or root access on the host.

2. Execute the script that removes the older version:

sudo sh /home/skylight/docker-dist-23.04.1-GA-1.sh -c

If prompted, provide the password of the user having sudo privileges or root access on the host.

---

Note: Continue to the “Upgrading to the New Software Version” section if the “No previous version of Skylight orchestrator to commit all changes. Exiting.” message is returned.

---

3. When prompted enter:

y

=========================================================================================
Committing all changes of Skylight orchestrator 23.04.1-GA and cleaning up Skylight orchestrator 22.10-GA
Unable to rollback to Skylight orchestrator 22.10-GA after this action

=========================================================================================
Proceed ? (y/N)
y
Remove Skylight orchestrator images version 22.10-GA

[ . . . ]

Remove origin folder and data of Skylight orchestrator version 22.10-GA
Successfully cleaned up Skylight orchestrator system version 22.10-GA

Upgrading to the New Software Version

This procedure will install the upgraded software.

To upgrade to the new software

1. Open an SSH terminal session to port 22 of the appliance, log in to the appliance with one user having sudo privileges or root access on the host.
2. Execute the script that does the partition upgrade:

sudo sh /home/skylight/docker-dist-23.04.1-GA-1.sh

If prompted, provide the password of the user having sudo privileges or root access on the host.

3. When prompted enter:

y

Upgrade Skylight orchestrator version 22.10-GA to 23.04.1-GA
Proceed ? (y/N)
y
Validate mysql disk space for upgrade
Mysql disk space already to backup database
Stop Skylight orchestrator services version 22.10-GA
Backup database
Backup java for rollback

Install Skylight orchestrator version 23.04.1-GA

=========================================================================================

Installing Skylight orchestrator 23.04.1-GA (build 1-3934a3d)

=========================================================================================

4. When prompted enter:

y

Proceed ? (y/N)
y

5. Type the temporary installation folder to extract the Docker images or type enter to accept default: /tmp folder

Temporary installation folder? [enter to accept default: /tmp]
/tmp

Checking requirements ...

Requirements are ok.

Extracting installation package to /tmp/so_install ...

Please wait ...

Installation package extracted to /tmp/so_install .

Importing Skylight orchestrator distribution docker images from /tmp/so_install/docker_images.tar ...
b40ed86654e5: Loading layer 75.15MB/75.15MB
8dbe5f6a3a6e: Loading layer 115.9MB/115.9MB
[...]
Loaded image: so-drbd9-focal:23.04.1-GA

Setup user:group skylight:skylight ...
Found group 'skylight'
skylight:x:2000:

Found user 'skylight'
skylight:x:2000:2000:Accedian Skylight orchestrator:/home/skylight:/bin/bash

user:group skylight:skylight are created.

Resetting the main user in Skylight orchestrator containers to host UID and GID

Resetting 'so-mysql' container user to host UID and GID
Sending build context to Docker daemon 8.704kB
[...]
Successfully built c4c94bc55804
Successfully tagged so-ha:23.04.1-GA
Untagged: so-ha:23.04.1-GA-distro

Skylight orchestrator images installed
REPOSITORY TAG IMAGE ID CREATED SIZE
so-ha 23.04.1-GA c4c94bc55804 1 second ago 880MB
so-export 23.04.1-GA bd25f964c42e 3 seconds ago 252MB
so-core 23.04.1-GA b8eeefc85955 6 seconds ago 6.19GB
so-med 23.04.1-GA a24f3cfecd1a 56 seconds ago 4.48GB
so-nbi 23.04.1-GA e9640d899cd1 About a minute ago 4.48GB
so-mysql 23.04.1-GA a69b77268f50 2 minutes ago 8.31GB

Setup Skylight orchestrator configuration ...

Skylight orchestrator configuration is setup.

Installing Skylight orchestrator files and folders ...

Skylight orchestrator files and folders are created.

Setup Skylight orchestrator docker volumes...
Setup so-config volume...
so-config
so-config

Skylight orchestrator docker volumes are setup.

Setup the network for Skylight orchestrator docker containers...
Found network 'so-network'

[
  {
   "Name": "so-network",
   "Id": "3fe5c18c48cfd9aa3afdb445f861cd0f76e84720c36dfa53ad3eb1840c06a304",
   "Created": "2022-07-19T08:23:13.970164575Z",
   "Scope": "local",
   "Driver": "bridge",
   "EnableIPv6": true,
   "IPAM": {
      "Driver": "default",
       "Options": {},
       "Config": [
          {
            "Subnet": "172.18.0.0/24",
            "Gateway": "172.18.0.1"
          },
          {
           "Subnet": "fd00:1::/80",
           "Gateway": "fd00:1::1"
          }
       ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
       "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {},
    "Options": {},
    "Labels": {}
   }
]

Starting HA docker container by default...

Creating so-ha-23.04.1-GA ... done

HA docker container started.

Removing the temporary installation folder /tmp/so_install ...

=================================================================================================================

The installation of Skylight orchestrator 23.04.1-GA (build 1-3934a3d) is complete.

To manage Skylight orchestrator, run:

/usr/bin/socli.sh

=================================================================================================================

Import data upgrade version 23.04.1-GA
Upgrade database schema for Skylight orchestrator version 23.04.1-GA
Creating so-mysql-23.04.1-GA ... done
Successfully upgraded the database schema
Stopping so-mysql-23.04.1-GA ... done
Removing so-mysql-23.04.1-GA ... done
Going to remove so-mysql-23.04.1-GA
Import 'skylight' certificate into the keystore for Skylight orchestrator version 23.04.1-GA

=========================================================================================

Skylight orchestrator version 23.04.1-GA upgrade process was successful

=========================================================================================

Post-Installation Procedures

This section explains the procedures applicable after your upgrade.

The post-installation procedures are:

1. Restarting the Skylight orchestrator services
2. Post-Upgrade Checks

Restarting the Skylight orchestrator services

This procedure will start the upgraded software.

Restarting the Skylight orchestrator services for Hot Standby configuration

1. Open an SSH terminal session to the Skylight orchestrator CLI designated as the preferred active site on port 2200.

2. Log in as the skylight user.

3. Configure the preferred site to last active site, enter:

redundancy config preferred {site-a | site-b}

4. Start the Hot Standby process by entering:

redundancy control start

5. Verify redundancy status by entering:

redundancy show status

Restarting the Skylight orchestrator services for a Non-Hot Standby configuration

1. Open an SSH terminal session to the Skylight orchestrator CLI on port 2200.

2. Log in as the skylight user.

3. Start services by entering:

services start

4. When prompted enter:

y

---

Note: It may take up to 10 minutes for the Skylight orchestrator services to be healthy.

---

Verifying the Startup Status

To verify the startup status

1. Enter:

services status

A list of services (that are currently running) is displayed.
Ensure that the state of all services is shown as healthy.

2. Close the terminal session to port 2200 by entering:

exit

Post-Upgrade Checks

The final step of the upgrade is to check the network inventory and ensure that Skylight orchestrator Docker is operational. The following steps constitute a basic verification of the state of Skylight orchestrator.

To verify network inventory and check Skylight orchestrator functionality

1. Log in to the Skylight orchestrator web application in a supported web browser.
2. Click the Accedian icon to display the About window.
   The Build Version should indicate version 23.04.1.
3. Access the Device ▶ Managed Elements tab and verify that the total count of network elements at the bottom-left corner is accurate.
4. Click the icon at the bottom-right corner and verify that the current alarm count is accurate.
5. Access the Commission ▶ Jobs tab and verify that any pre-existing configuration jobs are still present.
6. Access the Collection ▶ Reporting Status tab and verify the following:
   • The list of performance elements with a pm collect agent.
   • The CSV time and Transfer time have both recently been updated for all elements with streaming enabled.
7. Access Admin ▶ Licensing to verify that the current License server is the as same as before upgrading:
   • Verify that the License server is Integrated Server if you imported the "Skylight-orchestrator - Internal License server" key on the Skylight orchestrator before upgrading​.
   • Verify that the connection with SDLS has been established if you configured the Skylight orchestrator connection to the license server before upgrading.

   ---

   Note: Importing "Skylight-orchestrator - Internal License server" key after upgrading Skylight orchestrator is no longer required.

   You can promptly change the license server from ‘Integrated Server' to 'SDLS’ and vice versa without restarting app_server.

   ---

8. Check your session status by accessing Sessions tab.

---

Note: All functionality is added by default; importing Feature Activation Key in order to enable functionalities after upgrading Skylight orchestrator is no longer required.

---

Rollback Procedure

This section explains how to roll back to the original operating system partition and restore the system to its initial state.

If you must roll back a cluster deployment, you must repeat the procedure on all appliances on which the upgrade was performed (successfully or unsuccessfully).

Rollback for Skylight orchestrator Procedure

This procedure explains how to roll Skylight orchestrator back to the previous release.

You would only need to perform the rollback procedure if problems occur during the appliance upgrade procedure, such as:

• An error message occurs during the upgrade process that prevents you from proceeding.
• The upgrade process terminates unexpectedly.
• The application does not function as expected after the upgrade.

Procedure Requirements

The following requirement applies to the procedure:

1. This procedure must be executed with one user having sudo privileges or root access on the host.

If operating in a Hot Standby configuration, stop the redundancy feature

1. Open an SSH terminal session to the Skylight orchestrator CLI on port 2200.

2. Log in as the skylight user.

3. Stop the Hot Standby process by entering:

redundancy control stop

---

Note: Redundancy feature can be stopped on both sites by running above command on a site only.

---

Rolling back the upgrade

Repeat this procedure on all appliances that were upgraded.

To roll back the upgrade

1. Using an SSH terminal session to port 22 of the appliance, log in to the appliance with one user having sudo privileges or root access on the host.
2. Execute the script that does the partition upgrade:

sudo sh /home/skylight/docker-dist-23.04.1-GA-1.sh -r

If prompted, provide the password of the user having sudo privileges or root access on the host.

3. When prompted enter:

y

Proceed ? (y/N)
y
Stop Skylight orchestrator version 23.04.1-GA
Removing image so-mysql:23.04.1-GA ...
Untagged: so-mysql:23.04.1-GA
Deleted: sha256:106f366acdfcc01ab086a4c16082deee7e4b38b1fd797cf8d6e44c17c1b8909a
Deleted: sha256:01e3bb19ce4fce80d8b2ceecf74d022e0edcb3832fe827351f05046d09f3d9dd
Deleted: sha256:b04590d7f551c4962e26031415b13b1e5f823926163e41c932be70eb5482c164
[...]

=========================================================================================

Skylight orchestrator rollback process was successful .

=========================================================================================

Restarting the Skylight orchestrator services

This procedure will start the upgraded software.

Restarting the Skylight orchestrator services for Hot Standby configuration

1. Open an SSH terminal session to the Skylight orchestrator CLI designated as the preferred active site on port 2200.

2. Log in as the skylight user.

3. Configure the preferred site to last active site, enter:

redundancy config preferred {site-a | site-b}

4. Start the Hot Standby process by entering:

redundancy control start

5. Verify redundancy status by entering:

redundancy show status

Restarting the Skylight orchestrator services for a Non-Hot Standby configuration

1. Open an SSH terminal session to the Skylight orchestrator CLI on port 2200.

2. Log in as the skylight user.

3. Start services by entering:

services start

4. When prompted enter:

y 

---

Note: It may take up to 10 minutes for the Skylight orchestrator services to be healthy.

---