Webserver Configuration Recovery

Under normal operating conditions, the Skylight orchestrator webserver configuration file is preserved and maintained in its full integrity. Under certain scenarios such as hard reboots or disk full situations, it is possible that this configuration file becomes corrupted. This renders the application unusable.

To protect from this situation, a configuration copy mechanism has been introduced that takes a daily backup of the web configuration. These configuration files are stored in /opt/accedian/skylight/backups/glassfish_config directory. This backup happens daily with no additional user intervention required.

Along with this daily backup process, a restoreGlassfishConfiguration.sh script is provided and is located under the /opt/accedian/skylight/scripts directory. It should be used if ever you encounter a corruption of the webserver configuration files.

Restoring on the Non-HA System

To manually restore Glassfish configuration

1. Run the restore Glassfish script below:
   /opt/accedian/skylight/scripts/restoreGlassfishConfiguration.sh <weekday name (e.g., Sunday)>
   For example: /opt/accedian/skylight/scripts/restoreGlassfishConfiguration.sh Monday
   

   ---

   Note: In the example above, the most recent Glassfish configuration file from the Monday folder will be applied to the Glassfish directory.

   ---

2. Restart app_server by entering:
   service start app_server

Restoring on the HA System

In the HA feature, checking whether or not the system was full root partition is not supported. This guideline is to recover the HA system when the full root partition issue occurs on the site that encounters this issue:

Example: site-a is the active site and site-b is the passive site.

Depending on whether the auto-failover option is enabled or disabled, the user needs to proceed accordingly:

Case #1: The auto-failover is enabled (default mode)

An auto-failover will occur, the site-b will become the active site.

1. Resolve the full root partition on the site that has this issue (e.g: site-a).

2. If the Skylight orchestrator is prior to version 21.01 or other cases that the script has not yet been detected, run the restore Glassfish script on the site of step 1 below:
   /opt/accedian/skylight/scripts/restoreGlassfishConfiguration.sh <weekday name (e.g., Sunday)>
   For example: /opt/accedian/skylight/scripts/restoreGlassfishConfiguration.sh Monday

IMPORTANT: If the Skylight orchestrator is version 21.01 or higher, this step must be skipped.

3. Via app_mon of the site in step 1, enter the command:
   reboot
   If prompted “Warning: The geo redundant setup is running. Proceed with reboot? [yes/no]:”, enter:
   yes

4. Depending on whether the preferred site option is site-a or none, the user needs to proceed accordingly:

• Preferred site-a: the site-a will automatically become the active site after a moment. It may take up to 15 minutes.
• Preferred none: the site-b is still the active site. If the user wants to switch back to the older active site, the user needs to follow the optional steps below:
  a. Wait for the HA_MALFUNCTION alarm to be cleared up (if yes). It may take up to 15 minutes.
  b. Via app_mon, enter the command:
  redundancy control switch

Case #2: The auto-failover is disabled

An auto-failover does not occur, the system will be hanged.

1. Resolve the full root partition on the site that has this issue (e.g: site-a).

2. If the Skylight orchestrator is prior to version 21.01 or other cases that the script has not yet been detected, run the restore Glassfish script on the site of step 1 below:
   /opt/accedian/skylight/scripts/restoreGlassfishConfiguration.sh <weekday name (e.g., Sunday)>
   For example: /opt/accedian/skylight/scripts/restoreGlassfishConfiguration.sh Monday
   

   ---

   Note: In the example above, the most recent Glassfish configuration file from the Monday folder will be applied to the Glassfish directory.

   ---

   
   IMPORTANT: If the Skylight orchestrator is version 21.01 or higher, this step needs to be skipped.

3. Wait for the HA_MALFUNCTION alarm to be cleared up (if yes). It may take up to 15 minutes.

4. Via app_mon, enter the command:
   redundancy control suspend

5. Via app_mon, enter the command:
   redundancy control resume

6. Wait for the app_server to be restarted on this site. It may take up to 10 minutes.