Troubleshoot Hot Standby Redundancy Startup Failures

This article provides a checklist for self-diagnosing issues when hot standby redundancy fails to start.

Troubleshoot Checklist

1. Verify physical connectivity.

• Use the ping command to test connectivity between Site-A and Site-B on both the replication and monitoring links.

2. Verify the HA license.

• Ensure the redundancy license is correctly imported on both sites:

docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c "ls -l /etc/drbd-proxy.license"

3. Verify Secure Boot status.

• Secure Boot must be disabled:

mokutil --sb-state

4. Verify NTP synchronization.

• Ensure that both sites are synchronized with the same NTP server.

5. Verify hostname resolution.

• Ensure the hostname resolves correctly:

docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c 'hostname -i'

6. Verify replication partitions

• Ensure the replication partitions are identical on both sides:
  • Check log: dmesg |grep -iE "disk size"
  • Check disk size: sudo blockdev --getsz <replication parititon>

7. Verify ‘kernel-devel’ package.

• Ensure that the kernel-devel package is installed and matches the current kernel version:

ls -ld /usr/src/kernels/$(uname -r)

8. Verify DRBD driver.

• Check that DRBD driver can be loaded successfully:

SO_VERSION=$(docker ps --filter "name=so-ha" --format "{{.Names}}" | cut -f 3 -d "-")
docker exec -it -uroot $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c "docker run --name so_drbd_module_loader --rm --privileged -v /lib/modules:/lib/modules -v /usr/src:/usr/src:ro $(docker images "*drbd*${SO_VERSION}-*" --format "{{.ID}}")"

Example:

$ docker exec -it -uroot $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c "docker run --name so_drbd_module_loader --rm --privileged -v /lib/modules:/lib/modules -v /usr/src:/usr/src:ro $(docker images "*drbd*" --format "{{.ID}}")"
...
...
=======================================================================
 
DRBD version loaded:
version: 9.2.15 (api:2/proto:118-123)
GIT-hash: 988c6ec892b40a3ef1b8e968ef277308753d7200 build by @cd112d90f1ef, 2026-04-08 09:32:37
Transports (api:21): tcp (9.2.15) lb-tcp (9.2.15) rdma (9.2.15)

Collect Information for Cisco Support

If Hot Standby Redundancy Startup fails and you require Cisco support, perform the following procedures so we can diagnose the root cause.

Note: Execute all commands on both sites.

Command output on socli (SSH Port 2200):

redundancy show statistics
redundancy show status options --debug
redundancy show status
redundancy show configuration
services status
config show

Command output on OS (SSH Port 22 with user having sudo privileges):

ip addr
docker version
cat /etc/hosts
cat /etc/os-release
cat /etc/fstab
cat /etc/docker/daemon.json
mount
uname -a
lsblk
df -kh
journalctl -b --no-pager 
cat /etc/fstab
export HISTTIMEFORMAT="%d/%m/%y %H:%M "; history
docker logs $(docker ps --filter "name=so-ha" --format "{{.ID}}")
docker logs $(docker ps --filter "name=so-core" --format "{{.ID}}")
docker logs $(docker ps --filter "name=so-med" --format "{{.ID}}")
docker logs $(docker ps --filter "name=so-nbi" --format "{{.ID}}")
docker logs $(docker ps --filter "name=so-mysql" --format "{{.ID}}")
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "cat /etc/drbd-proxy.license"
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "cat /volume/config/ha/hamonitorconfig.properties"
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "sudo service drbdproxy status"
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "sudo service corosync status"
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "sudo service pacemaker status"
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "sudo crm configure show"

docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") /bin/bash -c "dmesg -T| grep -iE 'drbd|proxy|split|auth|uuid|size|standalone|connect|disconnect|state change' | tail -n 120"

#check secure boot
sudo mokutil --sb-state

#hostname
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c 'hostname'
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c 'hostname -i'
docker exec -it $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c 'cat /etc/hosts'

rpm -qa|grep "multipa\|kernel-devel\|openssh\|json\|perl"

ls -ld /usr/src/kernels/$(uname -r)

SO_VERSION=$(docker ps --filter "name=so-ha" --format "{{.Names}}" | cut -f 3 -d "-") 
docker exec -it -uroot $(docker ps --filter "name=so-ha" --format "{{.ID}}") bash -c "docker run --name so_drbd_module_loader --rm --privileged -v /lib/modules:/lib/modules -v /usr/src:/usr/src:ro $(docker images "*drbd*${SO_VERSION}-*" --format "{{.ID}}")" 

Log file access (SSH Port 22 with user having sudo privileges):

sudo tar -czvf /home/skylight/all_log.tar.gz /home/skylight/so/logs/ /home/skylight/so/bin/logs/ /home/skylight/so/config/ /var/log/