Sensor Collector - High Availability

Overview

High Availability mode allows multiple Sensor Collectors (formerly Roadrunners) to be assigned to the same connector zone in an active/standby configuration, where only the active Sensor Collector collects and forwards data while the standby Sensor Collector remain in an idle state until triggered to assume activity.

Cisco Provider Connectivity Assurance (formerly Skylight performance analytics) triggers the switchover to the standby Sensor Collector when loss of connectivity to the active Sensor Collector is detected for a zone.

Configuration

• To configure a Sensor Collector zone for High Availability, create an OpenMetrics Scraper type connector configuration via API call in the normal manner.

Ensure that the "highAvailability" parameter in the connector configuration is set to true.

Example Connector Configuration payload:

POST: /api/v2/connector-configs
{
    "data": {
      "attributes": {
        "datatype": "connectorConfig",
        "highAvailability": true,
        "loglvl": "",
        "name": "OpenMetricsScraper-HA",
        "openMetricsScraperConfig": {
              "prometheusConfig": "global:\n  scrape_interval: 5s \nscrape_configs:\n- job_name: SQALocalNodeExporter\n  static_configs:\n  - targets: [\"driver4-leader-blcr:9100\",\"driver4-worker-5mh9:9100\",\"driver4-worker-z1xl:9100\"]"
      },
      "pollingFrequency": 500,
      "tenantId": "a174b09f-6e6a-467c-bbe0-473c6b96faa5",
      "type": "OpenMetricsScraper",
      "zone": "OpenMetricsScraper-HA"
    },
    "type": "connectorConfigs"
  }
}

Deployment

Docker

To deploy Sensor Collector instances into a High Availability zone for Docker:

1. Download a Sensor Collector tarball for the Connector Zone that was created with High Availability enabled (see Configuration).
2. Install to the location of your choice in the normal manner.
3. Download a second tarball from the same Connector Zone.

Important: You must download a second tarball from Analytics, rather than install a second instance using the tarball from the first instance, because downloading the connector generates a unique connector id for that instance. Installing two copies of Sensor Collector from the same tarball results in two instances with the same connector id, creating a conflict.

4. Using the second tarball download, install your second Sensor Collector instance to the location of your choice, again, in the normal manner.
5. Start your Sensor Collector instances. The first to establish its meta connection to Cisco Provider Connectivity Assurance (formerly Skylight performance analytics) will become active and begin scraping and shipping data to Cisco Provider Connectivity Assurance. The other Sensor Collectors will start in Standby mode and remain idle until Cisco Provider Connectivity Assurance detects a loss of connectivity.

Kubernetes

Support for the deployment of Sensor Collector instances into a High Availability zone for Kubernetes is planned for a future release.

Inspection

It is possible to query a Connector Zone via the API to determine, among other things, the following:

• What Sensor Collector instances are members of the Zone.
• The connectivity status of each Sensor Collector instance.
• Which of the Sensor Collector instances is operating as the active member of the zone.

In the example below, our zone contains two members and the Sensor Collector instance with connector id 9cfc4888-9023-4291-b747-21d14c53feb2 is the active member of the zone.

GET: /api/v3/connectors?zone={{connector_zone}}
{
    "data": [
        {
            "active": false,
            "id": "1653dfbc-0425-48dd-ba40-0bd61b9552ac",
            "status": "CONNECTED",
            "version": "0.519.0",
            "zone": "OpenMetricsScraper-HA"
        },
        {
            "active": true,
            "id": "9cfc4888-9023-4291-b747-21d14c53feb2",
            "status": "CONNECTED",
            "version": "0.519.0",
            "zone": "OpenMetricsScraper-HA"
        }
    ]
}  

The active Sensor Collector can be switched manually be using the following API endpoint. The connectorID specified should be the ID of a standby Sensor Collector. Using this endpoint will force the specified Sensor Collector to become Active and the Active Sensor Collector to become standby.

POST: /api/v3/connectors/:connectorID/activate

FAQ

• Does the UI support configuration of Sensor Collector zones for High Availability?
  At this time, High Availability mode for Sensor Collector is only supported via API calls.
• Is High Availability mode supported by all Sensor Collector variants?
  At this time, only the OpenMetrics Scraper variant of Sensor Collector can be deployed into a High Availability zone.
• If I have multiple Sensor Collectors in a High Availability zone, how is it determined which Sensor Collector will be the active member of the pair/cluster?
  The first Sensor Collector in the zone to establish its meta connection with Cisco Provider Connectivity Assurance will be selected to be the active member of the pair/cluster.
  All Sensor Collectors within the zone to subsequently connect to Cisco Provider Connectivity Assurance will start in standby mode.
• What can cause an activity switch to occur for my High Availability zone?
  A loss of connectivity on the meta connection between the active Sensor Collector and Cisco Provider Connectivity Assurance will result in another member of the High Availability pair/cluster being elected to be the active Sensor Collector (achieved via a config push to the Sensor Collector to become active).
• Why should I not install the same tarball twice?
  Download a second tarball from Cisco Provider Connectivity Assurance; do not download the first tarball twice. Downloading the same tarball twice creates a conflict and does not provide the High Availabiliy Sensor Collector configuration.