Roadrunner - High Availability

Overview

High Availability mode allows multiple Roadrunners to be assigned to the same connector zone in an active/standby configuration, where only the active Roadrunner collects and forwards data while the standby Roadrunners remain in an idle state until triggered to assume activity.

Analytics triggers the switchover to the standby Roadrunner when loss of connectivity to the active Roadrunner is detected for a zone.

Configuration

• To configure a Roadrunner zone for High Availability, create an OpenMetrics Scraper type connector configuration via API call in the normal manner.

Ensure that the "highAvailability" parameter in the connector configuration is set to true.

Example Connector Configuration payload:

POST: /api/v2/connector-configs
{
    "data": {
      "attributes": {
        "datatype": "connectorConfig",
        "highAvailability": true,
        "loglvl": "",
        "name": "OpenMetricsScraper-HA",
        "openMetricsScraperConfig": {
              "prometheusConfig": "global:\n  scrape_interval: 5s \nscrape_configs:\n- job_name: SQALocalNodeExporter\n  static_configs:\n  - targets: [\"driver4-leader-blcr:9100\",\"driver4-worker-5mh9:9100\",\"driver4-worker-z1xl:9100\"]"
      },
      "pollingFrequency": 500,
      "tenantId": "a174b09f-6e6a-467c-bbe0-473c6b96faa5",
      "type": "OpenMetricsScraper",
      "zone": "OpenMetricsScraper-HA"
    },
    "type": "connectorConfigs"
  }
}

Deployment

Docker

To deploy Roadrunner instances into a High Availability zone for Docker:

1. Download a Roadrunner tarball for the Connector Zone that was created with High Availability enabled (see Configuration).
2. Install to the location of your choice in the normal manner.
3. Download a second tarball from the same Connector Zone.

Important: You must download a second tarball from Analytics, rather than install a second instance using the tarball from the first instance, because downloading the connector generates a unique connector id for that instance. Installing two copies of Roadrunner from the same tarball results in two instances with the same connector id, creating a conflict.

4. Using the second tarball download, install your second Roadrunner instance to the location of your choice, again, in the normal manner.
5. Start your Roadrunner instances. The first to establish its meta connection to Analytics will become active and begin scraping and shipping data to Analytics. The other Roadrunners will start in Standby mode and remain idle until Analytics detects a loss of connectivity.

Kubernetes

Support for the deployment of Roadrunner instances into a High Availability zone for Kubernetes is planned for a future release.

Inspection

It is possible to query a Connector Zone via the API to determine, among other things, the following:

• What Roadrunner instances are members of the Zone.
• The connectivity status of each Roadrunner instance.
• Which of the Roadrunner instances is operating as the active member of the zone.

In the example below, our zone contains two members and the Roadrunner instance with connector id 9cfc4888-9023-4291-b747-21d14c53feb2 is the active member of the zone.

GET: /api/v3/connectors?zone={{connector_zone}}
{
    "data": [
        {
            "active": false,
            "id": "1653dfbc-0425-48dd-ba40-0bd61b9552ac",
            "status": "CONNECTED",
            "version": "0.519.0",
            "zone": "OpenMetricsScraper-HA"
        },
        {
            "active": true,
            "id": "9cfc4888-9023-4291-b747-21d14c53feb2",
            "status": "CONNECTED",
            "version": "0.519.0",
            "zone": "OpenMetricsScraper-HA"
        }
    ]
}  

The active Roadrunner can be switched manually be using the following API endpoint. The connectorID specified should be the ID of a standby Roadrunner. Using this endpoint will force the specified Roadrunner to become Active and the Active Roadrunner to become standby.

POST: /api/v3/connectors/:connectorID/activate

FAQ

• Does the UI support configuration of Roadrunner zones for High Availability?
  At this time, High Availability mode for Roadrunner is only supported via API calls.
• Is High Availability mode supported by all Roadrunner variants?
  At this time, only the OpenMetrics Scraper variant of Roadrunner can be deployed into a High Availability zone.
• If I have multiple Roadrunners in a High Availability zone, how is it determined which Roadrunner will be the active member of the pair/cluster?
  The first Roadrunner in the zone to establish its meta connection with Analytics will be selected to be the active member of the pair/cluster.
  All Roadrunners within the zone to subsequently connect to Analytics will start in standby mode.
• What can cause an activity switch to occur for my High Availability zone?
  A loss of connectivity on the meta connection between the active Roadrunner and Analytics will result in another member of the High Availability pair/cluster being elected to be the active Roadrunner (achieved via a config push to the Roadrunner to become active).
• Why should I not install the same tarball twice?
  Download a second tarball from Analytics; do not download the first tarball twice. Downloading the same tarball twice creates a conflict and does not provide the High Availabiliy Roadrunner configuration.