Telemetry Collector output to Kafka

This article documents the process of provisioning a Telemetry Collector to output its data stream to a receiver other than Provider Connectivity Assurance. In this article, we are specifically focusing on outputting to a Kafka bus, but the same overall process would apply to any alternative destination.

There are two broad steps which must be performed to achieve this:

Disable data transfer from your Telemetry Collector to the Sensor Collector for ingestion into PCA.
Configuration of your Telemetry Collector to output its data stream to your preferred destination

1. Disabling data transfer to Provider Connectivity Assurance

In normal operations, the Telemetry Collector is expected to forward its data stream into the Provider Connectivity Assurance ingestion pipeline via its data connection to the Sensor Collector.

For use cases such as this one, where the intent is for the Telemetry Collector to forward its output to an alternate location (such as a Kafka bus), we must disable this default behaviour.

The following sections discuss the process to achieve this for either a Docker or Kubernetes deployment. Note that although data is not streamed to Provider Connectivity Assurance, there is still connectivity to Provider Connectivity Assurance which allows the run time configuration to be passed to the Telemetry Collector.

Docker Deployment

This section documents how to disable the Provider Connectivity Assurance data path if you are deploying your Telemetry Collector into a Docker environment.

Assuming that you have followed the instructions for deploying Sensor Collector for Telemetry in docker environments, you will have a working directory for your Telemetry Collector deployment that looks like the following:

my-telemetry-collector
 ├── docker-compose.yml
 └── secrets/
    └──── secrets.yaml

Edit your docker-compose.yml to set the TELEMETRY_NO_PCA_DATAPATH variable to true.

Example:

version: "3.8"
services:
  telemetry-agent:
    container_name: "telemetry-agent"
    image: "gcr.io/sky-agents/agent-telemetry-arm64:r24.09"
    hostname: "telemetry-agent"
    networks:
      - agent-net
    environment:
      AGENT_MANAGEMENT_PROXY: "host.docker.internal"
      AGENT_MANAGEMENT_PROXY_PORT: 30001
      AGENT_MANAGEMENT_PROXY_USE_SSL: "true"
      TELEMETRY_NO_PCA_DATAPATH: "true"   ### <— ADD THIS LINE 
    ports:
      - "57000:57000"
    volumes:
      - ./secrets/secrets.yaml:/run/secrets/secrets.yaml
      - ./secrets/ca.pem:/usr/local/share/ca-certificates/ca.pem
      - ./secrets/tls.pem:/usr/local/share/ca-certificates/tls.pem
networks:
  agent-net:
    driver: bridge

When you start your agent, if you tail the agent logs you can confirm that the PCA data path has been successfully disabled by watching for the following logs:

******************************************************************************************************
********* PCA Data Link DISABLED - Metrics data will NOT be forwarded to PCA for ingestion!! *********
****************************************************************************************************** *******************************************************************************************************************************************************************************

Kubernetes Deployment

This section documents how to disable the PCA data path if you are deploying your Telemetry Collector into a Kubernetes cluster.

Follow the instructions for deploying a Telemetry Collector to Kubernetes as per the usual procedure, ensuring the flag for this feature is set, as per instructions below.

When running the ‘helm install’ command you must add the following additional parameter to set the flag which disables the PCA data path:

--set telemetryCollector.config.telemetryNoPcaDatapath=true

An example of the full command would look similar to the following):

helm install telemetry-agent oci://gcr.io/sky-agents/charts/telemetry-collector \
  --version 0.55.0 \
  --namespace cisco-sensor-collectors \
  --set agentAgentId="48e74f38-90ea-4830-83ba-ea19bc89fff4" \
  --set agentAuthenticationToken='ppQxTi8Ys7ZCyWSP-SmC8CdrrxJaPZoMjrv1JLbwnAIWob8sX5CrvSlp5c2aMqzpzb3HTfY' \
  --set agentManagement="sensor-collector-connector" \
  --set agentManagementPort=25777 \
  --set custom_ca_certificates.ca.enabled=true \
  --set custom_ca_certificates.ca.secretName=ca-certificate-secret \
  --set custom_ca_certificates.ca.secretKey=ca.pem \
  --set custom_ca_certificates.tls.enabled=true \
  --set custom_ca_certificates.tls.secretName=tls-certificate-secret \
  --set custom_ca_certificates.tls.secretKey=tls.pem \
 --set telemetryCollector.config.telemetryNoPcaDatapath=true ### <-- NEW LINE

2. Configuring The Telemetry Collector To Output To Kafka

Once you have disabled the Provider Connectivity Assurance ingestion pipeline for your Telemetry Collector, you will need to configure the collector to send its data to your preferred destination.

The Telemetry Collector encapsulates an embedded Telegraf instance which runs inside its container. The runtime configuration is sent to the Telemetry Collector from Provider Connectivity Assurance, and it contains (amongst other settings) base 64 encoded yaml for configuring Telegraf.

This configuration yaml typically includes an input plugin plus some number of processor plugins which are responsible for data transformation, with an appropriate output plugin injected under the hood to enable output to the PCA ingestion pipeline (this is what we disabled in the previous step).

To output your processed data to Kafka (or another destination) you must craft an appropriate output plugin configuration and append it to the telemetry portion of the run time configuration for your Telemetry Collector.

For documentation regarding the available output plugins for Telegraf you may refer to Telegraf documentation: https://docs.influxdata.com/telegraf/v1/plugins/

The Telemetry Collector User Guide contains information describing how to customize the configuration payload for a Telemetry Collector, so this page will not go into details regarding this process, but at a high level the procedure is as follows:

First, extract the agent configuration for your Telemetry Collector via an API call

The configuration json will include a stanza that looks like the following:

    "agentConfiguration": {
      "telemetry": {
        "dataCollection": "base 64 encode blob",
        "templateName": "Cisco-Telemetry-IOS-XR",
        "templateVersion": 1
      }
    },

Extract the base 64 encoded blob and decode it - this will result in the Telegraf configuration yaml
Append your output plugin configuration to the decoded yaml
Re-encode it using base 64
Update the agent configuration via API call, replacing the original base 64 blob with your newly encoded blob (which includes your desired output plugin).

If your Telemetry Collector is already running, the new configuration will be pushed down to it automatically. If you have not started it yet, then it will get this configuration when it is started and connects to Provider Connectivity Assurance.

Note that although data is not streamed to Provider Connectivity Assurance, there is still connectivity to Provider Connectivity Assurance which allows the run time configuration to be passed to the Telemetry Collector.