Deploying Roadrunner in Kubernetes

Helm is advertised as a package manager for Kubernetes, and allows operators to define services and their dependencies using yaml files. Accedian provides the necessary helm charts and pre-seeded values files needed to deploy Roadrunner as a pod in a Kubernetes cluster.

Working knowledge of Kubernetes should be considered as a pre-requisite for this workflow.

For StorageClass requirements, you can consult the documentation in the link below to use the right configuration for your specific environment (including Kubernetes engine, risk tolerances, preferred solution, etc.).

Kubernetes Storage Classes

Environment Requirements

Ensure you have the following:

1. Kubernetes v1.26 or later running on the compute resource where you want Roadrunner to run
2. A Kubernetes cluster created where you want to deploy Roadrunner
3. A default StorageClass that supports dynamic creation of RWO PersistentVolumes
4. A dedicated Kubernetes namespace for Roadrunner
   (this is optional, and can be created by Helm is operators)
5. kubectl with config to manage the Kubernetes cluster
6. Helm v3 or later available on the compute where the installation will be performed, and that has connectivity to your Kubernetes cluster.
7. A minimum of:
   • 2 vCPU
   • 4 GB of RAM
   • 100GB of storage

Preparation

To start, you'll need to grab two artifacts and make them available to the working directory for helm.

1. Roadrunner Helm chart bundle
2. Helper file bundle with Skylight connector configuration

Getting Roadrunner Helm Charts

The Roadrunner helm charts can be downloaded from our software page here:

• https://software.accedian.io/?prefix=roadrunner/analytics/

Or downloaded via CLI (replacing the release and version tags as necessary):

> curl https://software.accedian.io/roadrunner/analytics/23.04/helm/roadrunner-23.04-helm-0.456.17.tgz --output roadrunner-23.04-helm-0.456.17.tgz

OR

> helm pull https://software.accedian.io/roadrunner/analytics/23.04/helm/roadrunner-23.04-helm-0.456.17.tgz

The above example results in a file named roadrunner-23.04-helm-0.456.17.tgz being downloaded into your working directory.

SFTP this file to the deployment location in the working directory that you will perform the install from (should be the same location as where the helper artifacts from the next step were downloaded). There is no need to untar this file as helm can consume it as is.

Getting Helm Helper Artifacts

Skylight analytics provides an endpoint for downloading a helper bundle with some pre-populated artifacts which can be used during the Roadrunner Helm install. The helper files can be accessed via the UI or API.

Helm helper artifacts via UI

Navigate to the Roadrunner / connector configuration screen in Settings, find your connector config, and select the ellipsis icon for more options.
Select the download option for Kubernetes as shown in the example below:

Helm helper artifacts via API

After authenticating, you can download the artifacts by submitting a GET REST API request to Analytics as follows:

GET https://{{analytics tenant url}}/api/v2/distribution/download-roadrunner-helm?zone={{your zone}}

Where:
- analytics tenant url = The tenant URL for your Analytics deployment
- zone = The zone for the Roadrunner instance you wish to deploy (corresponds to the connector created in Analytics for your Roadrunner)

Helm helper artifact preparation

The result of either download option will be a tarball with some pre-populated files the deployment of your Roadrunner instance. Unpack the tarball into the working directory that you will perform your Helm install from:

> tar -xzvf <connector config name>-<download date>-helm.tar.gz

This will produce 2 files:

1. bootstrapToken
   • JWT to allow initial connection to Skylight to exhange certs and keys
   • Token is generated on download, and is only valid for a short period afterwards
2. values.yml
   • Roadrunner configuration around ports, mount points, where Skylight is deployed
   • It also specifies whether the Roadrunner mode of operation requires the deployment of any sidecars services. More on sidecars later.

Deployment

Deploying with Helm

Once you have downloaded the Helm artifacts for your connector and placed them in a location where helm can access them and your Kubernetes cluster, you're ready to deploy.

Via a terminal connected to the compute where the helm client is available, run the following:

> helm install {deployment name} ./{roadrunner helm artifact} --create-namespace --namespace {namespace} --values ./values.yaml --set bootstrapToken=$(cat ./bootstrapToken)

OR

> helm install agent-rr-in-k8s {URL to file} --values values.yaml --set bootstrapToken=$(cat ./bootstrapToken)

Where:

• The deployment name is the label Helm will apply to your Roadrunner deployment.
  This can be anything you want... but cannot contain capitals or spaces as per helm rules.
• {roadrunner helm artifact} is the helm chart that you downloaded earlier (i.e. roadrunner-23.04-helm-0.456.17.tgz)
  Alternatively you can specify the URL to the file if the helm environment has outbound internet connectivity.
• --create-namespace is optional if the namespace does not exist
• --namespace {namespace} tells Helm which Kubernetes namespace you want Roadrunner deployed into.
  Omitting this will deploy to the default namespace
• ./values.yaml is the pre-populated values file that was downloaded from Skylight (the above example assumes you are invoking Helm from the working directory which you downloaded and untarred the helper bundle to).
• --set bootstrapToken=$(cat ./bootstrapToken) applies the short lived JWT included in the helper artifacts bundle. Roadrunner needs this for its initial authentication with Analytics.

A successful deployment will look like the following:

> helm install agent-rr-in-k8s https://software.accedian.io/roadrunner/analytics/23.04/helm/roadrunner-23.04-helm-0.456.17.tgz --values values.yaml --set bootstrapToken=$(cat ./bootstrapToken)

NAME: agent-rr-in-k8s
LAST DEPLOYED: Mon May 15 13:13:35 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Roadrunner application version 0.456.17 installed for zone cf_agent_RR_K8S.

General Notes:
--------------
Analytics Endpoints:
   - Deployment: medley.medley.npav.accedian.net:443
   - Tenant    : united.medley.npav.accedian.net:443

Sensor Agent Gateway Notes:
---------------------------
Sensor Agent Admin Proxy    : agent-rr-in-k8s-connector:55777
Sensor Agent Metrics Gateway: agent-rr-in-k8s-connector:55888

Use helm to view all of your deployments:

> helm list
NAME           	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART              	APP VERSION
agent-rr-in-k8s	default  	1       	2023-05-15 13:13:35.872792 -0400 EDT	deployed	roadrunner-0.456.17	0.456.17

Checking new pod status

Use kubectl to confirm the new pod is running correctly.

1. Find out the pod name:

> kubectl get pods

NAME                              READY   STATUS    RESTARTS   AGE
connector-for-cf-agent-rr-k8s-0   1/1     Running   0          11m

2. Look it up:

> kubectl describe pod connector-for-cf-agent-rr-k8s-0    

Name:             connector-for-cf-agent-rr-k8s-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.4
Start Time:       Mon, 15 May 2023 13:13:36 -0400
Labels:           app=roadrunner
                  controller-revision-hash=connector-for-cf-agent-rr-k8s-6bd9d4c4f5
                  statefulset.kubernetes.io/pod-name=connector-for-cf-agent-rr-k8s-0
Annotations:      checksum/config: 91230865df6342dbe39c162dba8c869757e4c7dbe1090d171916bf5a2366433c
Status:           Running
IP:               10.1.0.26
IPs:
  IP:           10.1.0.26
Controlled By:  StatefulSet/connector-for-cf-agent-rr-k8s
Init Containers:
  roadrunner-init:
    Container ID:  docker://55a7345a2fa62db3360548204332e770535f0abf85aabe554b73081b2a476d23
    Image:         gcr.io/npav-172917/adh-roadrunner:0.456.17
    Image ID:      docker-pullable://gcr.io/npav-172917/adh-roadrunner@sha256:4e35d73d3606e32a7ff37af1cc098be8587024eed9a94a06476916a4eac0dc48
    Port:          <none>
    Host Port:     <none>
    Args:
      login
      --config=/tmp/config/adh-roadrunner.yml
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 15 May 2023 13:13:37 -0400
      Finished:     Mon, 15 May 2023 13:13:37 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /roadrunner-app from roadrunner-app (rw)
      /tmp/auth from bootstrap-secret (rw)
      /tmp/config from roadrunner-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kbmp9 (ro)
Containers:
  roadrunner:
    Container ID:  docker://2c84780c4338b7a84b87f779c3ce3aa738461b3ec6c0be2665e9d6bb331a94bc
    Image:         gcr.io/npav-172917/adh-roadrunner:0.456.17
    Image ID:      docker-pullable://gcr.io/npav-172917/adh-roadrunner@sha256:4e35d73d3606e32a7ff37af1cc098be8587024eed9a94a06476916a4eac0dc48
    Ports:         55777/TCP, 55888/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      data
      --config=/tmp/config/adh-roadrunner.yml
    State:          Running
      Started:      Mon, 15 May 2023 13:13:38 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /roadrunner-app from roadrunner-app (rw)
      /roadrunner-data from roadrunner-data (rw)
      /tmp/auth from bootstrap-secret (rw)
      /tmp/config from roadrunner-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kbmp9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  roadrunner-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  roadrunner-data-connector-for-cf-agent-rr-k8s-0
    ReadOnly:   false
  roadrunner-app:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  roadrunner-app-connector-for-cf-agent-rr-k8s-0
    ReadOnly:   false
  roadrunner-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      agent-rr-in-k8s-config
    Optional:  false
  bootstrap-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  agent-rr-in-k8s-bootstrapjwt
    Optional:    false
  kube-api-access-kbmp9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  9m14s  default-scheduler  Successfully assigned default/connector-for-cf-agent-rr-k8s-0 to docker-desktop
  Normal  Pulled     9m13s  kubelet            Container image "gcr.io/npav-172917/adh-roadrunner:0.456.17" already present on machine
  Normal  Created    9m13s  kubelet            Created container roadrunner-init
  Normal  Started    9m13s  kubelet            Started container roadrunner-init
  Normal  Pulled     9m12s  kubelet            Container image "gcr.io/npav-172917/adh-roadrunner:0.456.17" already present on machine
  Normal  Created    9m12s  kubelet            Created container roadrunner
  Normal  Started    9m12s  kubelet            Started container roadrunner

3. Check the logs:

> kubectl logs -f {pod name}

Defaulted container "roadrunner" out of: roadrunner, roadrunner-init (init)
17:13:38.285 initConfig ▶ DEBU 001 Using config file: /tmp/config/adh-roadrunner.yml
17:13:38.285 Run ▶ INFO 002 starting roadrunner version: 0.456.17
17:13:38.285 checkConfigVersion ▶ DEBU 003 Checking config version
17:13:38.285 checkConfigVersion ▶ DEBU 004 Adapting to old config format
17:13:38.286 mergeJWT ▶ WARN 005 Missing some or all required parameters from JWT, ignoring them: dhhost: ,  tenantHost: , zone: , tenantID: 9ca4447b-e32a-4dc2-af71-2a1219cc3373
17:13:38.286 Run ▶ WARN 006 config file is missing connectorId - falling back to using the zone (cf_agent_RR_K8S) as the connectorId
17:13:38.286 Run ▶ INFO 007 Setting log level from config to INFO
17:13:38.286 Run ▶ INFO 008 meta runner pointing to united.medley.npav.accedian.net:443
17:13:38.286 Run ▶ INFO 009 data runner pointing to united.medley.npav.accedian.net:443
17:13:38.286 certExpiryChecker ▶ INFO 00a checking certs for renewal in: /roadrunner-app/.ssh/
17:13:38.286 RenewCert ▶ INFO 00b Renewing client cert
17:13:38.293 SendCSRWithCert ▶ INFO 00c sending CSR to DH on host: united.medley.npav.accedian.net:443
17:13:38.673 RequestRuntimeToken ▶ INFO 00d Runtime token on file is valid.
17:13:38.673 connect ▶ INFO 00e Attempting to establish meta connection with DH on host medley.medley.npav.accedian.net in zone: cf_agent_RR_K8S
17:13:38.673 GetCerts ▶ INFO 00f found cert and private key
17:13:38.799 connect ▶ INFO 010 Successfully connected to DataHub (meta connection) on host: medley.medley.npav.accedian.net in zone: cf_agent_RR_K8S for tenant: 9ca4447b-e32a-4dc2-af71-2a1219cc3373.
...

Upgrading a deployment

> helm upgrade {deployment name} {new chart package or URL}

For example:

> helm upgrade agent-rr-in-k8s https://software.accedian.io/roadrunner/analytics/23.04/helm/roadrunner-23.04-helm-0.456.17.tgz

Release "agent-rr-in-k8s" has been upgraded. Happy Helming!
NAME: agent-rr-in-k8s
LAST DEPLOYED: Mon May 15 13:35:08 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Roadrunner application version 0.456.17 installed for zone cf_agent_RR_K8S.

General Notes:
--------------
Analytics Endpoints:
   - Deployment: medley.medley.npav.accedian.net:443
   - Tenant    : united.medley.npav.accedian.net:443

Sensor Agent Gateway Notes:
---------------------------
Sensor Agent Admin Proxy    : agent-rr-in-k8s-connector:55777
Sensor Agent Metrics Gateway: agent-rr-in-k8s-connector:55888

Sidecars

In the case where you are replacing an existing Docker based Roadrunner deployment with a Kubernetes deployment, depending on the type of Roadrunner being migrated, there are a few addition details to be aware of:

Filewatchers

For Filewatcher type Roadrunners, the Kubernetes deployment includes an SFTP server which is used for injecting files into a persistent volume within the Kubernetes cluster which acts as the ingestion directory for Roadrunner. You'll notice this sidecar definition in your connector config values.yaml file like follows:

> cat cat values.yaml

...
custom:
  sidecars:
    containers:
    ## Put templates for custom sidecars here ##

      ## SFTP Server sidecar for filewatcher.
      ## ------------------------------------
      ## NOTES:
      ##   To change username/password for SFTP server access:
      ##      - replace args with your desired credentials
      - name: atmoz-sftp
        image: docker.io/atmoz/sftp:latest
        ports:
        - containerPort: 22
        args: ["*{USERNAME}*:*{PASSWORD}*:10001::upload"]
        volumeMounts:
        - name: roadrunner-monitored
          mountPath: /home/roadrunner

Any data sources which are providing .csv files for ingestion will need to transfer them via this SFTP server. The deployment notes will provide instructions for determining the IP address and port to use for accessing the SFTP server.

 Roadrunner Filewatcher Notes:
 -----------------------------
 An SFTP sidecar service is included as part of this Roadrunner deployment to enable files to be submitted to Roadrunner for processing.

 The Roadrunner container is configured to monitor a shared volume which maps to the location '**home/roadrunner/upload**' on the SFTP server.
 This is the location which files should be SFTP'd to in order for Roadrunner to ingest them.

 The exposed Node Port for the SFTP service can be found by executing the following command at the command line:
     `kubectl -n default get -o jsonpath='{.spec.ports[?(@.name=="sftp-server")].nodePort}{"\n"}' svc file-rr-in-k8s-connector`

 To get monitored volume shared between the Roadrunner and SFTP services, run the following command:
     `kubectl get pv | grep default/roadrunner-monitored-connector-for-cf-filewatcher-rr-k8s-zone`

For example

> kubectl -n default get -o jsonpath='{.spec.ports[?(@.name=="sftp-server")].nodePort}{"\n"}' svc file-rr-in-k8s-connector

30312

Testing the SFTP service

Testing that connection out using ssh to make sure the port is open (following example is run on the Kubernetes node directly):

> ssh roadrunner@127.0.0.1 -p 30312

The authenticity of host '[127.0.0.1]:30312 ([127.0.0.1]:30312)' can't be established.
ED25519 key fingerprint is SHA256:PmRG***MjqbMywSRxk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:30312' (***) to the list of known hosts.
roadrunner@127.0.0.1's password:
Connection from user roadrunner 192.***.***.4 port 52400: refusing non-sftp session
This service allows sftp connections only.
Connection to 127.0.0.1 closed.

Test out SFTP itself by:

1. SFTPing to the service

> sftp -P 30312 roadrunner@127.0.0.1

roadrunner@127.0.0.1's password:
Connected to 127.0.0.1.

2. Confirming that there is a "upload" folder in the SFTP user's home directory

sftp> pwd
Remote working directory: /
sftp> ls
upload

3. Use "put" to upload an Accedian CSV

sftp> put *{CSV file from local machine}* upload/.
Uploading *{CSV file from local machine}* to /upload/./*{CSV file from local machine}*
*{CSV file from local machine}*                                                            100% 2599   502.2KB/s   00:00
sftp> exit

4. Check out the roadrunner logs to confirm it picked up the file.
   Notice the roadrunner is appended to the kubectl command as in filewatcher mode this command would have defautled to the sftp server logs.

> kubectl logs -f connector-for-cf-filewatcher-rr-k8s-zone-0 roadrunner
...
16:05:28.684 sendMetrics ▶ INFO 0cf Sending metrics to DH on host: {analytics URL} in zone: cf_filewatcher_RR_K8S_zone
16:05:35.559 queueFileForProcessing ▶ INFO 0d0 Queuing File: /monitored-dir/upload/*{CSV FILE}* for processing
16:05:35.561 processFile ▶ INFO 0d1 Processing: /monitored-dir/upload/*{CVS FILE}* of type: eth-vs
...

Updating Skylight Orchestrator

Reminder that metric export settings are configured in Skylight Orchestrator.
More info on that configuration:

• Configuring Global Collection Settings

Sensor Agent Roadrunners

The Kubernetes Roadrunner deployment for Sensor Agents exposes the Agent Proxy and Metrics Gateway endpoints (ie: the admin and management channels for the agents) via Node Ports.

The deployment notes provide instructions for determining those port assignments. Any existing sensor agents will need to have their configurations updated to point to the new deployment.

> kubectl -n default get -o jsonpath='{range .spec.ports[*]}[{.name}, {.nodePort}] {end}' svc agent-rr-in-k8s-connector

[pprof, 31048] [agent-proxy, 31580] [metrics-gateway, 31156]

Be sure to specify the correct port in your docker compose file as well using the AGENT_MANAGEMENT_PROXY_PORT environment variable:

❯ cat agent-sending-to-k8s.yml
version: '3'
services:
  twamp-agent-service:
    container_name: "demo-actuate"
    image: "gcr.io/sky-agents/agent-actuate-arm64:r23.04"
    ports:
      - "5862:5862/udp"
    hostname: "MAC"
    restart: "always"
    environment:
            AGENT_MANAGEMENT_PROXY: "192.168.1.140"
            AGENT_MANAGEMENT_PROXY_PORT: "31580"
            AGENT_DATA_PROXY: "192.168.1.140"
    volumes:
      - /home/agent_k8s/{UID-secret-file}.yaml:/run/secrets/secrets.yaml

Open Metrics Scraper

Nothing special for this one! No sidecars or Node Ports to expose.