Agent Installation Guidelines

Installation and Deployment Guidelines

Requirements on Runtime Environment

The Cisco Provider Connectivity Assurance Sensor Agents (formerly Accedian Skylight sensor: agents) are containerized test functions that require a container environment in order to run.

Currently, we recommend Docker 20.10.0 or later on AMD64/x86-64 or ARM64 linux systems, and management of containers using Kubernetes, Docker CLI commands or Docker-Compose. Kubernetes control of container lifecycle is supported with helm charts for all agents.
Capability NET_ADMIN is mandatory to enable in the environment, if this capability is not enabled the agent may not operate as expected.

Agent Typical Resource Consumption

For containers, the amount of RAM, CPU and disk is rarely provisioned strictly by the container environment. Instead, the usage is typically monitored and the environment can be configured to alert on overutlization of a certain resource.

As a guideline, the Assurance Sensor Agents will under typical usage consume up to:

A vCPU is considered as a thread on standard x86 CPU, running typically at 2.2 to 2.4GHz
CPU usage for actuate is shown as a range depending on load. Highest load is 5000 pps (for example 500 sessions of 10 pps each).

The logging from agents is through standard stdout mechanism, and it is up to the container environment to apply restrictions on how much data should be logged, and for how long. Keeping large amounts of logs will certainly increase the disk space used for the agents.

Agent Connectivity Requirements

In order to operate, the agent must be able to connect to a Sensor Collector instance using TCP on port 55777 (default port). This connection is referred to as the MANAGEMENT_PROXY and is used for management of the agent, and is authenticated using a secret provided in a YAML file to the agent upon start.

Secondly, the agent will communicate with a Sensor Collector on TCP port 55888 (default) to sent measurement data and metrics. This is referred to as the DATA_GATEWAY.

If you don't have a Sensor Collector instance up and running for agents, you must set it up. For information, see Sensor Collector (formerly Roadrunner).

Lastly, the agent needs connectivity to the test destination. This may be a reflector, responder, service or another agent - all depending on the test type and configuration. Each agent type has different connectivity requirements for their test traffic.

Summary of connections required to be opened through any firewalls in between

Note that all connections agent-Sensor Collector-analytics are "outbound", which means that it is supported to use NAT/PAT firewalls between each of these components, as all connections are initiated from the edge towards the central location. (Agent->Sensor Collector and Sensor Collector->Analytics cloud)

Agent Installation

The Assurance Sensor Agents are provided as packages directly loadable into the Docker environment. To load an agent into your Docker environment, use the docker pull command for the appropriate agent:

x86/AMD versions

actuate x86_64 or amd64:

sudo docker pull gcr.io/sky-agents/agent-actuate-amd64:r24.09

throughput x86_64 or amd64:

sudo docker pull gcr.io/sky-agents/agent-throughput-amd64:r24.09

trace x86_64 or amd64:

sudo docker pull gcr.io/sky-agents/agent-trace-amd64:r24.09

transfer x86_64 or amd64:

sudo docker pull gcr.io/sky-agents/agent-transfer-amd64:r24.09

ARM 64-bit versions

actuate arm64:

sudo docker pull gcr.io/sky-agents/agent-actuate-arm64:r24.09

throughput arm64:

sudo docker pull gcr.io/sky-agents/agent-throughput-arm64:r24.09

trace arm64:

sudo docker pull gcr.io/sky-agents/agent-trace-arm64:r24.09

transfer arm64:

sudo docker pull gcr.io/sky-agents/agent-transfer-arm64:r24.09

Downloading and exporting agent container as a file

If a Cisco Provider Connectivity Sensor Agent needs to be installed on a host that has no internet access, all agents can be downloaded from software.cisco.com under the Provider Connectivity Assurance category

Alternatively, an image fetched with docker pull can also be exported using docker save to a tar.gz file:

sudo docker save gcr.io/sky-agents/agent-actuate-amd64:r24.09 | gzip > agent-actuate-amd64.r24.09.tar.gz

This Docker image tar.gz file can then be loaded on the target plaform using docker load command:

sudo docker load -i agent-actuate-amd64.r24.09.tar.gz

Once loaded, the agent image should be present in the output from docker images:

sudo docker images

The output should show that the image has been loaded:

REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
agent-actuate-docker          yy.mm_nnnn           4fab523db717        1 week ago        24MB

Agent Startup

For an agent to be started and ready to perform tests, it needs a couple of configuration parameters to be defined:

• An external port mapping for the built-in RFC5357 TWAMP-light reflector (actuate agent only, and is not needed if no reflection is required)

• A secrets YAML file containing an agent UUID and initial key for authentication
  OR
  A token provided to the agent via environment variable
• IP address or FQDN, and port where it can find a Sensor Collector agent management service
• Capability flag NET_ADMIN enabled

These parameters can be supplied inside a Docker-Compose file, or specified as environment variables on the command line when starting the container with docker run.

CA trust bundle selection

At startup the agent checks the AGENT_CA_TRUST_BUNDLE environment variable for any of the supported options:

If no option is provided, the default mode is "intersect".

Define a new agent in the orchestration API

Each agent needs to have a definition, or configuration, in the orchestration system to be accepted and controllable.
To add a new agent to the orchestration use the API calls as outlined below. Four configuration attributes need to be set:

• agentId - a unique UUID identifying this agent
• agentName - a string of unicode characters
• gateway server - IP address or FQDN to Sensor Collector
• gateway port - port used for metrics data streaming (default 55888)

To generate a UUID for your agent, use the built-in UUID generator in Linux,

$ uuidgen
ae311d23-5ca7-fake-8317-25example54a

or an online service like UUID Generator.

Create a configuration object like the one below:

{"data": 
{
    "type": "agentConfigs",
    "attributes": {
        "agentId": "ae311d23-5ca7-fake-8317-25example54a",
        "dataGateway": {
            "server": "192.168.0.4",
            "port": 55888
        },
        "identification": {
            "agentName": "demo-agent-12345"
        }
    }
}
}

Then, POST it to the orchestration service to register this new agent:

POST
{{your-tenant-name}}/api/orchestrate/v3/agents/configuration

Step 2 - Grab secret from orchestration API and create the secrets file

As described in the previous sections, each agent needs an authentication token to be able to register to the orchestration and get managed. To retrieve the initial token, use the agent control UI or the API towards the orchestration service to define this agent and get the secret.

The API call requires only a unique agent ID to register this agent and return the secrets token.

Registering the agent with the orchestration, and generating the secret for authentication is then done by POST:ing the UUID to the orchestration API:

POST
{{your-tenant-name}}/api/orchestrate/v3/agents/ae311d23-5ca7-fake-8317-25example54a/secrets

The output from the POST will be the initial secret for this agent, and should be placed in a file for use by the agent when launching. An example secret is shown below:

agentConfig:
    identification:
        agentId: ae311d23-5ca7-fake-8317-25example54a
        authenticationToken: eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhY2NlZGlhbi5jb20iLCJzdWIiOiJhcGl1c2VyQG55ZGVsbC5zZSIsImV4cCI6MTYyAksIOwKNywiYXVkIjoiIiwidG9rZW5JRCI6MTM0OCwidGVuYW50SUQiOiJjODRhZWUzNC04ODhjLTRmMDYtYmFmMy00NGZkMmNhM2JkM2UifQ.mUu5ZrsfjIWipzf5zqOpA4PUqWbu_CSWaSEmPC3wygv4

For the examples below, this secret is saved in a file called
/home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml

Alternative Step 2 - token as environment variable instead of secrets file

In some cases it may be more suitable to provide the security token as an environment variable when launching the agent. If using this method, define the environment variables AGENT_ID and the AGENT_AUTHENTICATION_TOKEN set to the token generated in step 1, example:

--env AGENT_ID=ae311d23-5ca7-fake-8317-25example54a \
--env AGENT_AUTHENTICATION_TOKEN=eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhY2NlZGlhbi5jb20iLCJzdWIiOiJhcGl1c2VyQG55ZGVsbC5zZSIsImV4cCI6MTYyAksIOwKNywiYXVkIjoiIiwidG9rZW5JRCI6MTM0OCwidGVuYW50SUQiOiJjODRhZWUzNC04ODhjLTRmMDYtYmFmMy00NGZkMmNhM2JkM2UifQ.mUu5ZrsfjIWipzf5zqOpA4PUqWbu_CSWaSEmPC3wygv4

If using this method, note that the AGENT_ID also needs to be set through environment variable, since it cannot be fetched from the secrets file when not using a secrets file.

Step 3 - Launch an agent

Below are examples of two different methods to launch an actuate agent. The throughput, transfer and trace agents are launched in the exact same way, with the exception of the TWAMP reflection port which is not needed for the throughput agent.

Both launch methods described below will open UDP for TWAMP reflection on port 4000 (internal is by default 862), connect towards a Sensor Collector on 10.11.12.13:55777/tcp and define the secrets file as /home/agentuser/3dfe3705-42f3-fake-80bb-59a218a0900e.yaml

It is considered good practice to use the agent UUID as part of the file name for the secret so that if you deploy several agents on the same host they get their own secrets file.

See Agent secrets file options for other metods of providing the authentication secret to the agent

Step 3.a Launch agent using Docker-Compose

To use Docker-Compose, the configuration parameters should be put in a YAML file. AGENT_ID and access token are fetched from the secrets file defined so they need not be defined explicitly in the docker-compose file.
Note that the NET_ADMIN capability should be set, without this enabled the agent may not operate as expected.

Example for actuate agent:

version: '3'
services:
  twamp-sender-service:
    container_name: "my-actuate-agent"
    image: "gcr.io/sky-agents/agent-actuate-amd64:r24.09"
    cap_add:
    - NET_ADMIN
    - NET_RAW
    ports:
      - "4000:862/udp"
    restart: "always"
    environment:
      AGENT_MANAGEMENT_PROXY: "10.11.12.13"
      AGENT_MANAGEMENT_PROXY_PORT: "55777"
    volumes:
      - type: bind
        source: /home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
        target: /run/secrets/secrets.yaml

Example for throughput agent:

version: '3'
services:
  tcp-agent-service:
    container_name: "my-throughput-agent"
    image: "gcr.io/sky-agents/agent-throughput-amd64:r24.09"
    cap_add:
    - NET_ADMIN
    - NET_RAW
    restart: "always"
    environment:
      AGENT_MANAGEMENT_PROXY: "10.11.12.13"
      AGENT_MANAGEMENT_PROXY_PORT: "55777"
    volumes:
      - type: bind
        source: /home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
        target: /run/secrets/secrets.yaml

The agent can now be launched using the docker-compose up command:

sudo docker-compose up -d

If the agent configuration was placed in a file named something other than docker-compose.yml, then this needs to be specified on the command line:

sudo docker-compose -f mycomposefile.yml up -d

IMPORTANT:

Depending on your specific version of docker-compose, you might need to run:

sudo docker compose up

So if you have cut and paste the code using the hyphen and it doesn't work, try without the hyphen.

Step 3.b Launch agent on command line

Same setup, but using command line with environment variable options (--env) and port maps (-p)

Example for actuate agent:

sudo docker run -d -p 4000:862/udp \
--env AGENT_MANAGEMENT_PROXY="10.11.12.13" \
--env AGENT_MANAGEMENT_PROXY_PORT="55777" \
--cap-add=NET_ADMIN \
--cap-add=NET_RAW \
--mount type=bind,source=/home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml,target=/run/secrets/secrets.yaml \
gcr.io/sky-agents/agent-actuate-amd64:r24.09

The -d option makes Docker launch the container in the background, in "detached" mode.

Step 3.c Launch agent without secrets file

Same setup, but using command line with environment variable options (--env) and port maps (-p)

Example for actuate agent:

sudo docker run -d -p 4000:862/udp \
--env AGENT_ID="ae311d23-5ca7-fake-8317-25example54a"
--env AGENT_AUTHENTICATION_TOKEN="eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhY2NlZGlhbi5jb20iLCJzdWIiOiJhcGl1c2VyQG55ZGVsbC5zZSIsImV4cCI6MTYyAksIOwKNywiYXVkIjoiIiwidG9rZW5JRCI6MTM0OCwidGVuYW50SUQiOiJjODRhZWUzNC04ODhjLTRmMDYtYmFmMy00NGZkMmNhM2JkM2UifQ.mUu5ZrsfjIWipzf5zqOpA4PUqWbu_CSWaSEmPC3wygv4"
--env AGENT_MANAGEMENT_PROXY="10.11.12.13" \
--env AGENT_MANAGEMENT_PROXY_PORT="55777" \
--cap-add=NET_ADMIN \
--cap-add=NET_RAW \
gcr.io/sky-agents/agent-actuate-amd64:r24.09

The -d option makes Docker launch the container in the background, in "detached" mode.

Step 4 - Check that the agent is running

To check that the agent has started, use the docker ps command:

$ docker ps
CONTAINER ID        IMAGE                                     COMMAND             CREATED             STATUS                 PORTS                                          NAMES
437d53381d83        agent-actuate-docker:yy.mm_nnn   "/sbin/monitor"     5 weeks ago         Up 5 weeks (healthy)   0.0.0.0:4000->862/udp,  my-agent

The output shows for how long the agent has been running, and the UDP port mapping informs that port 4000 is mapped on the host IP for TWAMP reflection.

Using agents with multiple interfaces / VRF / namespaces

All agents support attaching to multiple interfaces by configuration in the docker compose file or via docker command line startup options. The agent can either be provided with a specific --mount mountpoint for a VRF / namespace or be provided with the full set of namespaces available in the /var/netns directory.
To be able to switch namespace, the agent needs to be started with the SYS_ADMIN privilege flag enabled.

Difference between interfaces and namespaces

Linux systems natively support using multiple physical and virtual (VLAN) interfaces, typically named "eth0", "ens4" or something like "eth3.1111" for a VLAN residing on port eth3.
On a standard linux system these interfaces cannot have overlapping IP addressing subnets and routes, which means that usually each interface has its own IP subnet, and possibly a set of static routes pointing out destination networks that are reachable through a router on this subnet. There is also only one default route available on the system.
With the addition of namespaces / VRF the linux kernel provides the ability to separate the routing function from the system-global and apply a separate routing domain for each namespace / vrf. With this separation you can have several interfaces defined, with the same IP address, and with different default routes.

Note that when an interface is reassigned to a different namespace than the default one, it will typically disappear from the interface list seen with "ip addr" command.

Example commands to add a namespace, and associate eth3.123 with this namespace

ip netns add mynamespace
ip link set eth3.123 netns mynamespace

Example docker compose file that gives the agent access to the namespace directory on the system (last volumes bind)

version: '3'
services:
  agent_actuate:
    container_name: "mycontainername"
    image: "gcr.io/sky-agents/agent-actuate-amd64:r24.09"
    cap_add:
    - NET_ADMIN
    - NET_RAW
    - SYS_ADMIN
    restart: always
    network_mode: host
    ports:
      - "862:862/udp"
    environment:
      AGENT_MANAGEMENT_PROXY: "12.13.14.15"
    volumes:
      - type: bind
        source: /home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
        target: /run/secrets/secrets.yaml
      - type: bind
        propagation: rslave
        source: /var/run/netns
        target: /var/run/netns

Agent Launch Parameters

When launching an agent it accepts a number of parameters that define the instatiation of the container. The mandatory ones, like AGENT_MANAGEMENT_PROXY, have been described in the previous sections. Below is the full list of supported parameters, that can be provided as environment variables through the container instantiation when starting the agent.

The agent is now ready to be configured using the orchestration API, see Working with the agents control API for more information on using the API, or agent orchestration API reference for full details.