- Print
- PDF
Installation and Deployment Guidelines
By default Docker logs have no log rotations. If Assurance Sensor Agents run for long periods of time please change the default log driver to not risk file system filling up. Please see the Docker documentation on logging:
Configure logging drivers
Requirements on Runtime Environment
The Cisco Provider Connectivity Assurance Sensor Agents (formerly Accedian Skylight sensor: agents) are containerized test functions that require a container environment in order to run.
Currently, we recommend Docker 20.10.0 or later on AMD64/x86-64 or ARM64 linux systems, and management of containers using Kubernetes, Docker CLI commands or Docker-Compose. Kubernetes control of container lifecycle is supported with helm charts for all agents.
Capability NET_ADMIN is mandatory to enable in the environment, if this capability is not enabled the agent may not operate as expected.
Agent Typical Resource Consumption
For containers, the amount of RAM, CPU and disk is rarely provisioned strictly by the container environment. Instead, the usage is typically monitored and the environment can be configured to alert on overutlization of a certain resource.
As a guideline, the Assurance Sensor Agents will under typical usage consume up to:
Agent | vCPU* | RAM | Disk |
---|---|---|---|
actuate | 0.05 - 1 | 0.2-0.5GB | 200MB |
throughput | 1 | 0.1GB | 200MB |
transfer | 0.1 | 0.1GB | 200MB |
trace | 0.1 | 0.1GB | 200MB |
A vCPU is considered as a thread on standard x86 CPU, running typically at 2.2 to 2.4GHz
CPU usage for actuate is shown as a range depending on load. Highest load is 5000 pps (for example 500 sessions of 10 pps each).
The logging from agents is through standard stdout mechanism, and it is up to the container environment to apply restrictions on how much data should be logged, and for how long. Keeping large amounts of logs will certainly increase the disk space used for the agents.
Agent Connectivity Requirements
In order to operate, the agent must be able to connect to a Sensor Collector instance using TCP on port 55777 (default port). This connection is referred to as the MANAGEMENT_PROXY and is used for management of the agent, and is authenticated using a secret provided in a YAML file to the agent upon start.
Secondly, the agent will communicate with a Sensor Collector on TCP port 55888 (default) to sent measurement data and metrics. This is referred to as the DATA_GATEWAY.
If you don't have a Sensor Collector instance up and running for agents, you must set it up. For information, see [Skylight Sensor Collector](/v1/docs/skylight-Sensor Collector-overview).
Lastly, the agent needs connectivity to the test destination. This may be a reflector, responder, service or another agent - all depending on the test type and configuration. Each agent type has different connectivity requirements for their test traffic.
Summary of connections required to be opened through any firewalls in between
Path | Direction | protocol | port |
---|---|---|---|
Agent management | Agent to Sensor Collector | tcp | 55777 |
Agent metrics data reporting | Agent to Sensor Collector | tcp | 55888 |
Sensor Collector | Sensor Collector to Analytics | tcp | 443 |
Agent test traffic | Agent to reflector or other agent | tcp/udp | configurable |
Note that all connections agent-Sensor Collector-analytics are "outbound", which means that it is supported to use NAT/PAT firewalls between each of these components, as all connections are initiated from the edge towards the central location. (Agent->Sensor Collector and Sensor Collector->Analytics cloud)
Agent Installation
The Assurance Sensor Agents are provided as packages directly loadable into the Docker environment. To load an agent into your Docker environment, use the docker pull command for the appropriate agent:
x86/AMD versions
actuate x86_64 or amd64:
sudo docker pull gcr.io/sky-agents/agent-actuate-amd64:r24.09
throughput x86_64 or amd64:
sudo docker pull gcr.io/sky-agents/agent-throughput-amd64:r24.09
trace x86_64 or amd64:
sudo docker pull gcr.io/sky-agents/agent-trace-amd64:r24.09
transfer x86_64 or amd64:
sudo docker pull gcr.io/sky-agents/agent-transfer-amd64:r24.09
ARM 64-bit versions
actuate arm64:
sudo docker pull gcr.io/sky-agents/agent-actuate-arm64:r24.09
throughput arm64:
sudo docker pull gcr.io/sky-agents/agent-throughput-arm64:r24.09
trace arm64:
sudo docker pull gcr.io/sky-agents/agent-trace-arm64:r24.09
transfer arm64:
sudo docker pull gcr.io/sky-agents/agent-transfer-arm64:r24.09
Downloading and exporting agent container as a file
If a Cisco Provider Connectivity Sensor Agent needs to be installed on a host that has no internet access, all agents can be downloaded from software.cisco.com under the Provider Connectivity Assurance category
Alternatively, an image fetched with docker pull can also be exported using docker save to a tar.gz file:
sudo docker save gcr.io/sky-agents/agent-actuate-amd64:r24.09 | gzip > agent-actuate-amd64.r24.09.tar.gz
This Docker image tar.gz file can then be loaded on the target plaform using docker load command:
sudo docker load -i agent-actuate-amd64.r24.09.tar.gz
Once loaded, the agent image should be present in the output from docker images:
sudo docker images
The output should show that the image has been loaded:
REPOSITORY TAG IMAGE ID CREATED SIZE
agent-actuate-docker yy.mm_nnnn 4fab523db717 1 week ago 24MB
Agent Startup
For an agent to be started and ready to perform tests, it needs a couple of configuration parameters to be defined:
- An external port mapping for the built-in RFC5357 TWAMP-light reflector (actuate agent only, and is not needed if no reflection is required)
- A secrets YAML file containing an agent UUID and initial key for authentication
OR
A token provided to the agent via environment variable - IP address or FQDN, and port where it can find a Sensor Collector agent management service
- Capability flag NET_ADMIN enabled
These parameters can be supplied inside a Docker-Compose file, or specified as environment variables on the command line when starting the container with docker run.
CA trust bundle selection
At startup the agent checks the AGENT_CA_TRUST_BUNDLE environment variable for any of the supported options:
mode | Description |
---|---|
"core" | Contains only Cisco Core CAs. For communication to and from Cisco |
"intersect" | Contains the Cisco Core Bundle and a the CAs that are trusted by all major root store programs. This is the default option |
"union" | Favors connectivity at the cost of security. Includes the Cisco Core Bundle and every trusted public CA |
"fedramp" | Use for US federal operations. Contains Cisco Core Bundle plus US CAs from the Intersect Bundle and the US DoD External PKI Trust Chains |
"mozilla" | Use Alpine Linux (Mozilla) distro trusted root store |
core |
If no option is provided, the default mode is "intersect".
Optionally, additional parameters can be defined like an agent name, an interface binding or additional logging options. These additional parameters are described in the documentation for each individual sensor: agent.
Define a new agent in the orchestration API
When quering and changing configuration through the orchestration API it is highly recommended to use a dedicated service account for this task and not an individual user account. Agents use JWT tokens for authentication and these will expire if the user account is deleted.
Each agent needs to have a definition, or configuration, in the orchestration system to be accepted and controllable.
To add a new agent to the orchestration use the API calls as outlined below. Four configuration attributes need to be set:
- agentId - a unique UUID identifying this agent
- agentName - a string of unicode characters
- gateway server - IP address or FQDN to Sensor Collector
- gateway port - port used for metrics data streaming (default 55888)
To generate a UUID for your agent, use the built-in UUID generator in Linux,
$ uuidgen
ae311d23-5ca7-fake-8317-25example54a
or an online service like UUID Generator.
Create a configuration object like the one below:
{"data":
{
"type": "agentConfigs",
"attributes": {
"agentId": "ae311d23-5ca7-fake-8317-25example54a",
"dataGateway": {
"server": "192.168.0.4",
"port": 55888
},
"identification": {
"agentName": "demo-agent-12345"
}
}
}
}
Then, POST it to the orchestration service to register this new agent:
POST
{{your-tenant-name}}/api/orchestrate/v3/agents/configuration
Step 2 - Grab secret from orchestration API and create the secrets file
As described in the previous sections, each agent needs an authentication token to be able to register to the orchestration and get managed. To retrieve the initial token, use the agent control UI or the API towards the orchestration service to define this agent and get the secret.
The API call requires only a unique agent ID to register this agent and return the secrets token.
Registering the agent with the orchestration, and generating the secret for authentication is then done by POST:ing the UUID to the orchestration API:
POST
{{your-tenant-name}}/api/orchestrate/v3/agents/ae311d23-5ca7-fake-8317-25example54a/secrets
The output from the POST will be the initial secret for this agent, and should be placed in a file for use by the agent when launching. An example secret is shown below:
agentConfig:
identification:
agentId: ae311d23-5ca7-fake-8317-25example54a
authenticationToken: eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhY2NlZGlhbi5jb20iLCJzdWIiOiJhcGl1c2VyQG55ZGVsbC5zZSIsImV4cCI6MTYyAksIOwKNywiYXVkIjoiIiwidG9rZW5JRCI6MTM0OCwidGVuYW50SUQiOiJjODRhZWUzNC04ODhjLTRmMDYtYmFmMy00NGZkMmNhM2JkM2UifQ.mUu5ZrsfjIWipzf5zqOpA4PUqWbu_CSWaSEmPC3wygv4
For the examples below, this secret is saved in a file called
/home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
Alternative Step 2 - token as environment variable instead of secrets file
In some cases it may be more suitable to provide the security token as an environment variable when launching the agent. If using this method, define the environment variables AGENT_ID and the AGENT_AUTHENTICATION_TOKEN set to the token generated in step 1, example:
--env AGENT_ID=ae311d23-5ca7-fake-8317-25example54a \
--env AGENT_AUTHENTICATION_TOKEN=eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhY2NlZGlhbi5jb20iLCJzdWIiOiJhcGl1c2VyQG55ZGVsbC5zZSIsImV4cCI6MTYyAksIOwKNywiYXVkIjoiIiwidG9rZW5JRCI6MTM0OCwidGVuYW50SUQiOiJjODRhZWUzNC04ODhjLTRmMDYtYmFmMy00NGZkMmNhM2JkM2UifQ.mUu5ZrsfjIWipzf5zqOpA4PUqWbu_CSWaSEmPC3wygv4
If using this method, note that the AGENT_ID also needs to be set through environment variable, since it cannot be fetched from the secrets file when not using a secrets file.
If a secrets.yaml file exists and is mounted it will have precedence over the environment variables.
Step 3 - Launch an agent
Below are examples of two different methods to launch an actuate agent. The throughput, transfer and trace agents are launched in the exact same way, with the exception of the TWAMP reflection port which is not needed for the throughput agent.
Both launch methods described below will open UDP for TWAMP reflection on port 4000 (internal is by default 862), connect towards a Sensor Collector on 10.11.12.13:55777/tcp and define the secrets file as /home/agentuser/3dfe3705-42f3-fake-80bb-59a218a0900e.yaml
The internal UDP port for TWAMP reflection can be changed through the agent management API and Analytics GUI, if the internal UDP port is changed, then the docker mapping to an external port may need to be changed to match the new setting.
It is considered good practice to use the agent UUID as part of the file name for the secret so that if you deploy several agents on the same host they get their own secrets file.
See Agent secrets file options for other metods of providing the authentication secret to the agent
Step 3.a Launch agent using Docker-Compose
To use Docker-Compose, the configuration parameters should be put in a YAML file. AGENT_ID and access token are fetched from the secrets file defined so they need not be defined explicitly in the docker-compose file.
Note that the NET_ADMIN capability should be set, without this enabled the agent may not operate as expected.
Example for actuate agent:
version: '3'
services:
twamp-sender-service:
container_name: "my-actuate-agent"
image: "gcr.io/sky-agents/agent-actuate-amd64:r24.09"
cap_add:
- NET_ADMIN
- NET_RAW
ports:
- "4000:862/udp"
restart: "always"
environment:
AGENT_MANAGEMENT_PROXY: "10.11.12.13"
AGENT_MANAGEMENT_PROXY_PORT: "55777"
volumes:
- type: bind
source: /home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
target: /run/secrets/secrets.yaml
- Use "bind" instead of direct -v mapping of files/directories
- Set a restart policy like in the example above docs.docker.com
- By default Docker logs have no log rotations. If Assurance Sensor Agents run for long periods of time please change the default log driver to not risk file system filling up. Please see the Docker documentation on logging:
Configure logging drivers - Capbilities:
- NET_ADMIN - always recommended to enable
- NET_RAW - recommended for Actuate, Trace and Throughput agents
- SYS_ADMIN - required when using namespaces/VRF
Example for throughput agent:
version: '3'
services:
tcp-agent-service:
container_name: "my-throughput-agent"
image: "gcr.io/sky-agents/agent-throughput-amd64:r24.09"
cap_add:
- NET_ADMIN
- NET_RAW
restart: "always"
environment:
AGENT_MANAGEMENT_PROXY: "10.11.12.13"
AGENT_MANAGEMENT_PROXY_PORT: "55777"
volumes:
- type: bind
source: /home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
target: /run/secrets/secrets.yaml
The agent can now be launched using the docker-compose up command:
sudo docker-compose up -d
If the agent configuration was placed in a file named something other than docker-compose.yml, then this needs to be specified on the command line:
sudo docker-compose -f mycomposefile.yml up -d
IMPORTANT:
Depending on your specific version of docker-compose, you might need to run:
sudo docker compose up
So if you have cut and paste the code using the hyphen and it doesn't work, try without the hyphen.
Step 3.b Launch agent on command line
Same setup, but using command line with environment variable options (--env) and port maps (-p)
Example for actuate agent:
sudo docker run -d -p 4000:862/udp \
--env AGENT_MANAGEMENT_PROXY="10.11.12.13" \
--env AGENT_MANAGEMENT_PROXY_PORT="55777" \
--cap-add=NET_ADMIN \
--cap-add=NET_RAW \
--mount type=bind,source=/home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml,target=/run/secrets/secrets.yaml \
gcr.io/sky-agents/agent-actuate-amd64:r24.09
The -d option makes Docker launch the container in the background, in "detached" mode.
Step 3.c Launch agent without secrets file
Same setup, but using command line with environment variable options (--env) and port maps (-p)
Example for actuate agent:
sudo docker run -d -p 4000:862/udp \
--env AGENT_ID="ae311d23-5ca7-fake-8317-25example54a"
--env AGENT_AUTHENTICATION_TOKEN="eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhY2NlZGlhbi5jb20iLCJzdWIiOiJhcGl1c2VyQG55ZGVsbC5zZSIsImV4cCI6MTYyAksIOwKNywiYXVkIjoiIiwidG9rZW5JRCI6MTM0OCwidGVuYW50SUQiOiJjODRhZWUzNC04ODhjLTRmMDYtYmFmMy00NGZkMmNhM2JkM2UifQ.mUu5ZrsfjIWipzf5zqOpA4PUqWbu_CSWaSEmPC3wygv4"
--env AGENT_MANAGEMENT_PROXY="10.11.12.13" \
--env AGENT_MANAGEMENT_PROXY_PORT="55777" \
--cap-add=NET_ADMIN \
--cap-add=NET_RAW \
gcr.io/sky-agents/agent-actuate-amd64:r24.09
The -d option makes Docker launch the container in the background, in "detached" mode.
Note: Tokens expire after 14 days, so if using this method you need to generate a new token after that period of time if you need to restart or set up a new agent. Started and running agents re-negotiate token well before the expiry happens, so as long as you dont destroy the container you will still be able to stop and start agents. docker-compose down destroys the container so use graceful stop instead.
Step 4 - Check that the agent is running
To check that the agent has started, use the docker ps command:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
437d53381d83 agent-actuate-docker:yy.mm_nnn "/sbin/monitor" 5 weeks ago Up 5 weeks (healthy) 0.0.0.0:4000->862/udp, my-agent
The output shows for how long the agent has been running, and the UDP port mapping informs that port 4000 is mapped on the host IP for TWAMP reflection.
Using agents with multiple interfaces / VRF / namespaces
All agents support attaching to multiple interfaces by configuration in the docker compose file or via docker command line startup options. The agent can either be provided with a specific --mount mountpoint for a VRF / namespace or be provided with the full set of namespaces available in the /var/netns directory.
To be able to switch namespace, the agent needs to be started with the SYS_ADMIN privilege flag enabled.
Difference between interfaces and namespaces
Linux systems natively support using multiple physical and virtual (VLAN) interfaces, typically named "eth0", "ens4" or something like "eth3.1111" for a VLAN residing on port eth3.
On a standard linux system these interfaces cannot have overlapping IP addressing subnets and routes, which means that usually each interface has its own IP subnet, and possibly a set of static routes pointing out destination networks that are reachable through a router on this subnet. There is also only one default route available on the system.
With the addition of namespaces / VRF the linux kernel provides the ability to separate the routing function from the system-global and apply a separate routing domain for each namespace / vrf. With this separation you can have several interfaces defined, with the same IP address, and with different default routes.
Note that when an interface is reassigned to a different namespace than the default one, it will typically disappear from the interface list seen with "ip addr" command.
Example commands to add a namespace, and associate eth3.123 with this namespace
ip netns add mynamespace
ip link set eth3.123 netns mynamespace
Example docker compose file that gives the agent access to the namespace directory on the system (last volumes bind)
version: '3'
services:
agent_actuate:
container_name: "mycontainername"
image: "gcr.io/sky-agents/agent-actuate-amd64:r24.09"
cap_add:
- NET_ADMIN
- NET_RAW
- SYS_ADMIN
restart: always
network_mode: host
ports:
- "862:862/udp"
environment:
AGENT_MANAGEMENT_PROXY: "12.13.14.15"
volumes:
- type: bind
source: /home/agentuser/ae311d23-5ca7-fake-8317-25example54a.yaml
target: /run/secrets/secrets.yaml
- type: bind
propagation: rslave
source: /var/run/netns
target: /var/run/netns
- SYS_ADMIN capabilities are required when binding to namespace
- NET_ADMIN and NET_RAW are recommended for agent actuate
- propagation mode "rslave" allows the agent to see new entries in the /var/run/netns - without this flag the agent cannot attach to namespaces that are created after the agent has started
Agent Launch Parameters
When launching an agent it accepts a number of parameters that define the instatiation of the container. The mandatory ones, like AGENT_MANAGEMENT_PROXY, have been described in the previous sections. Below is the full list of supported parameters, that can be provided as environment variables through the container instantiation when starting the agent.
Configuration | Comment |
---|---|
AGENT_ID | Optional, UUID for the agent. Is fetched from secret file if not provided |
AGENT_MANAGEMENT_PROXY | Mandatory, points to IP address or FQDN of the Agent Orchestration |
AGENT_MANAGEMENT_PROXY_PORT | Optional, defaults to 55777 |
AGENT_SOURCE_INTERFACE | Optional , Defaults to eth0 inside the container |
AGENT_MEASUREMENT_NETNS | Optional, defines which namespace (VRF) to attach to for measurement, this can also be set through the management API and Analytics GUI on a per-test-session basis |
AGENT_MANAGEMENT_NAMESPACE | Optional, This parameter defines which namespace the agent should use for management (connection to Sensor Collector) |
AGENT_CA_TRUST_BUNDLE | Optional, see /v1/docs/agent-install-guidelines#ca-trust-bundle-selection for all options |
AGENT_REFLECTORS_DEFAULT_STATE | Optional, agent: actuate only, set to "true" if an unmanaged agent should start up with TWAMP and UDP reflectors enabled - note that if the agent gets managed by the orchestration system, any reflector settings in the orchestration will override this environment variable setting |
AGENT_AUTHENTICATION_TOKEN | Optional, set to value of your authenticationToken if not using a secrets file |
AGENT_SECRETS_PATH | Defines where the agent looks for its authentication secrets inside the container, the default path is “/run/secrets/secrets.yaml” |
The agent is now ready to be configured using the orchestration API, see Working with the agents control API for more information on using the API, or agent orchestration API reference for full details.
© 2024 Cisco and/or its affiliates. All rights reserved.
For more information about trademarks, please visit: Cisco trademarks
For more information about legal terms, please visit: Cisco legal terms
For legal information about Accedian Skylight products, please visit: Accedian legal terms and tradmarks