Agent: transfer - Configuration

This article provides configuration basics for Skylight sensor agent: transfer.

See https://docs.accedian.io/docs/skylight-roadrunner for information on how to use Analytics UI to create a new roadrunner configuration for agents.

Overview

Skylight sensor: agent transfer implements a TCP and HTTP(s)/FTP client for testing availability and responsiveness of services.

Each agent supports up to 250 test destinations (referred to as sessions) defined, where each session is executed in a sequence based on the test interval parameter speficied in the session definition. For each session, a number of metrics are reported, reflecting the responsiveness of the service and its support functions.

To make use of all 250 session instances with a single agent, care needs to be taken to not use a too short repeat interval – and to keep session timeouts short. If all 250 sessions are configured with 5 seconds max test duration timeout value, then in theory the full execution of all 250 tests could take 5*250 seconds = 1250 seconds or close to 21 minutes. The repeat interval should in this case be set to at least 21 minutes. If a shorter repeat interval is set, there is risk that the last tests in the sequence will never execute, and there will be a "Job queue size exceeded" error seen in the Analytics UI for the session.

If a repeat interval of 60 seconds is required, best practice is to use no more than 15 sessions per agent transfer, and use timeout values of 2 seconds for L4 (TCP connect) and 4 seconds for max total transfer operation duration

                        "maximumTestConnectSec": 2,
                        "maximumTestDurationSec": 4,

To increase the session count above 15, spin up an additional transfer agent on the host, or reduce the repeat interval.

Below is a sequence diagram outlining 6 of the 7 metrics reported by the transfer agent; testTimeNameLookupMs, testTimeConnectMs, testTimeAppConnectMs, testTimePreTransferMs, testTimeStartTransferMs, testTimeTotalMs

The 7th metric, testTimeRedirect is only reporting data if an HTTP redirect occured as a result of the HTTP get call, where the testTimeRedirect metric will represent the time it took to complete the redirected DNS lookup, TCP handshake and SSL handshake phase towards the redirected service URL.
As an example, depending on your HTTP servers settings, configuring a transfer agent to test towards "https://exampledomain.com" may cause a redirect, where the agent ends up querying "https://www.exampledomain.com", resulting in a new DNS lookup for the redirected domain.
Unless you explicitly want to test the redirection function, it is usually best to construct your test URLs for the transfer agent to head directly towards the target URL since this provides a faster completion time of the test phase.

Testing Topology

The agent transfer is compatible with HTTP, HTTPs, FTP and basic TCP-service endpoints such as SSH or telnet. If the targeted service is a non-HTTP/FTP service the agent will only report data up to the TCP handshake phase, i.e the testTimeNameLookupMs and testTimeConnectMs are the only meaningful metrics reported if testing towards for example an SSH port.

Agent Configuration

The query example below shows the GET and response for session configuration of a transfer agent. Three transfer sessions are defined in this example.

GET https://{{tenant-server}}/api/orchestrate/v3/agents/sessions/{{agentId}}

{
            "attributes": {
                "_rev": "2-0054dce38b33566971aa7964997db110",
                "agentId": "1af7f1ea-307b-fake-8bb8-exam1824fd56",
                "session": {
                    "enable": true,
                    "period": "repeat",
                    "sessionId": "50c74d1a-4656-fake-b95d-examdbc6be94",
                    "sessionName": "Agent_https_Youtube_using_DNSQuad9",
                    "sessionType": "transfer",
                    "transfer": {
                        "destinationUrl": "https://www.youtube.com/",
                        "maximumTestConnectSec": 2,
                        "maximumTestDurationSec": 5,
                        "nameServer": "9.9.9.9",
                        "testInterval": 60,
                        "useProxy": false
                    }
                }
            },
            "id": "agentSession_2_1af7f1ea-307b-fake-8bb8-exam1824fd56_50c74d1a-4656-40ad-b95d-9be4dbc6be94",
            "type": "agentSessions"
        },
        {
            "attributes": {
                "_rev": "12-5b1e790d47c23c7bd7e07ecd62cf2096",
                "agentId": "1af7f1ea-307b-fake-8bb8-exam1824fd56",
                "session": {
                    "enable": true,
                    "period": "repeat",
                    "sessionId": "52805b25-0977-fake-9436-exam25f61515",
                    "sessionName": "Agent_GoogleDrive",
                    "sessionType": "transfer",
                    "transfer": {
                        "destinationUrl": "https://drive.google.com/",
                        "maximumTestConnectSec": 5,
                        "maximumTestDurationSec": 5,
                        "testInterval": 60,
                        "useProxy": false
                    }
                }
            },
            "id": "agentSession_2_1af7f1ea-307b-fake-8bb8-exam1824fd56_52805b25-0977-4e1b-9436-da5e25f61515",
            "type": "agentSessions"
        },
        {
            "attributes": {
                "_rev": "14-49c25498dede75339471336ace63191d",
                "agentId": "1af7f1ea-307b-fake-8bb8-exam1824fd56",
                "session": {
                    "enable": true,
                    "period": "repeat",
                    "sessionId": "57e9576b-7a6a-fake-ac88-examf46fbafc",
                    "sessionName": "Agent_SalesForce",
                    "sessionType": "transfer",
                    "transfer": {
                        "destinationUrl": "https://www.salesforce.com/",
                        "maximumTestConnectSec": 5,
                        "maximumTestDurationSec": 5,
                        "testInterval": 60,
                        "useProxy": false
                    }
                }
            },
            "id": "agentSession_2_1af7f1ea-307b-fake-8bb8-exam1824fd56_57e9576b-7a6a-420b-ac88-3cfbf46fbafc",
            "type": "agentSessions"
        }

As seen above, the agent session configuration object contains a revision field _rev that is used for configuration consistency. To update an agent's session configuration, the _rev string sent in the update query needs to match what the _rev is currently set to. That is, to update an agent's configuration, you should first GET the configuration to find the current _rev string. Then, use that string in the PUT query to apply the changes.

Add / delete transfer sessions

The orchestration can store up to 250 test sessions for each agent, where each will be executed in sequence. To add and delete test configurations, use the POST and DELETE API calls

To add a transfer session -

POST to {{tenant-server}}/api/orchestrate/v3/agents/session

With the configuration in the BODY

{
  "data": {
    "type": "agentSessions",
    "attributes": {
      "agentId": "{{agentId}}",
      "session": {
        "sessionId": "{{sessionId}}",
        "sessionName": "my new transfer session to www.google.com 😀",
        "sessionType": "transfer",
        "enable": true,
        "period": "repeat",
        "transfer": {
          "destinationUrl": "https://www.google.com/search?q=accedian",
          "maximumTestConnectSec": 5,
          "maximumTestDurationSec": 5,
          "testInterval": 60,
          "useProxy": false
          }
        }
      }
    }
}

The configuration above is enough to set up a continous test running once every minute, testing connectivity and response time of the google search engine looking up a popular company in the network monitoring and analytics space.

To remove a transfer session -

DELETE on {{tenant-server}}/api/orchestrate/v3/agents/session/{{sessionId}}

Transfer Agent Test Configuration Fields

Session Section

The parameters available for transfer session configuration are:

destinationUrl
required, tells the agent where to send its requests to. Accepted values include; URL with IPv4 address, IPv6 address or FQDN port number and path

username
optional, only used when the asset requested (HTTP/FTP) requires authentication. Accepted value is a string containing the basic auth username

password
optional, only used when the asset requested (HTTP/FTP) requires authentication. Accepted value is a string containing the basic auth password

nameServer
optional string used to query a different DNS server than the system default. Accepted value is an IPv4 or IPv6 address of a name server to use for this session

forceIpVersion
optional string. Accepted values are "IPv4" or "IPv6". This is useful if the DNS server provides both an IPv4 and IPv6 address when looking up the destination URL, and you want to explicly test the IPv4 or IPv6 connectivity

maximumTestConnectSec
optional integer >= 0, default value if not specified is 20. Denotes the timeout in seconds of the measurement connection phase

maximumTestDurationSec
optional integer >= 0, default value if not specified is 180. Denotes the timeout in seconds of the full transfer operation, until requested file/asset is fully retrieved by the transfer agent

testInterval
optional integer >= 1, default value if not specified is 60. Determines repeat frequency of this session.

Using proxy server with transfer agent

To use a proxy server with the transfer agent you can either set the proxy parameters individually on each session in the UI or via API, or provide the proxy settings as environment variables when starting the transfer agent container.
Example using docker compose file:

services:
  transfer-service:
    container_name: demo-transfer
    environment:
      AGENT_MANAGEMENT_PROXY: "13.49.229.173"
      AGENT_MANAGEMENT_PROXY_PORT: 55777
      http_proxy=http://USERNAME:PASSWORD@proxy-server.mycorp.com:3128/
      https_proxy=http://USERNAME:PASSWORD@proxy-server.mycorp.com:3129/
    hostname: demo-transfer
    image: gcr.io/sky-agents/agent-transfer-amd64:r22.11
    volumes:
    - /home/mysecrets/transfer-secret.yaml:/run/secrets/secrets.yaml
version: '3'

Future API calls below for reference:

useProxy
optional boolean. Not yet implemented. Denoting if transfer agent should place its http(s) requests through a proxy server
proxyDestination
optional string. Not yet implemented. Accepted value is an IPv4 address, IPv6 address or FQDN of an http(s) proxy

proxyPort
optional integer [ 1 .. 65535 ] Not yet implemented
Default: 33457
Proxy destination port number

proxyUsername
optional string. Not yet implemented. Accepted value is a string containing the username for the proxy service

proxyPass
optional string. Not yet implemented. Accepted value is a string containing the password for the proxy service

Transfer agent exit / status codes

Common status codes and their meaning

Since the transfer agent is testing towards a real service, there are situations when that service, or the network path, is not behaving as expected. To support finding the cause of transfer agent not reporting any, or just some metrics, there is a "testStatusCode" reported with each test run. Normally the status code is 0 which means everything went as expected. Below is a list of common situations and their status code.

Related Documentation and References

Agent: actuate - Configuration
Agent: throughput - Configuration
Agent: trace- Configuration

Agent API documentation: https://api.analytics.accedian.io/agent-orchestration.html