Agent Installation Tips and Tricks

This article provides information on how to troubleshoot connection issues between sensor: agent and the orchestration.

Overview

The most common issues seen when deplying agents comes from authentication and communication between agent and the gateways in roadrunner, typically due to firewalls blocking traffic or mistyped/misconfigured IP addresses or host names.

It is important that you always run the latest versions of all components to ensure proper operation. Agents do support backwards compatibility, but since each agent release brings fixes and enhancements to operation it is imperative to keep agent revision in line with roadrunner and analytics versions. See release notes for current version compatibility.

Use "telnet" to Check Basic Connectivity

The "telnet" command which is included in most linux distributions provide a quick and easy way to see if the host where you intend to deploy and agent can reach to the IP address and port where the roadrunner management proxy is located.

If the connection is successful, all is good:

accedian@myhost:~$ telnet 10.19.129.173 55777
Trying 10.19.129.173...
Connected to 10.19.129.173.
Escape character is '^]'.

If it fails, there might be a firewall blocking the communications:

accedian@myhost:~$ telnet 10.19.129.173 55880
Trying 10.19.129.173...
telnet: Unable to connect to remote host: Connection refused

Using the agentStatus Command for Debugging Installation

Starting with release 22.04 agents are shipped with a command-line tool called agentStatus that can be used to check current status for the agent. To run the tool simply call docker exec with the below parameter set

sudo docker exec -it 7e75c0742d37 /usr/bin/agentStatus help

The help text shown by executing above command shows which agentStatus
commands that are available in the agent. Depending on agent type there may be more or less available troubleshooting options, but three sections are always present - "data", "management" and "system".

Data section

data
        status     Show status of data reporting status.
        output     Show current data output settings.
        output log (enable|disable)    Enable or disable data output in the container log.
        output mqtt (enable|disable)   Enable or disable data output to the internal MQTT broker.
                                            Topic is "debug/performancedata/openmetric".

        debug                     Show debug status.
        debug (enable|disable)    Enable or disable debugging.
        debug <value>             Set debug level (engineering).
        debug lws <LWS_LEVEL>     Set websocket debug level (engineering).
                LWS_LEVEL: any combination of: err, info, warn, notice, debug, parser, header, ext, client, latency, default

Management section

management
        authentication  Show current authentication token information.
        configuration   Show current active configuration received from Sensor Orchestrate.
        status          Show current connection status to the management interface.

        debug                     Show debug status.
        debug (enable|disable)    Enable or disable debugging.
        debug <value>             Set debug level (engineering).
        debug lws <LWS_LEVEL>     Set websocket debug level (engineering).
                LWS_LEVEL: any combination of: err, info, warn, notice, debug, parser, header, ext, client, latency, default

System section

system
        status          Show system status.

Usage Examples of agentStatus Command

Example 1 - check authentication status

$ sudo docker exec -it 7e75c4322d37 /usr/bin/agentStatus management authentication
{
        "TenantId":     "9ce9d687-27e3-fake-fake-068275b6b66d",
        "TokenId":      "223647",
        "Issuer":       "accedian.com",
        "Subject":      "apiusername@yourdomain.com",
        "Audience":     "",
        "ExpirationDate":       "Mon 28 March 2022 05:27:53 UTC",
        "Expiration TS":        1648445273
}

As can be seen in the example above, this agents' token has expired, and needs to be renewed by restarting the agent with a fresh secrets file.

Example 2 - check management status

$ sudo docker exec -it 7e75c0742d37 /usr/bin/agentStatus management status
{
        "connection":   {
                "AuthenticationError":  "",
                "LastConfigUpdate":     "Fri 06 May 2022 13:14:18 UTC",
                "LastConnectionTime":   "Fri 06 May 2022 13:14:11 UTC",
                "LastDisconnectionTime":        "Never",
                "LastAuthenticationTime":       "Fri 06 May 2022 13:14:18 UTC",
                "ssl":  {
                        "Enabled":      true,
                        "AllowSelfSigned":      true,
                        "AllowInsecureCerts":   false,
                        "SkipHostnameCheck":    true,
                        "AllowExpiredCerts":    false
                }
        },
        "ManagementState":      "Operational",
        "TenantId":     "ce578dd3-03fa-4d28-baaa-47f03db4264c",
        "ConfigId":     "6c206e9a-e387-4e4b-9bdc-d9d3e76d3406",
        "ConfigRevision":       "9873a947-5d2a-4725-902b-59cc0cc7598b",
        "ManagementServer":     "123.123.1.456",
        "ManagementPort":       55777

This example indicates that this agent is connected to orchestration and has received a configuration.

Example 3 - show data (metrics) connection status

$ sudo docker exec -it 7e75c0742d37 /usr/bin/agentStatus data status
{
        "connection":   {
                "DataBrokerServer":     "123.123.1.456",
                "DataBrokerServerPort": 55888,
                "AuthenticationError":  "",
                "LastTransmissionTime": "Fri 06 May 2022 13:19:13 UTC",
                "LastTransmissionTimeAck":      "Fri 06 May 2022 13:19:14 UTC",
                "LastConnectionTime":   "Fri 06 May 2022 13:14:19 UTC",
                "LastDisconnectionTime":        "Never",
                "LastAuthenticationTime":       "Fri 06 May 2022 13:14:20 UTC",
                "ssl":  {
                        "Enabled":      true,
                        "AllowSelfSigned":      true,
                        "AllowInsecureCerts":   true,
                        "SkipHostnameCheck":    true,
                        "AllowExpiredCerts":    true
                }
        },
        "MeasurementState":     "Operational",
        "DataBrokerQueueDepth": 3600,
        "RingBatchWaiting":     0
}

This example shows that the agent data reporting path through roadrunner is operational. The "LastTransmissionTime" parameter should update periodically to reflect that new data is being sent.