Agent Internal Logging

Starting with release 23.07, agents will have container logs. This feature allows the Agent Tech Support Generator to collect logs.

Logs that are sent to the Container Orchestration Engine (COE), to the console, were reduced and simplified.

Local logs are stored in RFC5425 format with some custom structured data.

Log Levels

With this new logging system, levels are used as labels rather that actual levels. This means that it is possible to request only debug and info, for example.

Log Types

There are currently three types of logs: init, status, and default.

Init logs

Init or initialisation logs are usually the startup logs of a process. They are post-fixed with _init.

These logs are preserved and have no rotation.

Status Logs

Status logs are intended for status information that apear periodically. They are on a logs rotation.

Default Logs

All the rest of the logs have log rotation.

Unmatched Logs (/var/log/unmatched)

Logs that do not match another log file will be added to the unmatched log file.

Console Logs

Console logs, grabbed by the Container Orchestration Engine, can be modified via two methods: agent config and agentStatus.

By default, session status messages are no longer displayed on the console to help prevent flooding of logs. They are now in their seperate files. For debug purposes, it is possible to enable the console output.

Agent Configuration Method

Via the agents API, a log section can be added to control the console output.

"log": {
   "console": {
      "enable": true,
      "severity": [
         "warning",
         "error",
         "info",
         "debug"
      ]
   }
},

AgentStatus Method

AgentStatus gives some options to change console behavior. Changes made using this method are not persisted.

log
----
        stats                                       Show log statistics.
        status                                      Show log status.
        console                                     Show console status.
        console enable                              Enable console output.
        console disable                             Disable console output.
        console sessionStatus enable                Enable display of session status on console.
        console sessionStatus disable               Disable display of session status on console.
        console severity <list>                     Set severity to output.
                                                                  List must be one or more space separated:
                                                                  info notice warning error  debug

Examples

Enable/Disable Console

# agentStatus log console disable
# agentStatus log console enable

Enable/Disable Session Status

# agentStatus log console sessionStatus enable
# agentStatus log console sessionStatus disable

Location of Logs

All the agent logs are located in /var/log.

logcat

The logcat tool can be used to format the logs in a human-readable format.

Usage logcat:
        --help                          Display help
        --trunc=<column>      Truncate line. Optional value, default is 200.
        --version                     Display version information

Logging Tricks

Display a Single log

# logcat /var/log/unmatched

Display Unified Logs

# logcat /var/log/* | sort -u

Log Examples

Raw RFC5424 Format

# cat /var/log/apiGateway_sessions
<134>1 2023-07-26T15:02:47.026681+00:00 - apiGateway 36 2770274 [chronicle@22420 ver="1" libver="GO1.0" line="68" function="SensorActuateApiGateway/nb.SendSessionStatus" type="status"] session 3d393ecf-9cab-4a4f-8730-9476dd5990b8 (sid 348) report status running: TxPkts 1158964, RxPkts 1158964, LostPkts round-trip 0, DelayAvgUs far-end 1668 near-end 669
<134>1 2023-07-26T15:02:47.036197+00:00 - apiGateway 36 2770275 [chronicle@22420 ver="1" libver="GO1.0" line="68" function="SensorActuateApiGateway/nb.SendSessionStatus" type="status"] session e4ac2a75-475c-4d74-ae7f-db61ad29a936 (sid 255) report status running: TxPkts 1158974, RxPkts 1158974, LostPkts round-trip 0, DelayAvgUs far-end 1609 near-end 388
<134>1 2023-07-26T15:02:47.036857+00:00 - apiGateway 36 2770276 [chronicle@22420 ver="1" libver="GO1.0" line="68" function="SensorActuateApiGateway/nb.SendSessionStatus" type="status"] session d9b2310b-9211-4272-8e4b-3cd2e6a4bdf7 (sid 337) report status running: TxPkts 1158964, RxPkts 1158964, LostPkts round-trip 0, DelayAvgUs far-end 1855 near-end 601
<134>1 2023-07-26T15:02:47.129433+00:00 - apiGateway 36 2770277 [chronicle@22420 ver="1" libver="GO1.0" line="68" function="SensorActuateApiGateway/nb.SendSessionStatus" type="status"] session 27a3ae15-a97f-4262-9c1b-d34c82c2764e (sid 302) report status running: TxPkts 1158964, RxPkts 1158964, LostPkts round-trip 0, DelayAvgUs far-end 1606 near-end 653
<134>1 2023-07-26T15:02:47.131027+00:00 - apiGateway 36 2770278 [chronicle@22420 ver="1" libver="GO1.0" line="68" function="SensorActuateApiGateway/nb.SendSessionStatus" type="status"] session 169f2803-4338-408d-89fa-360c1dadb82e (sid 251) report status running: TxPkts 1158974, RxPkts 1158974, LostPkts round-trip 0, DelayAvgUs far-end 1634 near-end 488

Using logcat

# logcat /var/log/apiGateway_sessions
2023-07-26T15:03:44.341905+00:00    info       apiGateway          session e320664c-03d2-403b-bb9b-52b915bd1474 (sid 35) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2127 near-end 543
2023-07-26T15:03:44.342255+00:00    info       apiGateway          session 115a0586-f25a-49c6-ae8b-b3cd74beae94 (sid 24) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 1611 near-end 550
2023-07-26T15:03:44.443778+00:00    info       apiGateway          session 96e4ef80-ce95-4a3d-bdc9-49351a08c75d (sid 54) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2213 near-end 621
2023-07-26T15:03:44.444255+00:00    info       apiGateway          session 9b3880f1-5816-45a5-92e8-361112aba45c (sid 37) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2341 near-end 640
2023-07-26T15:03:44.444809+00:00    info       apiGateway          session 0f398566-3219-464d-af21-0aae1cfd8d1d (sid 53) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2230 near-end 650
2023-07-26T15:03:44.446707+00:00    info       apiGateway          session 657bcaad-59da-43a2-b952-d464282eae32 (sid 39) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2313 near-end 621
2023-07-26T15:03:44.450769+00:00    info       apiGateway          session e6a36f24-e126-47a7-bcfc-64d6938fcea0 (sid 41) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 1605 near-end 565
2023-07-26T15:03:44.664999+00:00    info       apiGateway          session f86bd887-b699-413b-b596-6e9c59c3b14a (sid 36) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2132 near-end 560
2023-07-26T15:03:44.727767+00:00    info       apiGateway          session fb21cf6c-75d9-452c-9590-fb9be1dcc436 (sid 51) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 1889 near-end 451
2023-07-26T15:03:44.759379+00:00    info       apiGateway          session 81a50b9d-ffc7-465f-a640-87bc1d894ee4 (sid 64) report status running: TxPkts 1159384, RxPkts 1159384, LostPkts round-trip 0, DelayAvgUs far-end 2128 near-end 548
2023-07-26T15:03:44.759820+00:00    info       apiGateway          session 5b35d713-7ecc-417e-8d84-2ee918514cac (sid 43) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2341 near-end 676
2023-07-26T15:03:44.760246+00:00    info       apiGateway          session c38f989d-88a0-4c92-b955-7e73b860779a (sid 47) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 1813 near-end 516
2023-07-26T15:03:44.767452+00:00    info       apiGateway          session 3c17ac83-85ad-4086-ad01-705b6e864dc6 (sid 120) report status running: TxPkts 1159383, RxPkts 1159383, LostPkts round-trip 0, DelayAvgUs far-end 1715 near-end 443
2023-07-26T15:03:44.903768+00:00    info       apiGateway          session 0bc2c749-038d-4b0d-9bc8-338b096384b1 (sid 61) report status running: TxPkts 1159384, RxPkts 1159384, LostPkts round-trip 0, DelayAvgUs far-end 1817 near-end 543
2023-07-26T15:03:44.904261+00:00    info       apiGateway          session 117dbc2b-f913-4abd-94b6-b0f74ecda967 (sid 97) report status running: TxPkts 1159384, RxPkts 1159384, LostPkts round-trip 0, DelayAvgUs far-end 1900 near-end 454
2023-07-26T15:03:44.904646+00:00    info       apiGateway          session 7be44f05-bc6f-46d9-a338-942a79736907 (sid 40) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2039 near-end 491
2023-07-26T15:03:44.906967+00:00    info       apiGateway          session 812d400e-4eb7-46a2-a4c9-d6aaeedfcfbc (sid 87) report status running: TxPkts 1159384, RxPkts 1159384, LostPkts round-trip 0, DelayAvgUs far-end 2403 near-end 638
2023-07-26T15:03:44.907410+00:00    info       apiGateway          session 0d41cf24-f4a6-47e8-a47a-fa0513100b33 (sid 86) report status running: TxPkts 1159384, RxPkts 1159384, LostPkts round-trip 0, DelayAvgUs far-end 1548 near-end 577
2023-07-26T15:03:44.908974+00:00    info       apiGateway          session cf3c7831-7afa-4724-83fd-6e58fcde5905 (sid 50) report status running: TxPkts 1159385, RxPkts 1159385, LostPkts round-trip 0, DelayAvgUs far-end 2122 near-end 545
2023-07-26T15:03:44.913930+00:00    info       apiGateway          session f988dc52-a2ba-4603-8da4-c530a3c11961 (sid 80) report status running: TxPkts 1159384, RxPkts 1159384, LostPkts round-trip 0, DelayAvgUs far-end 2224 near-end 524