✨ New: Try our AI‑powered Search (Ctrl + K) — Read more

Send Logs to Two Servers Using rsyslog

Prev Next

Sensor Management supports sending log data to two central log servers. All logs in the containers' storage path /volume/logs can be sent to the server.

Server Configuration Procedure

This section guides users on how to configure the server that receives log data from Sensor Management.

Ubuntu Operating System

For server configuration using the TCP protocol

Procedure:

  1. SSH to the server via port 22.
ssh <hostname>@<ip> 
  1. Configure the server to receive logs by creating the config file in/etc/rsyslog.d folder. Enter the following command:
sudo nano /etc/rsyslog.d/remote_server_rsyslog_configuration.conf
  1. Add the template to the file for the server configuration using the TCP protocol:
# TCP listener
module(
  load="imtcp"
)
# Start up listener at port 514
input(
  type="imtcp"
  port="514"
)
$template DynamicFile,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
  1. Save the file and exit.
  2. Restart the rsyslog service on the server after completing the configuration by entering the following command:
sudo service rsyslog restart

For server configuration using the TLS protocol

Preconditions:

  1. Ensure the rsyslog-gnutls package is installed by entering the following command:
dpkg - l | grep rsyslog-gnutls

If the packaged is installed, its information will be displayed.

  1. Replace appropriate cert files in the path /ect/rsyslog.d/ of the template server configuration.
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/<ca-cert.pem>"
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/<server-cert.pem>"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/<server-key.pem>"

Where:
<ca-cert.pem> is the Certificate Authority (CA).
<server-cert.pem> is the Server Private Key.
<server-key.pem> is the Server Certificate with CA.

  1. Open the default rsyslog port (514) or a custom port (for example, 10514) specified in the template server configuration. Skip this step if the firewall is disabled.

To check the firewall status:

a. Enter the following command:

sudo ufw status
  • If the output shows Status: inactiveit means the firewall is not enabled, and no further action is required.
  • If the output shows Status: active:

b. Open the port by entering:

sudo ufw allow <port>/udp
sudo ufw allow <port>/tcp

Where: <port> is the port of the server.

c. Ensure the ports are opened by entering:

sudo ufw status

The output should display the following:

Status: active

To                         Action      From
--                         ------      ----
514/udp                    ALLOW       Anywhere
514/tcp                    ALLOW       Anywhere
10514/udp                  ALLOW       Anywhere
10514/tcp                  ALLOW       Anywhere 

Procedure:

  1. SSH to the server via port 22.
ssh <hostname>@<ip> 
  1. Configure the server to receive logs by creating the config file in/etc/rsyslog.d folder. Enter the following command:
sudo nano /etc/rsyslog.d/remote_server_rsyslog_configuration.conf
  1. Add the template to the file for the server configuration using the TLS protocol:
# Set certificate files
global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog.d/<ca-cert.pem>"
  DefaultNetstreamDriverCertFile="/etc/rsyslog.d/<server-cert.pem>"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/<server-key.pem>"
)

# TCP listener
module(
  load="imtcp"
  PermittedPeer=["*.example.com"]
  StreamDriver.AuthMode="x509/name"
  StreamDriver.Mode="1"
  StreamDriver.Name="gtls"
)

# Start up listener at port 10514
input(
  type="imtcp"
  port="10514"
)
$template DynamicFile,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

local1.notice ?DynamicFile
  1. Save the file and exit.

  2. Restart the rsyslog service in the server after finishing the configuration by entering command:

sudo service rsyslog restart

RedHat Operating System

For server configuration using the TCP protocol

Procedure:

  1. SSH to the server via port 22.
ssh <hostname>@<ip>
  1. Configure the server to receive logs by creating the config file in the /etc/rsyslog.d folder. Enter the following command:
sudo nano /etc/rsyslog.d/remote_server_rsyslog_configuration.conf
  1. Add the template to the file for the server configuration using the TCP protocol:
# TCP listener
module(
  load="imtcp"
)

# Start up listener at port 514
input(
  type="imtcp"
  port="514"
)
$template DynamicFile,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

local1.notice ?DynamicFile
  1. Save the file and exit.

  2. Restart the rsyslog service on the server after completing the configuration by entering:

sudo service rsyslog restart

For server configuration using the TLS protocol

Preconditions:

  1. Ensure the rsyslog-gnutls package is installed by entering the following command:
rpm -q rsyslog-gnutls

If the packaged is installed, its information will be displayed.

  1. Replace appropriate cert files in the path /ect/rsyslog.d/ of the template server configuration.
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/<ca-cert.pem>"
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/<server-cert.pem>"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/<server-key.pem>"

Where:
<ca-cert.pem> is the Certificate Authority (CA).
<server-cert.pem> is the Server Private Key.
<server-key.pem> is the Server Certificate with CA.

  1. Open the default rsyslog port (514) or a custom port (for example, 10514) specified in the template server configuration. Skip this step if the firewall is disabled.

To check the firewall status:

a. Enter the following command:

sudo firewall-cmd --state
  • If the output shows not running means the firewall is not enabled, and no further action is required.
  • If the output shows running:

b. Open the port by entering:

sudo firewall-cmd --permanent --add-port=<port>/udp
sudo firewall-cmd --permanent --add-port=<port>/tcp
sudo firewall-cmd --reload

Where: <port> is the port of the server.

c. Ensure the ports are opened by entering:

sudo firewall-cmd --list-ports

The output should display the following:

514/udp 514/tcp 10514/udp 10514/tcp
  1. Disable SELinux by entering command:
sudo setenforce 0
  1. Ensure the SELinux is disabled by entering command:
getenforce

If the output displays Permissive, the SELinux is disabled.

Procedure:

  1. SSH to the server via port 22.
ssh <hostname>@<ip>
  1. Configure the server to receive logs by creating the config file in /etc/rsyslog.d folder. Enter the following command:
sudo nano /etc/rsyslog.d/remote_server_rsyslog_configuration.conf
  1. Add the template to the file for the server configuration using the TLS protocol:
# Set certificate files
global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog.d/<ca-cert.pem>"
  DefaultNetstreamDriverCertFile="/etc/rsyslog.d/<server-cert.pem>"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/<server-key.pem>"
)

# TCP listener
module(
  load="imtcp"
  PermittedPeer=["*.example.com"]
  StreamDriver.AuthMode="x509/name"
  StreamDriver.Mode="1"
  StreamDriver.Name="gtls"
)

# Start up listener at port 10514
input(
  type="imtcp"
  port="10514"
)
$template DynamicFile,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

local1.notice ?DynamicFile
  1. Save the file and exit.

  2. Restart the rsyslog service on the server after completing the configuration by entering the following command:

sudo service rsyslog restart

Sensor Management Configuration Procedure

Configure rsyslog to send logs to Server using TCP

  1. Login to smcli. For access information, see Sensor Management CLI Reference.

  2. Configure the TCP transfer protocol by entering:

syslog edit transfer-protocol TCP
  1. Configure the server's address:
 syslog edit <server> host <x.x.x.x>

Where:
<server> is server1 or server2.
<x.x.x.x> is the IP address.

  1. Configure the server's port:
syslog edit <server> port <x>

Where:
<server> is server1 or server2.
<x> is the port number that must match the port configured on the server and opened on the firewall.

  1. Configure disk storage to store log files during disconnection between Sensor Management and the sever:
syslog edit disk-space <x>

Where:
<x> is the amount of disk space allocated for temporary storage during periods when Sensor Management system is disconnected from the server. The unit is Gibibytes.

Skip this step if users accept the default 5GB allocation.

Note: In practical terms, with the default disk space allocation of 5 GB:

  • You can store approximately 100 log files per disconnected server, with each log file being about 5 MB in size.
  • The remaining allocated space is utilized by the internal design and structure of the rsyslog template to manage these files.
  1. Show the configuration by entering:
syslog show configuration

The output displays:

Skylight: syslog show configuration 

[ Syslog configuration ]

Syslog transfer status : Disabled

Remote server configurations :
    Primary server :
        Host           : 10.220.0.40
        Port           : 514
        Permitted peer : ---
        Disk space     : 5g
    Secondary server :
        Host           : 10.220.0.41
        Port           : 10514
        Permitted peer : ---
        Disk space     : 5g

Transfer protocol : TCP
  1. Enable or disable the log transfer:
    Note: Server configuration must be completed before enabling.
syslog edit remote <x>

Where:
<x> is the enable or disable mode.

To enable sending logs to the server, enter the following command:

syslog edit remote enable

Otherwise, to disable sending logs to the server, enter the following command:

syslog edit remote disable
  1. After completing the configuration to send logs to the remote server, verify the status by entering the following command:
syslog show status

The output displays:

Skylight: syslog show status 
[ Syslog status ]
Remote server status :
    Primary server :
        Host              : 10.220.0.40
        Port              : 514
        Connection status : Up
    Secondary server :
        Host              : 10.220.0.41
        Port              : 10514
        Connection status : Up

Connection status: Up indicates that Sensor Management has successfully connected and sent log files to the server.
Ensure that the server receives log files from Sensor Management in the directory: /var/log/remote/syslog.

Configure rsyslog to send logs to Server using TLS

  1. Login to smcli. For access information, see Sensor Management CLI Reference.

  2. Configure the TLS transfer protocol by entering:

syslog edit transfer-protocol TLS
  1. Configure the server's address:
syslog edit <server> host <x.x.x.x>

Where:
<server> is server1 or server2.
<x.x.x.x> is the IP address.

  1. Configure the server's port:
syslog edit <server> port <x>

Where:
<server> is server1 or server2.
<x> is the port number that must match the port configured on the server and opened on the firewall.

  1. Configure server1's permitted-peer:
syslog edit server1 permitted-peer <x>

Where:
<x> is permitted-peer that matches the configured server.

  1. Import the certificates.
    Note: Certificate files must be stored at /volume/data.

To find the exact location for storing the certificate, follow these steps:

a. Access smcli (skip this step if you are already in the smcli).

b. Enter the config show command.
The output will display: Containers' storage path '/volume' is mounted on the host at /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e.
Based on this, the certificate must be stored in the following folder location: /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data.
For the steps to generate certificates, refer to the official rsyslog documentation: Encrypting Syslog Traffic with TLS (SSL)

c. Import the Certificate Authority (CA).

syslog import ca-file <x>

Where:
<x> is the path to the Certificate Authority file, not including the base path /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data.

Example: If the ca-cert.pem is stored in the /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data/rsyslog_cert folder, the command will be:

syslog import ca-file rsyslog_cert/ca.pem

d. Import the Client Private Key.

syslog import key-file <x>

Where:
<x> is the path to the Client Private Key file, not including the base path /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data.

Example: If the client-key.pem is stored in the /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data/rsyslog_cert folder, the command will be:

syslog import key-file rsyslog_cert/client.key

e. Import the Client Certificate with CA.

syslog import cert-file <x>

Where:
<x> is the path to the Client Certificate file, not including the base path /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data.

Example: If the client-key.pem is stored in the /var/lib/embedded-cluster/openebs-local/pvc-7cb3a9cf-a0ba-4d43-9cf0-703e21dd362e/data/rsyslog_cert folder, the command will be:

syslog import cert-file rsyslog_cert/client.pem
  1. Configure disk storage to store log files during disconnection between Sensor Management and the sever:
syslog edit disk-space <x>

Where:
<x> is the amount of disk space allocated for temporary storage during periods when Sensor Management system is disconnected from the server. The unit is Gibibytes.

Skip this step if users accept the default 5GB allocation.

Note: In practical terms, with the default disk space allocation of 5 GB:

  • You can store approximately 100 log files per disconnected server, with each log file being about 5 MB in size.
  • The remaining allocated space is utilized by the internal design and structure of the rsyslog template to manage these files.
  1. Show the configuration by entering:
syslog show configuration

The output displays:

Skylight: syslog show configuration 

[ Syslog configuration ]

Syslog transfer status : Disabled

Remote server configurations :
    Primary server :
        Host           : 10.220.0.40
        Port           : 514
        Permitted peer : external.example.com
        Disk space     : 5g
    Secondary server :
        Host           : 10.220.0.41
        Port           : 10514
        Permitted peer : internal.example.com
        Disk space     : 5g

Transfer protocol : TLS
  1. Enable or disable the log transfer:
    Note: Server configuration must be completed before enabling.
syslog edit remote <x>

Where:
<x> is the enable or disable mode.

To enable sending logs to the server, enter the following command:

syslog edit remote enable

Otherwise, to disable sending logs to the server, enter the following command:

syslog edit remote disable
  1. After completing the configuration to send logs to the remote server, verify the status by entering the following command:
syslog show status

The output displays:

Skylight: syslog show status 

[ Syslog status ]

Remote server status :
    Primary server :
        Host              : 10.220.0.40
        Port              : 514
        Connection status : Up
    Secondary server :
        Host              : 10.220.0.41
        Port              : 10514
        Connection status : Up

Connection status: Up indicates that Sensor Management has successfully connected and sent log files to the server.
Ensure that the server receives log files from Sensor Management in the directory: /var/log/remote/syslog

Clear rsyslog configuration in Sensor Management

When a user no longer needs to send log files to one or both servers, enter the following command to clear the log sending configurations.

syslog edit <server> none

Where:
<server> is server1 or server2.