Metrics Collection and Export

Collection and export of performance monitoring data can be done by Skylight orchestrator. The collection portion of the process involves establishing a permanent SSH connection to/from orchestrator and storing the retrieved data points into binary files on the Skylight orchestrator file system.

Once the data is collected, the export mechanism comes into play. The export mechanism is made up of two components:

• CSV Producer: is responsible to periodically scan these binary files and produces CSV files for the newly received data points.
• CSV Exporter: takes these CSV files, and distributes them to northbound systems via SFTP, SFTP/Key, FTP or Rsync protocol.

This article presents these metrics collection mechanisms and the export functions of performance data in Skylight orchestrator.

Metrics Collection

The Skylight sensor technologies can use one of two (and in one case both) mechanisms for data collection. At a high level the mechanisms are identical in nature:

• They establish a permanent SSH connection to Skylight orchestrator and stream performance metric data points on a real time basis.
• Once the data points are received by Skylight orchestrator, they are stored in real-time to the file system in binary storage files.

These two mechanisms are called PM Collect and PT Agent. Under normal operations, they work identically, producing one file per device per measurement type on a periodic basis. Some functional differences emerge under disconnection and recovery scenarios. These are explained below.

The PM Collect Mechanism

The PM collect mechanism is used in the following products:

• TE/CE/NE Skylight element
• GT/GX/LT/LX Skylight element
• VFS and FSX Skylight element
• Skylight sensor: control (Flowmeter, NFV SOAM (DMM, PL, SLM) and Regulator)

This mechanism establishes a real-time data feed from the devices to orchestrator over an SSH tunnel that is initiated by the device. The mechanism can be enabled, and when enabled, it can be configured to send performance data for the following types of measurements:

• PAA, Policy, Regulator, DMM, PL, SA, SA Metrics
• SLM, Flowmeter, TWAMP, Port, Shaper, SFP and System Health metrics

Please refer to the Skylight Orchestrator CSV File Description guide for detailed information of these measurement types.

Configuration of this mechanism can be controlled through Skylight orchestrator via the Collection ▶ Device Configuration tab. For each device, you can enable the mechanism and if enabled, you can specify the measurement types you wish orchestrator to receive and the frequency at which data will be sent.

Some measurement types allow data points to be streamed every second whereas other types such as system metrics are only available at greater intervals. Please review device documentation to understand the possible intervals for each measurement type.

The device configuration page also allows you to define the IP address the device will use to connect to orchestrator. For convenience, Skylight orchestrator can be configured to automatically manage the IP address setting on the devices.

A means to configure these settings in bulk is also provided via the Commissioning application. This application includes a PM step that can be used to configure the devices in bulk via an execution job and/or automatically on discovery of a new device using the Discovery Trigger. Please refer to the Skylight Orchestrator User Guide for additional details on how to use these functions.

On loss of communications to Skylight orchestrator, the device will cache its data in local memory and attempt to re-establish communications. The amount of time a device can retain data is captured under section Skylight Elements Data Retention. Upon reconnection to Skylight orchestrator, devices using this mechanism will prioritize real-time data points over older backlogged data. Backlogged data will be sent in a controlled fashion in between real-time intervals.

The CSV producer will generate CSV files as data points are received and stored into binary storage. This effectively means that under recovery operations, the exported CSV files may produce data in non-chronological order. This is expected behavior and northbound systems must be able to cope with data points that do not arrive in chronological order.

The PT Agent Mechanism

The PT agent mechanism is used in the following products:

• Skylight orchestrator
• Skylight sensor: control (performance sessions data)

This mechanism establishes a real-time data feed from the sensors to orchestrator over an SSH tunnel that is initiated by the orchestrator. Performance data is sent to orchestrator for the following types of measurements:

• TWAMP Stateful, TWAMP Stateless, ETH-DM, ETH-LB, ETH-VS
• ICMP Echo, UDP Echo & 2xOneWay

Please refer to the Skylight Orchestrator CSV File Description for detailed information on these measurement types.

There is no configuration involved in the collection of the performance session data. It is enabled and established as soon as orchestrator establishes connectivity to a sensor that supports this mechanism.

Once data is retrieved, it is stored in binary files in Skylight orchestrator. The binary files are stored in directories. One directory will exist for each performance session and each performance session will have multiple files that are filled and purged based on the retention policy. The default retention policy is of one week, but additional weeks can be configured if sufficient disk space is allotted to the platform. This policy applies to data retention in the binary storage files.

On loss of communications to Skylight orchestrator, the sensor will cache its data in local memory and attempt to re-establish communications. The amount of time a sensor using this mechanism can retain data is captured undersection section Guidelines for Collecting Metrics from Skylight Sensor Control.

Upon reconnection to Skylight orchestrator, sensors using this mechanism will recover data points in chronological order, with the older data being sent first. A 6:1 recovery rate is supported such that a communications loss of 6 hours will be fully recovered in an hour.

Once the data is in the binary files, the CSV producer will generate CSV files if the export of performance session data is enabled. This can be enabled via the Collection ▶ Server Configuration panel via a configurable checkbox. The CSV producer will generate CSV files as data points are received and stored into binary storage. This effectively means that under recovery operations, the exported CSV files may produce data in non-chronological order.

Even though data for a given sensor using this mechanism is sent in chronological order, it is not necessarily true that communications have been lost to all sensors of this type in the network.

Sensors that have not lost communications will have continued to send data to orchestrator and orchestrator will have continued to produce real-time CSV files for those sensors. It is therefore possible for orchestrator to produce CSV files with real-time data for some sensors in the network, and backlogged data for sensors that have just re-established communications after a prolonged outage.

This is expected behavior and northbound systems must be able to cope with data points that do not arrive in chronological order.