Metrics Export

The metrics export process is divided into the CSV Producer and the CSV Exporter. The article that follows explains the characteristics of each component.

CSV Producer

The producer’s role is very straight forward: it periodically looks for any new data points in the collected performance data binary files and generates CSV files for these data points. The CSV Producer produces files in compressed format.

The frequency at which the producer runs is configured under the Collection ▶ Server Configuration tab. The frequency is determined by the Grouping period configuration setting.

Under normal operations, the CSV Producer runs once per period and produces one CSV file per device and per measurement type. As an example, for a device that has PAA and Regulator measurement types available and enabled, the CSV Producer will generate two files for this device every period. The CSV files are named in the following format:

C123-0024_history_2018-12-11_21h10m25sZ_TWAMP-SL_all_300.csv.gz

Where:

C123-0024 is the serial number of the device

history is a fixed value

2018-12-11_21h10m25sZ is the date in ISO 8601 format. Z indicates that the time value is UTC time (time in Greenwich, England). If the time zone is not indicated, the timestamp is in the local time zone of the Skylight orchestrator server. It is the timestamp of when the file was produced by the CSV Producer process.

TWAMP-SL is the measurement type. Possible types:

• PT Agent types:
• 2xOneWay, Echo-ICMP , Echo-UDP, ETH-DM, ETH-LB, ETH-VS
• TWAMP (stateful), TWAMP-SL (stateless)
• PM Collect types:
• paa, port, policy, regulator, shaper, soam_dmm, soam_pl, soam_sa, flowmeter
• soam_sa_metrics, soal_slm, twamp, sfp, system

all is a fixed value.
300

• For PT Agent types, this is the CSV Producer Period (expressed in seconds).
• For PM Collect types, this is the collected period (expressed in seconds).

The CSV Producer has two features that are available for measurements captured using the PT Agent mechanism. These features are described below.

CSV Producer Rewind

The CSV Producer keeps track of which data points have already been written to file such that every run, it provides only newly received data. For data points collected using the PT Agent mechanism, a means to “rewind” the CSV Producer to a point in time in the past is provided. Using this mechanism allows CSV data to be replayed.

You can rewind CSV export to generate CSV files starting from a date and time of your choosing. Rewinding sets the CSV generation timestamp pointer to the date and time that you specify. CSV files are regenerated starting from that date and time.

For example, if Skylight orchestrator loses connectivity to the export destination, you may want to rewind and re-export data.

To rewind, you use the operate csvexport rewind command which is available in mgr-commands mode. For a detailed description of the command, see the Skylight Appliance Monitor Command Reference. The following procedure explains how to use the command.

It should be noted that the rewind process will result in the production of a substantial amount of CSV files and data. Each CSV file will be limited to 2 MiB in size (compressed) but depending on the retention period configured in orchestrator and the number of performance sessions configured in the system, several Gigabytes of performance data will be produced in a short time period.

This is not a concern for Skylight orchestrator, however, the produced files will be sent to northbound systems. In order to protect these systems, data transmission of replayed data is limited to 50 MiB per minute (compressed).

This is in addition to the normal CSV file pacing of the CSV Exporter. Please refer to the CSV export mechanism for information on how Skylight orchestrator throttles data transmissions to northbound systems. This data flow should be considered before using this operation.

Procedure Requirements

1. Part of this procedure is performed in the Skylight orchestrator console (Appliance Monitor CLI). To access the console, you need to know the credentials of the visionems user.
2. Part of this procedure is performed in mgr-commands mode. To enter this mode, you need a valid Skylight orchestrator user account (an account for the Skylight orchestrator web interface).

Procedure Steps

When you set the time and date for the rewind command, you must consider the data retention period which controls when older data is deleted. If you rewind to a time for which data was already deleted, CSV export will start from the oldest data available. For more information, see Augmenting Performance Session Retention Times.

To create and apply an export template

1. Access the Skylight orchestrator console by opening an SSH session to the appliance IP address, port 2200.

2. Log in as the visionems user.

3. At the Skylight: prompt, enter this command to go into mgr-commands mode:
   mgr-commands

4. At the mgr/mgr# prompt, create an export template by entering this command:
   operate csvexport rewind dateTimeToRewindTo
   where:
   dateTimeToRewindTo is the date and time that you want to rewind to.
   Must be in this format: yyyy-MM-ddTHH:mm:ss
   You will be prompted to confirm that you wish to restart export.

5. To confirm that you wish to continue, enter Y and press Enter.

6. Exit mgr-commands mode by entering:
   quit

CSV Export Template

For data points collected using the PT Agent mechanism, a means to export a subset of the indicators collected (rather than all indicators) is provided. This involves applying an export template to the CSV Producer.

This export template will specify the indicators to include during the CSV file. The following indicators can be specified:

cksum   dStdDev   direction   dmax   dmean   dmin   dp25   dp50   dp75   dp95   dpHi   dpLo   dpMi   duplicatepkts   dvmax   dvmean   dvp25   dvp50   dvp75   dvp95   dvpHi   dvpLo   dvpMi firstpktOffset   firstpktSeq   intervalms   jStdDev   jmax   jmean    jmin    jp25 jp50   jp75   jp95   jpHi   jpLo   jpMi   lastpktOffset   lastpktSeq  lostburstmax   lostburstmin   lostperc   lostperiods   lostpkts   melmax   melmin   misorderpkts   mos   peerId   rxbytes   rxpkts   serialVersionUID   statRound   statStatus   statTime   syncStatus   toolatepkts   tosmax   tosmin   ttlmax   ttlmin   vpriomax   vpriomin

Procedure Requirements

1. Part of this procedure is performed in the Skylight orchestrator console (Appliance Monitor CLI). To access the console, you need to know the credentials of the visionems user.
2. Part of this procedure is performed in mgr-commands mode. To enter this mode, you need a valid Skylight orchestrator user account (an account for the Skylight orchestrator web interface).

Procedure Steps

This procedure involves creating an export template and then applying it to the CSV export job that is in progress. Only one CSV export job exists on Skylight orchestrator. The template will be used for all session types as soon as it is applied.

To create and apply an export template

1. Access the Skylight orchestrator console by opening an SSH session to the appliance IP address, port 2200.

2. Log in as the visionems user.

3. At the Skylight: prompt, enter this command to go into mgr-commands mode:
   mgr-commands

4. At the mgr/mgr# prompt, create an export template by entering this command:
   exporttemplate yourTemplate RR1 metric1 metric2 ...
   where:
   yourTemplate is the name of the template
   RR1 is the name of the result record to which the template will be applied.
   This must be set to RR1.
   metric1 metric 2 are the metrics to be included in CSV export files for all session types.
   You must enter all the metrics that you want to export, separated by a space.

5. Apply the template to the ongoing CSV export job by entering this command:
   csvexport yourTemplate
   The template is immediately applied to the ongoing export job.

6. Check that the template has been applied by entering this command:
   show csvexport
   The status of the CSV export job is displayed. The name of the template that you just applied should be displayed.

7. Exit mgr-commands mode by entering:
   quit

CSV Exporter

Once the CSV files are produced, they can be sent to external systems using an export mechanism. Performance data can be transferred to another system (such as Skylight performance analytics or an OSS) for visualization and analysis.

CSV export of performance data is enabled in the Skylight orchestrator web interface (Collection ▶ Server Configuration page). The CSV export checks every 15 seconds (10 seconds for rsync protocol) for new CSV files and if found, transfers the files to the configured destination. Once CSV files have been transferred to their destination(s), they are removed from the file system.

As mentioned previously, the CSV Producer will always produce files in compressed format. This is to optimize disk I/O and space on the orchestrator platform. The files will be decompressed on transmission if the Use gzip compression option is unchecked in the Collection ▶ Server Configuration page.

Should the CSV export process not be able to transfer its CSV files, it will keep the files in local storage. As this storage is limited, a purge mechanism is in place to protect Skylight orchestrator from disk full situations. The following purge algorithm runs every 20 minutes and applies these rules:

• If available disk space falls below 5 GiB, the oldest CSV files will be deleted until the file system returns to at least 5 GiB of free storage.

Once normal communications to the CSV destinations is resumed, all backlogged CSV files will be sent. In order to protect northbound systems, Skylight orchestrator will throttle file transmission to a maximum of 12,000 files per minute.