Data Freshness

Cisco Provider Connectivity Assurance (formerly Skylight performance analytics) aims for approximately 5 minutes for data processing latency before results are available for query via the analytics metrics APIs or UI. This is a statistic that we monitor for all of our customers, whether the solution is hosted by us or not - it's part of the managed service, and we make every effort to keep it at a reasonable level.

It is important to understand that the data is being collected in real-time, however, the time from data generation to data availability depends on the amount of data and the duration of the collection interval. This is often perceived as a gap in the data when it is simply the time it takes for Cisco Provider Connectivity Assurance to process the data.

Data Availability

These factors affect data availability in Cisco Provider Connectivity Assurance:

• The time it takes for the sensor orchestration capabilities within Provider Connectivity Assurance (formerly Skylight orchestrator) or collecting agent to generate a CSV file or send data as a batch
• The time it takes for Cisco Provider Connectivity Assurance to process the batches of data or CSV files

You can expect both of the above functions to take a bit of time to process; the processing time depends on the amount of data and the size of the time interval set to collect this data.

Remember that Cisco Provider Connectivity Assurance isn't just storing that incoming PM data as is, it is:

1. Normalizing and transforming

• All incoming data, whether it's from a Cisco Provider Connectivity Assurance Sensor or a third-party source, goes through a normalization process so that similar metrics can be discussed/viewed in a common way (terms and units). Sometimes this an easy as a name change, other times it requires some math to convert to a common unit.

2. Computing custom metrics

• Accedian professional service can help you create custom/derived metrics that get calculated at ingestion time so they can be leverages for alerts or other stream processes.

3. Enriching with metadata

• Provisioned session metadata is added to the data records to faciliate further stream processing, and later aggregations by metadata (i.e. getting the average latency across a region per class of service).
• Check out Understanding the magic of Metadata for more info on metadata.

4. Cleaning

• Default and custom data cleaning rules are applied so that downstream streaming and batch services have the option to work on raw or cleaned data.
• More info here -> How does data cleaning work?

5. Generating alerts

• Alerting for session data happens as a streaming analytics job on the normalized, cleaned, and enriched data
• More info here -> Alerting Overview

6. Loading into a high performance time series datastore

• Data is batch loaded every minute into our query datastore that specializes in time series aggregations

Example for Five-minute Sampling

A sampling frequency of five minutes can cause a gap of 10 minutes. This is because the sensor orchestration capabilities within Provider Connectivity Assurance will need about five minutes to prepare the CSV file and then Cisco Provider Connectivity Assurance will take three to five minutes to process this CSV file.

If data is collected from network probes every five minutes, such as 9:00, 9:05, 9:10, and so on, and you look for results at 9:04, then the 9:00 sample may still be being processed and you will see the last ingested data point (that is, 8:55).

For more information, see Accessing Current and Historical Data.

Using the Time Selector

One cause of big gaps between data generation and data availability is setting the collection interval via the Time Selector at too large a period, for example, seven days. Set this time to a smaller period for better granularity and faster data visualization.

For more information, see Time Selection