How does data cleaning work?

Our data cleaning feature lets you can control what makes the cut and avoid having bad data impact your aggregations and analysis work.

What is data cleaning?

When data is ingested, Cisco Provider Connectivity Assurance (formerly Skylight performance analytics) will run it through a filter to tag any suspicious data as "dirty". This bad data is then excluded from computations and visualizations to avoid skewing the results.

Data cleaning rules apply to directional measurements, meaning, we'll only mark measurements for a specific direction of P2R, R2P, or RT of a session as "dirty", and leave the measurements for other directions as usable when rule thresholds are hit, and so on.

Where do I enable it?

There are two types of data cleaning:

1. Automatic - this is based on error codes from Cisco Provider Connectivity Assurance sources and is always enabled.
2. User defined rules - scrub what and how makes sense to your deployment.

Automatic data cleaning

We will tag data as "dirty" based on error codes in the data files sent from Cisco Provider Connectivity Assurance sources. These rules are not configurable. If the error code reported as part of the statStatus is not 0, the result record will be flagged. This applies to DMM, echo-udp, twamp-sf, twamp-sl, and eth-dmm session types.

Refer to Error Codes Documentation for more information on these.

User-defined data cleaning rules

Custom cleaning rules can be configured by admins in the application Settings.

You can add a rule for any metric from any object type. The rule format is that when a metric is either ≥ or ≤ a fixed value for a sustained period of time (minimum of 1 minute), then mark all metrics for that data point and for the same direction as "dirty".

Remember that data cleaning occurs at ingestion time meaning that changes to rules will only apply to records being processed from that time on.

How do I know it's working?

These are the icons you will see to determine data cleaning is off or on

On	Off

Example in action