How does data cleaning work?

Updated
  • Updated on Aug 22, 2025
  • Published on Jun 10, 2021
  • 2 minute(s) read
  • Jonathan Poole
  • AC
 Prev Next

The data cleaning feature allows you to control which data is included, preventing bad data from affecting your aggregations and analysis.

What is data cleaning?

When data is ingested, Cisco Provider Connectivity Assurance (formerly Skylight performance analytics) will run it through a filter to tag any suspicious data as "dirty". This bad data is then excluded from computations and visualizations to avoid skewing the results.

Data cleaning rules apply to directional measurements. This means only measurements for the specific direction (P2R, R2P, or RT) in a session are marked "dirty". Measurements for other directions remain usable when rule thresholds are met.

25.07_How does data cleaning work_1.png

Where do I enable it?

There are two types of data cleaning:

  1. Automatic: Based on error codes from Cisco Provider Connectivity Assurance sources and is always enabled.
  2. User-defined rules: Customize data cleaning to fit your deployment needs.

Automatic Data Cleaning

Data is tagged as "dirty" based on error codes in files from Cisco Provider Connectivity Assurance sources. These rules are not configurable. If the error code reported as part of the statStatus is not 0, the result record will be flagged. This applies to DMM, echo-udp, twamp-sf, twamp-sl, and eth-dmm session types.

Name Unit Description
statStatus int Statistic result status (two 16bit field)
+-+-+-+-+-+-+-+-+-+-+-+-+
| statLevel | errorCode |
| lo 16 bit | hi 16 bit |
+-+-+-+-+-+-+-+-+-+-+-+-+
statlevel: valid statistic levels in result.
If statLevel = LOSS|JITTER|DELAY, the result record contain information about loss, jitter and delay.
LOSS = 0x0100 (256)
EST_LOSS = 0x04000 (16384)
JITTER = 0x0200 (512)
DELAY = 0x0400 (1024)
TUNNEL_LOSS = 0x00040 (64)
LOSS|JITTER|DELAY = 0x0700 (1792)
EST_LOSS:
Estimation is done when we have not received any packet to calculate possible gap in sequence number at the time to process the ResultRecord.
TUNNEL_LOSS:
Contain possible packet loss in NFV tunnel (false loss)
errorCode: if code = 0 the result record is OK.
If code > 0 see Error Codes (33xx).

Refer to Error Codes Documentation for more information.

User-Defined Data Cleaning Rules

Administrators can configure custom cleaning rules by nativating to Settings â–¶ Policies â–¶ Data cleaning.
25.07_How does data cleaning work_2.png

You can add a rule for any metric from any object type. The rule format is that when a metric is either ≥ or ≤ a fixed value for a sustained period of time (minimum of 1 minute), all metrics for that data point and direction are marked as "dirty."

Note: Data cleaning occurs at ingestion, so rule changes will only apply to records being processed from that time on.

How do I know it's working?

These icons in the upper-right corner indicate whether data cleaning is enabled or disabled:

ON OFF
Button_Data Cleaning - ON.png Button_Data Cleaning - OFF.png

Note: The Data cleaning icon is not available when in the Settings menu.

© 2025 Cisco and/or its affiliates. All rights reserved.
 
For more information about trademarks, please visit: Cisco trademarks
For more information about legal terms, please visit: Cisco legal terms
For legal information about Accedian Skylight products, please visit: Accedian legal terms and tradmarks