How does data cleaning work?

There's nothing more annoying than getting deep into a troubleshooting session or escalating issues only to find out you were working with bad data. 😫

That's why we developed our data cleaning feature - so that you can control what makes the cut and avoid having it impact your aggregations and analysis work.

What is data cleaning?

When data is ingested, Skylight analytics will run it through a filter to tag any suspicious data as "dirty". This bad data is then excluded from computations and visualizations to avoid skewing the results.

Data cleaning rules apply to directional measurements, meaning we'll only mark measurements for a specific direction of P2R, R2P or RT of a session as "dirty", and leave the measurements for other directions as usable when rule thresholds are hit, etc..

Where do I enable it?

There are two types of data cleaning:

1. Automatic - this is based on error codes from Accedian sources and is always enabled.
2. User defined rules - scrub what and how makes sense to your deployment.

Automatic data cleaning

We will tag data as "dirty" based on error codes in the data files sent from Accedian sources. These rules are not configurable. If the error code reported as part of the statStatus is not 0, the result record will be flagged. This applies to DMM, echo-udp. twamp-sf, twamp-sl and eth-dmm session types.

Refer to Error Codes Documentation for more information on these.

User defined data cleaning rules

Custom cleaning rules can be configurable by admins in the application settings.

You can add a rule for any metric from any object type. The rule format is that when a metric is either ≥ or ≤ a fixed value for a sustained period of time (minimum of 1 minute), then mark all metrics for that data point and for the same direction as "dirty".

Remember - data cleaning occurs at ingestion time meaning that changes to rules will only apply to records being processed from that time on.

How do I know it's working?

These are the icon you will see to determine data cleaning is off or on

On	Off

Example in action