Alert Policies

Overview

In Settings ► Policies ► Alerting, set up the conditions that raises or clears alerts for specific metric thresholds and time frames.

Alerts are derived from conditions set against metrics within scoped (filtered) datasets.

Opening the Alerting tab displays a list of alert policies that have been set up, along with information for each alert policy including the name, policy type, and last modified date.

These controls are available:

• Search: Enter keywords to filter alert policies by name
• Include archived: If toggled on, archived alert policies appear in the list
• Enabled: If toggled on, the alert policy is enabled
• : Click then select Archive to archive the alert policy

Setting up an Alert Policy

Session-based alerts can be set on a single metric for each policy and based on categorical metadata filtering.

Capture-based alerts can use application-based or zone-based filtering, and support multiple ingested metrics per policy.

Session-based Alert Policies

Assuming that the system contains data as seen below:

Roughly twice an hour, the data has two peaks. In this example, we will set up an alert policy that raises an alert when values are greater than or equal to 10 ms.

1. Go to Settings ▶ Policies ► Alerting.
2. Click + on the upper right. The Alert policy sidebar appears.
3. In Detection name, enter the alert policy name.
4. (Optional) Click Add description, then enter a description for the alert policy.
5. In Policy type, select Session.
6. (Optional) Reduce the scope of the policy by filtering to a set of objects. Click , select a category, and then click ✓.
   
7. In Conditions, click +, then select the metric to create an alert for. For example, Delay (p95).
   
8. Set up the conditions that raises and clears an alert.
   
9. (Optional) Select the baseline icon to set the alert policy to trigger and clear depending on the difference between the metric value compared to its baseline.

   • Create the baseline alert by provisioning the given fields
     

   • In this example, an alert would be raised when a session’s delay average value is 50 percent above the baseline value for 80 percent of the received data points within a five minute period. The alert would then be cleared once the session’s delay average value is less than 25 percent above the baseline for at least five minutes.

   • Also note that you can also set data-cleaning and time-exclusion toggles for baseline alerts.

10. Control the data to be considered in the calculation that triggers an alert by toggling these fields:
    • Data cleaning
    • Use only busy hours
    • Exclude maintenance windows
11. Click ✓ to save your changes.

Capture-Based Alert Policies

Capture alert policies use Applications or Zones in combination with capture metric conditions.

Applications are provisioned in Inventory ▶ Applications; zones are provisioned in Inventory ▶ Zones.

Zones are typically identified by IP subnet, for example:

• /local with subnet 192.168.0.0/16
• /local/broadcast where 255.255.255.255/32.

Applications are services typically identified by port, for example:

• ssh — uses port 22
• http — port 80 or 8080

Capture Metric Conditions

Capture alert metrics are computed on one-minute intervals using time and bytes per second. The following table can serve as an indicator on how to calculate the value for an alerting policy threshold.

Example: User Experience

Given the User Experience metric aggregated using a sum for the SMTP application and 1-minute granularity.

Based on the above chart, we determine 1.5 s to be a reasonable threshold. An alert policy can be set up as follows:

Example: Client Traffic

Given the Client Traffic metric aggregated using an average for the SMTP application with 1-minute granularity.

Based on the above chart, we determine 2.4 Mbps to be a reasonable threshold. An alert policy can be set up as follows:

Use an Alert Policy in Monitoring or Analysis

Skylight Analytics keeps track of alerts raised, cleared and active alerts as KPIs. To use the alert policy in monitoring or analysis, go to the policy by using the policy name and select an alert metric (typically Alerts Raised or Alerts Active).

Alerts Raised: When a metric value meets the trigger condition of an alert policy, then an alert will be raised on this object. After the alert is raised, this alert is active in the system. In the dashboard, the number of “Alerts Raised” is a historical view of how many alerts were raised in a certain query period.

Alerts Cleared: When a metric value meets the clear condition of an alert policy, then an alert will be cleared on this object. In the dashboard, the number of “Alerts Cleared” is a historical view of how many alerts were cleared in a certain query period.

Active Alerts: After the alert is raised, this alert will be in active state. When the alert is cleared, the active alert will be deleted.

The number of active alerts is the number of alerts active at the end of the query period (interval).

The way this is retrieved differs depending on the type of interval you are looking at.

If you are looking at an interval that is last X time (for example, last 8 hours), then the count is the number of active alerts currently in DB.

But if you are looking at a historical interval, then the number reported is the estimated number of active alerts at the end of the historical interval.

This estimated number is calculated as follows:

historical_active = active_now - raised alerts since end of interval + cleared alerts since end of interval

Update an Alert Policy

If any update was made on an alert policy, current active alerts won’t be reset. It will use new conditions or filters to evaluate the policy for future metrics based on current alert states.

Disable an Alert Policy

You can disable or enable an alert policy on the first column from UI.

If the alert policy is disabled, the system stops to evaluate the disabled policy, the alert state use the last state before disabling. A disabled policy can be enabled again.

Archive an Alert Policy

You can archive unneeded alert policies by clicking ... on the last column and then Archive. Once archived, the policy will not be evaluated for future data and current active alerts will be reset. Archived policies are only kept for historical view. Archived alert policies cannot be reused.

By default, archived alert policies are not displayed, but you can enable the Include archived toggle to include archived alert policies in the view list.