Using the Performance Assurance Agent

Overview

This article describes how to use the Performance Assurance Agent (PAA™) for monitoring network performance.

The Performance Assurance Agent (PAA) is a hardware-assisted active-measurement function that computes Layer-2 or Layer-3 (IPv4 UDP) network delay (latency), delay variation (jitter), frame/packet loss, and continuity checks.

Two units with matching settings communicate with each other using measurement samples to collect data and measure system performance. A measurement sample is a frame/packet containing timing and sequence information. When such a frame/packet arrives at its destination, measurements (delay, delay variation and frame/packet loss) can be taken. The PAA operates continuously.

The PAA is capable of concurrently testing and maintaining multiple flows of active probes. The characteristics of the test frames/packets for each flow are set to match the Layer-2 (PCP value of the VLAN tag) or Layer-3 (IP ToS/DSCP value) characteristics of the services being monitored. These frames/packets may be independently directed to different peers or to the same peer, using a different class of service (CoS) and/or VLAN ID as appropriate. Up to two VLAN tags can be specified (.1Q-in.1Q).

The PAA can be configured in a point-to-point or point-to-multi-point fashion. In other words, you can enable a single PAA instance to exchange data with another PAA instance, or enable it to communicate with several PAA instances simultaneously. Typically, the PAA instance would be configured between the network-side port of the first unit and the network-side port of the second unit (point-to-point) in order to get live latency, jitter and frame/packet loss measurements. It is also possible to select a client-side port on a given unit for PAA measurements when it is directly connected to another unit, or through a network device in the same point-to-point fashion.

Concepts and Definitions

The Performance Assurance Agent performs measurements on network delay, delay variation, frame loss, and continuity. Before configuring PAA, it is important to understand basic concepts and definitions.

The sampling interval determines how frequently PAA frames/packets are sent between end-points. In other words, the sampling interval determines the granularity of the measurements. Configuring PAA with a smaller interval may seem more advantageous at first glance, but it also requires more system resources, which ultimately limits the maximum number of PAA instances that can be configured in the unit. To ensure PAA operates within the system capacity, a built-in resource manager keeps track of the resources that are used as PAA instances are added, and informs the user if the maximum capacity is reached.

The reference period determines how frequently performance measurements are computed and reported. For example, in the case of delay measurements, the minimum and maximum values are reported and the average value is calculated based on all samples received during the reference period.

Thresholds can be configured to trigger alarms and inform the user or a network monitoring system should any key performance index level be crossed.

Clock synchronization (NTP, PTP, or GPS) is mandatory for PAA one-way delay measurements. All units must be synchronized to perform one-way delay measurements. If synchronization is not enabled, PAA will only display two-way delay measurements.

Setting Up a Probe

To configure a PAA probe

1. Access the page SOAM ▶ PAA ▶ Configuration.
   A summary of all PAA probes is displayed.

2. Click Add to create a new probe or click the probe name to edit an existing probe.

3. Complete the required fields, then click Apply.

For more information on specific parameters, refer to the following tables.

PAA Configuration (SOAM ▶PAA ▶Configuration)
General

Local and Peer Indexes

Layer-2 Parameters (Layer-2 Probes Only)

Layer-2 Parameter Validation (Layer-2 Probes Only)

EVC Fault Propagation (Layer-2 Probes Only)

UDP Parameters (Probes with UDP over IPv4 or IPv6 Only)

UDP and UDP IPv6 Parameter Validation (Probes with UDP over IPv4 and over IPv6 Only)

Continuity

One-Way

Two-Way

Deleting a Probe

To delete a PAA probe

1. Access the page SOAM ▶ PAA ▶ Configuration.

2. Click the name of the probe you want to delete.

3. Click Delete.

Viewing Probe Status

To view the status of all PAA probes

1. Access the page SOAM ▶ PAA ▶ Status.

2. Click a probe name to view its detailed information.

For more information on specific parameters, refer to the following table.

PAA Status (SOAM ▶ PAA ▶ Status)

Viewing Probe Results

To view a summary of all PAA probe results

1. Access the page SOAM ▶ PAA ▶ Results.

2. Click a probe name to view detailed results of a probe.

For more information on specific parameters, refer to the following table.

PAA Results (SOAM ▶ PAA ▶ Results)

Packet Loss

One-Way Delay

One-Way Delay Variation

Two-Way Delay

Two-Way Delay Variation

IGMP Join Delay

IGMP Leave Delay

VLAN 1

VLAN 2 (Layer-2 Probes Only)

DSCP (Probes with UDP-over-IPv4 Only)

Configuration Examples

• Example 1: Configuring PAA as point-to-multipoint at layer 2
• Example 2: Configuring PAA without auto-discovery mode
• Example 3: Configuring layer-2 parameter validation
• Example 4: Configuring PAA as point-to-point at layer 3
• Example 5: Configuring multicast PAA at layer 2
• Example 6: Configuring multicast PAA at layer 3
• Example 7: Configuring PAA to measure IGMP Join/Leave delay

Example 1: Configuring PAA as point-to-multipoint at layer 2

This example shows how to configure PAA between a central unit (Assurance Sensor 1, formerly Skylight element 1) and two remote units (Assurance Sensor 2 and Assurance Sensor 3).

Physical Setup:

Configuration Procedure:

1. Configure two PAA instances in Assurance Sensor 1.

2. Configure a PAA instance in Assurance Sensor 2.

3. Configure a PAA instance in Assurance Sensor 3.

4. Validate the PAA status.

5. View the PAA results.

Configure two PAA instances in Assurance Sensor 1

1. Connect to the central unit (Assurance Sensor 1).

2. Access the page SOAM ▶ PAA ▶ Configuration and click Add.

3. Create a PAA instance (e.g. PAA1): enter all parameters, as shown in the image below, and click Apply. This instance will be associated with Assurance Sensor 2.

Note that Local index, Peer index and Destination MAC address are left at their default value. This puts the unit in auto-discovery mode, where it will automatically find the peer MAC address and associate with the far-end unit based on the PAA name (the PAA name must be the same in both units).

4. Repeat the last two steps to create a second PAA instance (PAA2) on the same port for VLAN 102. This instance will be associated with Assurance Sensor 3.
   

Configure a PAA instance in Assurance Sensor 2

1. Connect to Assurance Sensor 2.

2. Access the page SOAM ▶ PAA ▶ Configuration and click Add.

3. Create a PAA instance (PAA1): enter all parameters, as shown in the image below, and click Apply. This instance will be associated with the instance named PAA1 in Assurance Sensor 1.
   

Before going any further, it is important to further examine the PAA status.

4. Access the page SOAM ▶ PAA ▶ Status
   The PAA instances you just created are in the Associating state. This is normal as the peer instances have not been created yet and PAA is trying to discover them. As soon as the peer instances are active, the state will change to "Associated."
   

Configure a PAA instance in Assurance Sensor 3

1. Connect to Assurance Sensor 3.

2. Access the page SOAM ▶ PAA ▶ Configuration and then click Add.

3. Create a PAA instance (PAA2): enter all parameters, as shown in the image below, and click Apply. This instance will be associated with the instance named PAA2 in Assurance Sensor 1.
   

Validate the PAA status

1. Connect to the central unit (Assurance Sensor 1).

2. Access the page SOAM ▶ PAA ▶ Status.
   If all has been configured correctly, the PAA instances should be associated and functioning as expected, as shown in the following figure:
   

3. Click index 1 to see more details on the PAA alarms and status:
   

View the PAA results

1. Connect to the central unit (Assurance Sensor 1).

2. Access the page SOAM ▶ PAA ▶ Results.
   This menu shows the main results in a table format for all PAA instances:
   

3. If needed, click a specific index number (e.g. “1”) to view detailed PAA results.

Example 2: Configuring PAA without auto-discovery mode

This is a variant of Example 1 where, instead of using the auto-discovery mode and letting the PAA instances associate by their name, you will specify the index and peer MAC address for each instance. In this mode, you can use different PAA names at each endpoint if desired.

Configuration Procedure:

1. Gather the MAC address information from all units.

2. Configure two PAA instances in Assurance Sensor 1.

3. Configure a PAA instance in Assurance Sensor 2.

4. Configure a PAA instance in Assurance Sensor 3.

Gather the MAC address information from all units

1. Connect to Assurance Sensor 1 and access the page Port ▶ Configuration.
   

2. Write down the MAC address of the port that is associated with the PAA instance.

3. Repeat these steps for Assurance Sensor 2 and Assurance Sensor 3.

Configure two PAA instances in Assurance Sensor 1
Best Practice: before configuring the PAA instances, plan how the index numbers and other parameters will be assigned.

PAA Configuration Parameters Summary for Example 2

1. Connect to Assurance Sensor 1.

2. Access the page SOAM ▶ PAA ▶ Configuration and then click Add.

3. Using the parameters from the table above, configure the first PAA instance as follows:
   

4. Then configure the second PAA instance as follows:
   

5. Access the page SOAM ▶ PAA ▶ Status.
   Unlike what you saw in the previous example, you will notice that the PAA instances are already in the "Associated" state although the peer end has not yet been configured. This is normal because the PAA instances are associated by their index and MAC address. In this mode, a PAA instance is always in the Associated state.
   

Configure a PAA instance in Assurance Sensor 2

1. Connect to Assurance Sensor 2.

2. Access the page SOAM ▶ PAA ▶ Configuration and then click Add.

3. Using the parameters from the table above, configure the first PAA instance as follows:

Configure a PAA instance in Assurance Sensor 3

1. Connect to Assurance Sensor 3.

2. Access the page SOAM ▶ PAA ▶ Configuration and then click Add.
   

Configure a PAA instance in Assurance Sensor 3

1. Connect to Assurance Sensor 3.

2. Access the page SOAM ▶ PAA ▶ Configuration and then click Add.

3. Using the parameters from the table above, configure the first PAA instance as follows:
   

Example 3: Configuring the validation of layer-2 parameters

This example describes how PAA can be configured to validate that the layer-2 parameters seen by the peer unit are as expected. For example, if you set up two units to exchange PAA frames on VLAN 101 with priority (PCP) 5, you can validate that the correct priority is received at the far end.

Physical Setup:

Layer-2 PAA with priority validation

Configuration Procedure:

1. Configure a PAA instance in Assurance Sensor 1 and Assurance Sensor 2.

2. Configure the PAA instance in Assurance Sensor 1 to validate the far-end priority.

3. Validate the operation.

4. Force a priority change.

5. View the results.

Configure a PAA instance in Assurance Sensor 1 and Assurance Sensor 2

1. Connect to Assurance Sensor 1.

2. Refer to Example 1 and configure the PAA instance in Assurance Sensor 1 as described in Step 1, but set the value of VLAN 1 priority to 5.

3. Connect to Assurance Sensor 2.

4. Refer to Example 1 and configure the PAA instance in Assurance Sensor 2 as described in Step 2, but set the value of VLAN 1 priority to 5.

Configure the PAA instance in Assurance Sensor 1 to validate the far-end priority

1. Connect to Assurance Sensor 1.

2. Access the page SOAM ▶ PAA ▶ Configuration and click the probe name PAA1.

3. Scroll down to the Layer-2 Parameters Validation section and put a check mark in the following checkboxes:

• Validate Tx/Rx VLAN 1 ID
• Couple Tx/Rx VLAN 1 ID
• Validate Tx/Rx VLAN 1 priority
• Couple Tx/Rx VLAN 1 priority

4. Click Apply to save the changes. The screen should be as follows:
   

Validate the operation

1. Access the page SOAM ▶ PAA ▶ Results in Assurance Sensor 1 and click index 1.

2. Scroll down to the VLAN 1 section. The screen should be as follows, which indicates everything is normal; the peer has not detected any mismatch:
   

Force a priority change

1. Insert a device between Assurance Sensor 1 and Assurance Sensor 2 to change the priority of the tagged frames sent to Assurance Sensor 2. For example, change the PCP value from 5 to 3. This can be achieved using a Assurance Sensor or any device that can remark the VLAN priority.

View the results

1. Access the page SOAM ▶ PAA ▶ Results in Assurance Sensor 1 and click index 1.

2. Scroll down to the VLAN 1 section. The screen should show the following information, which now indicates a priority mismatch has been detected:
   

The peer was expecting VLAN 101/PCP 5 frames but received VLAN 101/PCP 3 instead. A priority mismatch was detected on 17 PAA frames (so far).

3. Access the page Show ▶ Alarm.

4. Set the filter to Description, set the value to PAA, then click Search.
   The display is limited to PAA-related alarms only. As shown in the following screen, the PAA priority mismatch was also reported by an alarm:
   

It is important to note that this change of priority did not affect the other performance measurements being performed by this PAA. Consequently, this change of priority in the network could have gone unnoticed if you had not used this option to detect it.

Example 4: Configuring PAA as point-to-point at layer 3

This example shows how to configure PAA between two units (Assurance Sensor 1 and Assurance Sensor 2) in a layer-3 network. Note that this configuration can also be used in a point-to-multipoint topology (as shown in Example 1).

Physical Setup:

Layer-3 PAA in point-to-point topology

Setting up PAA at layer 3 requires the following:

• An IP address configured on the interfaces used by PAA.
• A route to forward the PAA packets out through the proper interface. If the peer IP is in the same sub-network, this step is not required as the route is automatically added when the IP interface is created. If the peer IP is in a different sub-network, a specific route needs to be defined.

Configuration Procedure:

1. Configure IP interfaces in both units.

2. Configure a PAA instance in Assurance Sensor 1.

3. Configure a PAA instance in Assurance Sensor 2.

Configure IP interfaces in both units

1. Connect to Assurance Sensor 1.

2. Access the page System ▶ Configuration ▶ Interface and click Add.

3. Create an interface on Port 1 with IP address 10.10.10.11/24, as shown in the following screen, and click Apply.
   

4. Do the same in Assurance Sensor 2 and create an interface on Port 9 with IP 10.10.10.12.

Configure a PAA instance in Assurance Sensor 1

1. Connect to Assurance Sensor 1.

2. Access the page SOAM ▶ PAA ▶ Configuration and click Add.

3. Create a PAA instance (e.g., PAA1): enter all parameters, as shown in the image below, and click Apply.
   

Configure a PAA instance in Assurance Sensor 2

1. Connect to Assurance Sensor 2.

2. Access the page SOAM ▶ PAA ▶ Configuration and then click Add.

3. Configure a PAA instance as shown in the previous steps, but with peer IP address 10.10.10.11, and click Apply.

Example 5: Configuring multicast PAA at layer 2

This example has one unit as the PAA source and all the other units as sinks. The source unit will generate PAA multicast frames; the sink units will receive these frames and calculate the results. All results will be displayed on the sink units. There will be no results on the source unit.

Physical Setup:

Layer-2 Multicast PAA

Configuration procedure:

1. Configure PAA in the source unit.

2. Configure a traffic policy for multicast PAA in the sink units.

3. Configure PAA in the sink units.

4. View the results.

Configure PAA in the source unit (Assurance Sensor 1)

1. Connect to Assurance Sensor 1.

2. Access the page SOAM ▶ PAA ▶ Configuration and click Add.

3. Create a PAA instance (e.g. L2_Multicast_Source): enter all parameters as shown in the figure below, and click Apply. Take note of the Local and Peer Index numbers: you will use these values when configuring the sink units. The destination MAC address must be a multicast address, i.e. a MAC address that starts with "01".
   

Configure a traffic policy for multicast PAA in the sink units

1. Connect to a sink unit (e.g., Assurance Sensor 2).

2. Access the page Traffic ▶ Filters ▶ L2 filters and click Add to configure a layer-2 filter. Configure the fields as in the following screen and click Add:
   

3. Access the page Traffic ▶ Policies and open a policy on Port 9 (e.g., policy 9-1). Configure the fields as in the following screen and click Apply:
   

Configure PAA in the sink units

1. Connect to a sink unit (e.g., Assurance Sensor 2), access the page SOAM ▶ PAA ▶ Configuration and click Add.

2. Create a PAA instance (e.g., L2_Multicast_Sink): enter all parameters as shown in the figure below, and click Apply. Take note that the Local and Peer Index numbers are reversed to match the source unit configuration. The Destination MAC address must be the same multicast MAC address that was configured in the source unit.
   

3. Repeat these steps for any other sink unit.

View the results

1. Connect to a sink unit (e.g., Assurance Sensor 2).

2. Access the page SOAM ▶ PAA ▶ Results and click the PAA instance index 2.
   

Example 6: Configuring multicast PAA at layer 3

This example has one unit as the PAA source and all the other units as sinks. The source unit will generate PAA multicast packets; the sink units will receive these frames and calculate the results. All results will be displayed on the sink units. There will be no results on the source Assurance Sensor.

Setting up multicast PAA at layer 3 requires the following:

• A multicast IP address (the valid IPv4 range is 224.0.0.0 to 239.255.255.255).
• An IP address configured on the source unit interface used by PAA.
• A route to forward the PAA packets out through the proper interface on the source unit.
• A traffic policy in the sink units to send the PAA multicast packets to the CPU.

Physical Setup:

Layer-3 Multicast PAA

Configuration Procedure:

1. Configure an IP interface in the source unit.

2. Configure a route in the source unit.

3. Configure PAA in the source unit.

4. Configure a traffic policy for multicast PAA in the sink units.

5. Configure PAA in the sink units.

6. View the results.

Configure an IP interface in the source unit (Assurance Sensor 1)

1. Connect to Assurance Sensor 1.

2. Access the page System ▶ Configuration ▶ Interface and click Add.

3. Create an interface on Port 1 with IP address 10.10.10.11/24 as shown in the following screen and click Apply.
   

Configure a route in the source unit (Assurance Sensor 1)

1. Access the page System ▶ Configuration ▶ Interface, scroll down to the IPv4 routes section and click Add. Configure the route as follows:

Configure PAA in the source unit (Assurance Sensor 1)

1. Access the page SOAM ▶ PAA ▶ Configuration and click Add.

2. Create a PAA instance (e.g., L3_Multicast_Source): enter all parameters as shown in the figure below, and click Apply. Take note of the Local and Peer Index numbers: you will use these values when configuring the sink units. The destination IP address must be a multicast address.
   

Configure a traffic policy for multicast PAA in the sink units

1. Connect to a sink unit (e.g., Assurance Sensor 2).
2. Access the page Traffic ▶ Filters ▶ IPv4 filters and click Add to configure an IP filter. Configure the fields as in the following screen and click Add:
   

3. Access the page Traffic ▶ Policies and open a policy on Port 9 (e.g., policy 9-1). Configure the fields as in the following screen and click Add:
   

Configure PAA in the sink units

1. Connect to a sink unit (e.g., Assurance Sensor 2), access the page SOAM ▶ PAA ▶ Configuration and click Add.

2. Create a PAA instance (e.g., L3_Multicast_Sink): enter all parameters as shown in the figure below, and click Apply. Take note that the Local and Peer Index numbers are reversed. The Destination IP address must be the same multicast IP address that was configured in the source unit.
   

PAA should now be functional in this sink unit. Repeat the last two steps on any other sink units that need to be configured.

View the results

1. Access the page SOAM ▶ PAA ▶ Results in Assurance Sensor 2 and click the PAA index 2.
   

Frequently Asked Questions

One-way delay value is larger than two-way delay

Question: PAA results report one-way delay values that are larger than the two-way delay. Furthermore, the peer unit reports negative one-way delay values. How is this possible?

Explanation: This is usually caused by a clock synchronization issue. When NTP or PTP synchronization is enabled in the unit, it usually takes some time for the local clock to converge and align on the reference clock. PAA displays one-way delay results when synchronization is enabled, but this does not mean that the clock has converged yet. The same process takes place in the peer unit. To achieve accurate one-way delay measurements, both units must be synchronized. If there is an offset between the clocks, it will be reflected in the time stamps that are used to measure the delay.

The following illustrates the effect of a clock offset on one-way delay measurements:

If Clock 2 is 10 ms ahead of Clock 1, unit 2 will timestamp with an offset of +10 ms, and vice versa in unit 1. If the actual one-way delay is 1 ms, unit 2 will measure a delay of 11 ms while unit 1 will measure a delay of -9 ms.

Solution: Units equipped with the GPS option offer the best possible synchronization method and perform well with one-way delay measurements. Otherwise, when synchronizing via NTP or PTP, allow enough time for the clocks to converge. The quality of the synchronization path also plays an important role in the synchronization process. Generally speaking, when the network delay is much larger than the clock offset between units, the impact on the one-way delay measurements will not be significant.

PAA period sometimes reports a few more, or fewer, samples than expected

Question: I sometimes have PAA periods that report a few more, or fewer, samples than expected. For example, I configured PAA with a reference period of 1 minute and a sampling rate of 100 ms. I expect all periods to register 600 samples, but I sometimes see 599 or 601 samples.

Explanation: This can be caused by delay variation in the network. For example, a sudden increase in the delay can cause a PAA frame to reach destination just after a period was closed. The "late" frame is still valid, but it will be registered in the next period. This is a common phenomenon and it does not affect the validity of the performance measurements.

The first PAA period is longer or shorter than configured

Question: I configure all my PAA instances with a reference period of 1 minute and a sampling rate of 1 second. All periods show 60 samples, as I expected. However, I noticed that just after I enabled a PAA instance, the first period is different: it can be either shorter (e.g., 43 samples) or longer (e.g., 64 samples). How can this variation be explained?

Explanation: When a new PAA instance is created, the system ensures that its period is synchronized so that all PAA instances configured with the same period duration will start their measurements at the same time and report consistent measurements.

For example, all 1-minute periods are synchronized to start at xxh:00m:00s, xxh:01m:00s, xxh:02m:00s, etc. All 5-minute periods are synchronized to start at xxh:00m:00s, xxh:05m:00s, xxh:10m:00s, etc.

Note that enabling time synchronization (NTP, PTP or GPS) on a unit while PAA instances are active may trigger resynchronization of the periods.