One-Way Sync: Provider Connectivity Assurance Remote Clock Detection

Introduction

Importance of One-Way KPIs

Keeping track of one-way Key Performance Indicators (KPIs) such as packet loss, jitter, delay and delay variation is important in today’s complex networks because changes or deviations that affect the path in only one direction will not be seen in their full extent with a roundtrip method.

Consider a temporary delay rise from 20 to 60 milliseconds in a roundtrip measurement. There is no way of knowing if this delay bump was caused by a dramatic increase from 10 to 50 in one direction, or a more modest increase from 10 to 30 on both paths.

An application that relies on one-way KPIs, such as Voice over IP or backhaul synchronization, will be impacted differently depending on if
the packet loss or delay variations are concentrated to one direction of the path as opposed to being shared over the two directions.

How To Measure One-Way Delay

To be able to measure delay in only one direction for a transmitted measurement packet, there is a need to have a distributed and common concept of time at the sending and receiving end. To achieve this there are several methods available:

• Use GPS as external clock source at both ends
• Use a packet-based sync from a common time server (NTP & PTP)
• Use one end as NTP or PTP server for the other end

Figure 1. GPS Common Time Source

Since both ends in a GPS setup agree on the same time, it is easy for the receiving side to simply subtract the transmit timestamp from the receive timestamp to calculate the one-way latency.

The Cisco PCA Remote Clock Detection method is reminiscent of the packet-based sync methods and can, in a simplified way, be explained as a “High-Resolution” NTP or PTP sync method.

The next section provides an overview of these packet-based sync methods, allowing the reader to better understand Cisco PCA Remote Clock Detection.

NTP and PTP Sync Methods

Differences & Similarities Between NTP and PTP

Both NTP and PTP standards are based on packet exchanges between a Client and a Server. Fundamentally, not much differs between the two protocols; the Client periodically sends UDP packets towards the Server to fetch the current date and time.

The difference lies in how frequently these packets are sent as well as that PTP packets can be more transparently forwarded in the network devices along the sync path (if all intermediate network devices support PTP). With a generally higher frequency of packet exchange, PTP is considered more accurate than NTP due to the larger number of samples to select from when adjusting the clock.

A High-resolution NTP, like the one present in Cisco NID and Node devices, provides an even higher packet rate than PTP and is therefore considered to have comparable accuracy to PTP if used on a non-sync-aware network infrastructure. If the end-to-end network instead has full support for PTP (PTP-TC / PTP-BC), PTP will provide superior sync accuracy over the Cisco high-resolution method.

Figure 2. Packet Based Common Time Source

In-Line Packet Based Sync

The same method can also be used in-line if one of the PM devices functions as an NTP or PTP server. In this scenario, the sync packets are typically placed in the highest CoS class available to achieve the best result. Conceptually, it doesn’t matter which side syncs to the other, as long as they use the same time, the delay can be calculated at the receiving end by subtracting the transmit and receive timestamps.

Figure 3. Packet Based In-line sync

All packet-based sync methods are based on one assumption: the forward and reverse path between client and server are symmetric in terms of minimum latency. For example, if there is a fixed asymmetry caused by having 1Gbps links in one direction and 10Mbps links in the other, the received time-of-day at the client will carry a static offset error. This error is equal to the difference in minimum latency between the two, divided by two.

As another example, the minimum delay for a 128-byte packet (the maximum size for NTP) to traverse a 1 Gbps link is approximately 1 microsecond, while on a 100 Mbps link, the delay is about 10 times longer, around 10 µs.

NTP / PTP offset error in this case would be 4.5 µs ((10 – 1) / 2).

Adding into the equation a router that adds 1ms symmetric overhead causes no change to the absolute offset error, so from a performance monitoring and latency measurement standpoint it lowers the error percentage.

If the fixed asymmetry is known, it can be accounted for when interpreting the measurement results. It is important to note that only the baseline or static offset is impacted by the asymmetry. Any fluctuating delay variations during measurement will indicate the actual one-way variations; i.e Delay_max - Delay_min for any given interval will give an accurate sample of how large the delay variation was during the interval, even if there is a static offset due to asymmetry.

Cisco Provider Connectivity Assurance Remote Clock Detection

Overview

The Cisco PCA Remote Clock Detection is a patented clock extraction algorithm similar in operation to the in-line NTP/PTP sync method described in the section titled Packet-Based Sync in Practice. It is different, however, in some important aspects:

• A selected PM session is used to extract the sync information
• Due to high packet frequency (10 pps or more), accuracy is gained
• There is no adjustment of either remote or local clock
• In the Sender, a Virtual Remote Clock is kept for each Responder
• Standard dual One-Way session types are used; RFC5357 TWAMP or ITU-T Y.1731 ETH-DM
  • Dual one-way sessions are PM sessions that allow two timestamps for each direction: T0 and T1 for Sender->Responder direction and T2 & T3 for the return path.
• The Responders are any standard RFC 5357 TWAMP or ITU-T Y-1731 ETH-DM third-party devices that employ hardware-assisted time stamping for PM packets.

Operation

Figure 4. Cisco PCA Remote Clock Detection

If comparing the picture above with Figure 3 one can easily see that the Responder acts like the NTP/PTP server and the Sender has an NTP/PTP client for each Responder it talks to. Since the Sender device may communicate with multiple Responders, each likely having different clocks, it cannot adjust its own time-of-day based on the time observed from a single Responder acting as a "pseudo NTP/PTP server." Instead, a Virtual clock is kept for each unique remote Responder device, and this is used when interpreting the PM session timestamps to present the one-way KPI metrics.

As with all packet-based synchronization methods it will, as mentioned in the section titled Packet-Based Sync in Practice, experience a fixed offset error if the sync line is traversing a network with asymmetric minimum delay values. For end-to-end measurements over multi-hop layer-3 networks where one-way latencies of tens of milliseconds are common, this asymmetry can in many cases be neglected, removing the need to compensate in retrospect. The variations you want to measure, caused by buffering in the network devices, will be accurately presented and are usually in order of magnitudes larger than the comparatively small error caused by the network asymmetry.

Summary

The advantage of the Cisco PCA Remote Clock Detection method is that one Sender device may measure towards hundreds or thousands of Responders without having to provide a common synchronization source between them all. The very large number of samples used compared to standard NTP or PTP also provides additional robustness to the method.

By providing a vendor agnostic in-line sync method that seamlessly interacts with unmodified third-party implementations of RFC 5357 TWAMP, or Y-1731 ETH-DM, the user is given access to a very easy-to-use and powerful one-way network KPI and SLA monitoring tool. With integrated result storage, thresholding, XML data mediation and reporting, the Provider Connectivity Assurance product suite delivers a complete solution for securing end-to-end performance in any modern network.

Sync Quality

Sync Quality Graph

The PCA analytics can present the sync quality graph as a debug tool for the purpose of interop testing towards a TWAMP or ETH-DM responder. This graph is intended for R&D use and does not represent an absolute reading of sync quality in percent. It does, however, provide information on how well a remote third-party responder works in lab scenarios.

For one-way measurements, the accuracy of the Provider Connectivity Assurance TWAMP and ETH-DM are within 50us as long as the sync remains established; this is true for values above 30 in the Sync Reliability graph. If the sync cannot be established, the Provider Connectivity Assurance system will report roundtrip/2 metrics for the specific path. This is also registered in the result records and is visible through the API exports for each record.

Values between 30 and 90 are only usable for Cisco R&D to troubleshoot when interoperating with a third-party reflector in order to find issues relating to timestamping or clock drift at the remote end.

Provider Connectivity Assurance Measurement Accuracy

Components of Measurement Accuracy

There are several factors contributing to the overall absolute accuracy of an active packet-based measurement:

• Measurement device time-stamping accuracy (Provider Connectivity Assurance Sensor Control)
• Responder time-stamping accuracy (third-party or Cisco device)
• Granularity of sampling (number of packets per second used for measurement)

For calculating one-way delay, the accuracy of the synchronization must also be taken into consideration. For a packet-based synchronization method (NTP/PTP/Provider Connectivity Assurance Remote Clock Detection) this is composed of:

• Network minimum delay fixed asymmetry (link capacity asymmetry)
• Network minimum delay variation

Device Dependent Accuracy

The Provider Connectivity Assurance Sensor Control time stamping accuracy is +- 10 µs for the combination of the TX and RX timestamps.

The responder time stamping accuracy depends on the device and implementation; examples are shown below:

• Cisco Assurance Sensor SFP and Module TWAMP sender/responder +- 1 µs
• Ericsson TWAMP responder +- 10 µs
• Nokia TWAMP responder +- 10 µs

The granularity of sampling (packets-per-second) impacts the accuracy of both packet loss and delay measurements, as the network under test is only sampled a limited number of times per second. For packet loss, paths with sporadic small losses will be underestimated because the probability of not seeing the loss condition is greater than the loss percentage it reports when caught. This underestimation becomes more pronounced with lower packet-per-second sampling. The same principle applies to network delay measurements, where brief delay spikes will go unnoticed if they are shorter than the sampling interval.

Network Dependent Accuracy with Remote Clock Detection

Remote Clock Detection, being a packet-based synchronization method like NTP and PTP derives its baseline out of the link capacity contribution that forms minimum roundtrip delay values. The networks capacity asymmetry contributes to the accuracy in the following way, as previously described in the section titled In-Line Packet Based Sync.

Uplink 1Gbps Downlink 100Mbps Sync Error: +- 4.5 µs

Uplink 100Mbps Downlink 10Mbps Sync Error: +- 45 µs

Uplink 20Mbps Downlink 10Mbps Sync Error: +-25.5 µs

This contribution to inaccuracy cannot be influenced and is fixed due to the differing propagation delays when clocking a packet onto links with different capacities.

Provider Connectivity Assurance Accuracy Statement

Taking into consideration:

• Symmetric capacity on up & downlink +- 0 µs
• Good quality responder +-10 µs
• Measurement network min RTT <= 1ms

The Provider Connectivity Assurance solution will provide accuracy of at least 50us, if sync can be established.

Accuracy/error is calculated as follows:

At least 10% of the synchronization messages are transported through the network with a delay variance of less than 10%, the overall measurement error is then less than 5% of the minimum propagation delay (plus the hardware/system error).

For example, in a network with a minimum propagation delay of 1 milisecond, the measurement error would be less than 45 µs (25 + 10 + 10 µs).

The above has been proven true both in good lab conditions, in medium-loaded regional networks and in harsh-condition jittery over-the-internet scenarios.

If the condition of 10% sync messages with less than 10% delay variance cannot be met, the sync quality parameter will indicate a value below 30 in the sync quality graph, and roundtrip/2 values are presented.

Accuracy and Sample-Based Active Performance Monitoring

Above considered, active measurements using synthetic traffic to measure network quality can never give a 100% accurate answer of the delay and packet loss metrics because there will always be gaps when no measurement packet is sent.

However, for long-term network quality surveillance, this method is preferred to other solutions due to its simplicity and ease of implementation compared to passive solutions requiring complex correlation.

To illustrate the gaps of knowledge derived from an active performance monitoring session, consider a TWAMP session (82 byte UDP payload size) configured at 10 packets per second. The actual bitrate and amount of time this TWAMP will be on the link is calculated as:

As for packet loss, at 10pps, a single lost packet will correspond to reporting 100ms of packet loss. However, the actual packet lost was just 1 packet, which takes just 11.84 µs to transmit at Gigabit speed. This does, of course, depend on both the packet size used for the measurements and the link speed at which the measurement occurs. The key point is that an active performance monitoring method cannot provide an absolute truth. To get the absolute correct packet loss rate for a 24-hour period, for example, the only option would be to store all packet data from both ends of the path for the entire duration and then correlate it retrospectively.

All in all, this points out the importance of not only looking at the accuracy of the individual measurement sample but also focusing on the more important long-term trends that come out of packet-based performance monitoring. The exact spike max of a delay bump may never be measured with the comparatively few measurement packets that are injected, so the accuracy of that individual measurement is less important than the overall trend of a large amount of PDV and packet loss measurement results.

For modern IP transport networks, an accuracy of 50 µs for sub-millisecond links is sufficient to extract important network quality KPIs, which are defined in the millisecond range.