Delay: 1 ISP, 1 good month, 1 bad night

Sprint was the only provider to install the Global Positioning System receivers necessary for precise measurements of one-way delay – the time it takes for each packet to move across Sprint’s network. For many network designers, delay is even more important than throughput in predicting application performance.

Sprint was the only provider to install the Global Positioning System receivers necessary for precise measurements of one-way delay – the time it takes for each packet to move across Sprint’s network. For many network designers, delay is even more important than throughput in predicting application performance.

Sprint’s delay measurements were generally low and consistent, suggesting Sprint’s network won’t have any adverse ramifications on application performance. However, we did observe one 30-minute period in which something in the provider’s New York point of presence went seriously wrong. During this one bad patch — the only one during the entire four-week test – delays approached the 1-second mark.

That kind of delay is large enough to have serious ill effects on application performance. For TCP traffic, such high delay can lead to retransmissions and loss of TCP sessions, as cited by several public studies (For more, click here).

Initially, we asked all ISPs to install GPS receivers at all sites so we could measure one-way delay. GPS is necessary to ensure precise synchronization between sender and receiver. Absent synchronization, measurements can be so far off that it’s possible to record “negative delay,” where the receiver’s timestamp will indicate that it gets a packet before the transmitter has sent it.

All carriers except Sprint balked at the GPS idea, with most citing cost and logistical difficulties of GPS installation. Many ISP POPs are located in basements, far from the view of the sky, which a GPS receiver requires. Further, not all ISPs own the buildings that house their POPs. Getting permission to put up an antenna can be difficult.

As a fallback plan, we offered to measure other ISPs’ delay by synchronizing transmitters and receivers using the network time protocol (NTP). Unfortunately, this method didn’t work out. The issue that killed NTP as our back-up plan was jitter — variations in timing — between the clock sources and the cNode measurement instruments.

Despite our best efforts to reduce jitter (including use of multiple NTP sources), we couldn’t reduce it below our goal of 2 msec. Rather than use imprecise clocks, we simply dropped one-way delay measurements for all providers not using GPS.

Informed that other providers wouldn’t be using GPS, Sprint still requested we go ahead with one-delay numbers for its network. The company said it already had committed to use GPS, and had installed receivers specifically for this project.

Generally, Sprint’s delay numbers were quite low, with a minimum of 8.71 msec and an average of 18.77 msec across all measurements. Further, the standard deviation of latency for all city pairs was 7.74 msec, indicating relatively little variance from the average across different city pairs or traffic types (TCP and User Datagram Protocol [UDP]).

In nearly all cases, the one-way delays we recorded were only a bit higher than the speed-of-light propagation delays for the distances involved.

As good as Sprint’s overall delay numbers are, they didn’t prevent something from going awry at its New York POP on Aug. 27. Starting around 10:15 p.m. EST and lasting until almost 10:45 p.m., delay measurements climbed a 100fold for streams headed from New York to Chicago or Fort Worth, Texas. Maximum delays to San Jose did not increase, and traffic headed to New York showed no sign of trouble.

At the same time, the cNodes recorded several other signs of trouble. In the 5-minute period just before delay climbed through the roof, jitter rose markedly. Once the high delay began, Sprint’s routers also dropped packets on the New York-sourced streams.

In the worst case, we recorded maximum delay of 924 msec – or nearly 1 second – on a UDP stream from New York to Fort Worth. That’s the kind of delay normally associated with NASA ground control talks with its astronauts, not the sort of thing found on a leading ISP’s backbone.

Fortunately, the anomaly subsided after half an hour and delay measurement returned to normal levels.

Sprint says its internal monitoring systems did not observe record-high delay on Aug. 27. The company said it saw a “queuing event” – that is, some buildup in its routers’ buffers – but said even this was equivalent to at most 100 msec of delay, not the 900-plus msec we observed.

It’s possible the difference is attributable to different measurement methods. The cNodes send continuous traffic and take measurements continuously, while many network management systems use sampling techniques or spot checks that could have missed the high delay.