Sprint was the only provider to install the Global Positioning System receivers necessary for precise measurements of one-way delay - the time it takes for each packet to move across Sprint\u2019s network. For many network designers, delay is even more important than throughput in predicting application performance.Sprint\u00a0was the only provider to install the Global Positioning System receivers necessary for precise measurements of one-way delay - the time it takes for each packet to move across Sprint\u2019s network. For many network designers, delay is even more important than throughput in predicting application performance.Sprint\u2019s delay measurements were generally low and consistent, suggesting Sprint\u2019s network won\u2019t have any adverse ramifications on application performance. However, we did observe one 30-minute period in which something in the provider\u2019s New York point of presence went seriously wrong. During this one bad patch\u00a0-- the only one during the entire four-week test - delays approached the 1-second mark.That kind of delay is large enough to have serious ill effects on application performance. For TCP traffic, such high delay can lead to retransmissions and loss of TCP sessions, as cited by several public studies (For more,\u00a0click here).Initially, we asked all ISPs to install GPS receivers at all sites so we could measure one-way delay. GPS is necessary to ensure precise synchronization between sender and receiver. Absent synchronization, measurements can be so far off that it\u2019s possible to record "negative delay," where the receiver\u2019s timestamp will indicate that it gets a packet before the transmitter has sent it.All carriers except Sprint balked at the GPS idea, with most citing cost and logistical difficulties of GPS installation. Many ISP POPs are located in basements, far from the view of the sky, which a GPS receiver requires. Further, not all ISPs own the buildings that house their POPs. Getting permission to put up an antenna can be difficult.As a fallback plan, we offered to measure other ISPs\u2019 delay by synchronizing transmitters and receivers using the network time protocol (NTP). Unfortunately, this method didn\u2019t work out. The issue that killed NTP as our back-up plan was jitter\u00a0-- variations in timing\u00a0-- between the clock sources and the cNode measurement instruments.Despite our best efforts to reduce jitter (including use of multiple NTP sources), we couldn\u2019t reduce it below our goal of 2 msec. Rather than use imprecise clocks, we simply dropped one-way delay measurements for all providers not using GPS.Informed that other providers wouldn\u2019t be using GPS, Sprint still requested we go ahead with one-delay numbers for its network. The company said it already had committed to use GPS, and had installed receivers specifically for this project.Generally, Sprint\u2019s delay numbers were quite low, with a minimum of 8.71 msec and an average of 18.77 msec across all measurements. Further, the standard deviation of latency for all city pairs was 7.74 msec, indicating relatively little variance from the average across different city pairs or traffic types (TCP and User Datagram Protocol [UDP]).In nearly all cases, the one-way delays we recorded were only a bit higher than the speed-of-light propagation delays for the distances involved.As good as Sprint\u2019s overall delay numbers are, they didn\u2019t prevent something from going awry at its New York POP on Aug. 27. Starting around 10:15 p.m. EST and lasting until almost 10:45 p.m., delay measurements climbed a 100fold for streams headed from New York to Chicago or Fort Worth, Texas. Maximum delays to San Jose did not increase, and traffic headed to New York showed no sign of trouble.At the same time, the cNodes recorded several other signs of trouble. In the 5-minute period just before delay climbed through the roof, jitter rose markedly. Once the high delay began, Sprint\u2019s routers also dropped packets on the New York-sourced streams.In the worst case, we recorded maximum delay of 924 msec - or nearly 1 second - on a UDP stream from New York to Fort Worth. That\u2019s the kind of delay normally associated with NASA ground control talks with its astronauts, not the sort of thing found on a leading ISP\u2019s backbone.Fortunately, the anomaly subsided after half an hour and delay measurement returned to normal levels.Sprint says its internal monitoring systems did not observe record-high delay on Aug. 27. The company said it saw a \u201cqueuing event\u201d - that is, some buildup in its routers\u2019 buffers - but said even this was equivalent to at most 100 msec of delay, not the 900-plus msec we observed.It\u2019s possible the difference is attributable to different measurement methods. The cNodes send continuous traffic and take measurements continuously, while many network management systems use sampling techniques or spot checks that could have missed the high delay.