It's time to lay to rest the notion that ISPs can't deliver telephone-company-level reliability. The fact is, some routed IP network backbones now meet or exceed telco-grade performance. That's the key finding of a groundbreaking study of ISP backbone network performance conducted for Network World by Internet measurement experts Andrew Corlett and Robert Mandeville, along with Network Test, a Network World Global Test Alliance partner.

---

Site-specific performance for each provider
Buyer's Guide: IP VPN services offered the providers (Excel)
How we did it
Not in the game
Delay: 1 ISP, 1 good month, 1 bad night
Choosing an ISP: 10 questions
Special thanks
Archive of Network World reviews

---

 

In a first for public network measurement, the seven participating ISPs placed one of our measurement devices in four locations across their U.S. backbone networks. We turned on the 28 measurement devices and let them run for a month, nonstop, collectively generating an astounding 4.5 billion packets. All told, we collected 156 million discrete measurements. If a network hiccuped for even 10 microsec, we knew about it.

The participants were Cable & Wireless, Level 3 Communications, Qwest, Savvis Communications, Sprint, Verio and WilTel Communications  (formerly Williams Communications). A few other major players opted not to take part, citing various reasons (see “Not in the game").

Here's what we found:

• Two providers - C&W and Savvis - delivered picture-perfect availability. Both had zero downtime, with Savvis running trouble-free for the full monthlong test. C&W also had perfect uptime, but its test window began a few days later than other providers because of a test configuration error on its part.
  
• The networks of four providers - C&W, Level 3, Savvis and WilTel - met or exceeded the vaunted "five nines" standard for network uptime during normal operations.
  
• Sprint's numbers for average delay were the theoretical minimum rates for a beam of light traveling cross-country (see story).
  
• Average jitter for all ISPs was measured in microseconds, well below the point where application performance could suffer.
  
• Packet loss for all providers averaged just 0.01%.
  

It wasn't all good news, though. Two providers - Qwest and Verio - suffered outages that added up to more than an hour each during our monthlong test. Multiple providers asked for extended maintenance periods, meaning that (at least in theory) their networks could be offline while providers work on routers and switches. And while all ISPs put up excellent numbers for average jitter, the maximum jitter for all providers was well into the hundreds of milliseconds. That's high enough to virtually guarantee lousy application performance.

This project was a massive undertaking. Including planning and implementation, it took more than a year to complete. It involved setting up 29 test sites and generating 4,558,388,076 packets during the month of August 2002. We collected 156,050,656 discrete measurements.

One caveat regarding this test is that the results are not a simple guide for picking the best provider. Big as this project was, it measured only core networks. We plan to conduct additional tests focusing on provider edge and customer premises circuits.