Network-upgrade horror story

It sounded like a no-brainer when we started the project in 2003. The idea was to replace our aging, 155Mbps ATM-over-SONET network with a new network based on 10G Ethernet over dense wavelength division multiplexing.

Nobody could have imagined the glitches, snafus and legal holdups that we ran into over the past four years. We finally issued the contract for the project in May, and, needless to say, we learned a bunch of valuable lessons along the way. Here's the whole saga, from the beginning:

During the nine years that the ATM-over-SONET system has been in place, the metropolitan network has grown to 55,000 nodes encompassing two San Francisco campuses, four hospitals and more than 200 remote sites, including regional clinics spread throughout California. The campus network also has evolved into an essential, mission-critical utility, right up there with water and electric power.

Reliability had become a worry, however. Of great concern was the ticking clock: network devices that were at -- or rapidly heading toward -- end-of-life. That means no vendor support for such essentials as software patches, technical support and replacement of failed hardware components. Cisco's support for the Catalyst 5500s and LS-1010s was waning.

In addition, the demands of video distribution, telemedicine and medical-imaging technologies were quickly making the network outdated. It lacked QoS or multicast capabilities. That meant e-mail, Web surfing, video and medical images all got the same "best effort" treatment. Video packets were broadcast indiscriminately, causing bottlenecks and congestion. Applications that needed greater bandwidth or QoS, such as those used for remote clinician consultation and patient diagnosis and medical research, could not be carried efficiently -- or at all -- on the network.

Click to see: Next-generation metropolitan area network topology