Hurricane Katrina literally blew away the telecom infrastructure in New Orleans, but carriers and enterprises are rebuilding and changing what they do.
BellSouth responded quickly, replacing the flooded generators and fiber-optic multiplexers and moving the equipment to higher floors. Digital loop carriers in the field were raised onto platforms 6 to 8 feet above ground level. Some services came back within a few hours, and other functions took weeks, depending on the severity of damage, availability of replacement parts, and other factors, according to CTO Bill Smith.
We toured the Mid-City central office where floodwaters had filled the basement to within a few feet of the ceiling, rendering inoperable the enormous batteries that power the phone system for that area. Ten months later paint crews were just brushing away the high-water mark with a fresh coat of white paint. New batteries, occupying a fraction of the space of the old ones, had been installed upstairs in the three-story building, away from the potential damage of another storm.
BellSouth also has upgraded the switches in the office with compact new technology that leaves vast empty spaces where the older, larger gear once sat.
In all, BellSouth invested nearly $1 billion in a rebuilding effort that included replacing 100,000 copper cable pairs with fiber optics. Except for some heavily damaged neighborhoods where BellSouth is waiting to see whether people are going to move back, the major rebuilding effort has been completed.
For businesses, this new network could mean the availability of advanced telecom services. "That potential certainly exists, because any time you move fiber closer to the end user, you've got more opportunity, whether it's DSL on very short copper loops or potentially fiber-to-the-curb/fiber-to-the-home type of systems," Smith says. For example, Gigabit Ethernet will be more widely available to enterprises in New Orleans as a result of the new fiber, Smith says.
Verizon rebuilds and reroutes
Taking facilities out of New Orleans and away from the coast is an option carriers such as Verizon are embracing. The Verizon Business/MCI backbone suffered four optical fiber cuts when electrical transmission towers collapsed during the storm. Several regeneration facilities also were flooded.
Verizon Business decided to replace those fiber routes with aerial and buried routes north of the coast. The rebuild also prompted the carrier to accelerate its ultralong haul (ULH) project in New Orleans by 12 to 18 months.
ULH uses dense wavelength division multiplexing (DWDM) to deliver a fivefold increase to capacity on those fiber routes, and dual rails of parallel electronics for improved resiliency and latency, says Dick Price, director of business continuance and emergency management for Verizon Business.
ULH also allows for more flexibility in rerouting traffic and reduces the number of regeneration sites, Price says, cutting the number of points of failure and reducing maintenance costs. The SONET architecture that ULH replaces required signal regeneration every 25 miles; ULH requires it every 1,200 miles, according to Price.
"We were able to implement latest and greatest state-of-the-art ULH technology in there a little bit sooner than we had originally anticipated," he says. "We got the new service up and running away from the coast for better protection. At the same time, we were able to improve the overall technology in that area to provide better and more robust services."
AT&T speeds up Gulf region upgrades
AT&T came away from Katrina relatively unscathed. The carrier prepared for the storm by making sure its 162 sites in the Gulf region had plenty of backup power, and that the logistics for fueling and refueling generators was worked out in advance.
As a result, AT&T lost only one regeneration site to flooding, affecting only "a handful" of customers for no longer than 24 hours, says Robin Bienfait, senior vice president of global network operations for AT&T Business Services. The carrier logically rerouted its photonic mesh backbone around the site.
AT&T is continually enhancing the resiliency and reliability of the photonic mesh, Bienfait says, but Katrina and other hurricanes moved up the timetable for enhancing the resiliency of the Gulf region backbones by at least 12 months. "Because of Katrina, we had to look at where the bigger paths are, and Florida gets hit all the time," she says. "So a lot of people have benefited. I am building a stronger backbone through the area."
While no specific new services are being offered, Bienfait says the storm has made customers more aware of disaster-recovery planning and more likely to redesign their networks to take advantage of newer, available services. For example, customers might be more likely now to make the switch from legacy Layer 2 frame relay and ATM services to Layer 3 IP VPNs, Bienfait says.
"When you've had a catastrophe or disaster, it gives you the opportunity to rebuild," she says. "Sometimes the better option is not to rebuild and replace but rebuild and redesign."
Verizon Wireless adds redundancy
Verizon Wireless lost 20 cell sites in downtown New Orleans, but fortunately had relocated its switching facility north of New Orleans, to Covington, La., just before the storm. "That kept us whole on the brain of the cellular network," says Hans Leutenegger, vice president of network operations for Verizon Wireless' South Area. "Preparing ourselves for the storm long before it ever happened was probably our biggest strength."
In addition to relocating the switching center, Verizon Wireless put permanent, redundant generators in its cell sites that were fueled by natural gas. This alleviated the need to access each cell site manually to replenish the generators with diesel fuel, Leutenegger says.
Still, there were gaps to fill after the storm. Verizon Wireless did not anticipate the damage to the local carrier infrastructure and to alternate providers' networks, which it uses to reach its switching facility and the outside world.
"That's our Achilles' heel," Leutenegger says. "We relied on our vendors to provide that service and if they can't keep their network up we're no better off."
Verizon Wireless has now doubled the number of redundant fiber paths coming out of its switch from two to four. The operator also expanded its roster of alternate providers for those paths. "If one of those four paths stays up, we have full connectivity to the outside world without any kind of loss," Leutenegger says.
Verizon Wireless also built two licensed microwave networks - one from the Covington switch out to Baton Rouge, La., the other to Mississippi - for emergency backup backbone service to its cell sites.
In all, Verizon Wireless spent tens of millions of dollars to rebuild its data-optimized EV-DO network, Leutenegger says. Verizon Wireless is just beginning to implement EV-DO Revision A technology, which adds higher data rates and QoS to EV-DO, into its network for service initiative in 2007.
New Orleans will be one of the first cities to get it because of the needs arising from Hurricane Katrina. A survey conducted by Telephia states that monthly mobile use in New Orleans grew 41% between the first quarters of 2005 and 2006 - at least five times higher than the nationwide average increase.
"The usage we saw in New Orleans 30 days after [Katrina] was more than we see [monthly] in New York City," Leutenegger says. "The amount of traffic we saw for the month or two after was absolutely phenomenal," because of EV-DO's broadband data capabilities and its use by Federal Emergency Management Agency, insurance agencies and people replacing DSL lines lost to the storm, he says.
Because of that, Verizon Wireless increased the capacity of its EV-DO network in New Orleans to handle a 50% to 100% increase in traffic, Leutenegger says. And in addition to having EV-DO and making the top of the list for the upgrade to Revision A, New Orleans now has the most redundant wireless network in the country, Leutenegger says.
"It will have the most redundant network," he says. "It's got the building blocks, the foundation, for all future services we can put on there. New Orleans is right up there with anybody as far as technology implemented."
Cingular focuses on resiliency
Following Katrina, Cingular accelerated its timetable for integrating the Cingular network with that of the acquired AT&T Wireless in New Orleans, says Will Schutts, Cingular executive director of network operations for Alabama, Mississippi and Louisiana.This may also have moved up the schedule for upgrading Cingular's network in New Orleans from GSM/GPRS/EDGE to Universal Mobile Telecommunications System/high-speed downlink packet access, which would increase data rates from 135K to as much as 700Kbps, burstable to more than 1Mbps.
But like mother AT&T, which owns 60% of Cingular, the wireless operator is focused more on improving its network resiliency than offering advanced service. Cingular lost an undetermined number of cell sites from flooding, wind and power loss, but service was quickly restored using landline transport, microwave, spread spectrum microwave and satellite facilities, Schutts says.
Cingular built a new switching facility on the north shore of Lake Pontchartrain with redundant power and three switches, and expanded its number of portable generators from 1,200 to 4,500. It also is modifying its disaster-recovery and contingency plans, and installing additional spread spectrum, satellite and backhaul equipment.
"We've considerably expanded the scope of those facilities in light of Katrina," Schutts says, adding that Cingular spent $116 million last year to prepare for and recover from Katrina and other storms. The $165 million Cingular expects to spend on its Gulf region facilities this year will be on projects conceived from lessons learned during Katrina, he says.
Although the money is earmarked and service is up, there's no time for complacency. "I wouldn't say it's business as usual because we're spending a lot of time on modifying our disaster-recovery plans based on what we learned last year," Schutts says. "Also, we're expanding the network. We're seeing a lot of growth in [New Orleans] and surrounding areas right now [with] people working on [recovery] projects. We have a lot of projects going on in that area right now."
About 1.4 million of the 4.7 million lines in Louisiana, Mississippi and Alabama were affected by the storm.
Better than before, but not hurricaneproof
Disaster-recovery modification is first and foremost on Qwest's agenda following Katrina. The carrier lost one regional office in Claremont, Miss., knocking out power on a fiber route that ran beside railroad tracks along the coast.
Katrina displaced that fiber three blocks away but did not cut it, according to Chris Coon, Qwest vice president of network operations. Trailers were brought in to connect the fiber to a new site for power while Qwest rebuilt the destroyed regional office farther north of Claremont Harbor and the coast. In Qwest's case, the carrier is fixing what was broken during Katrina, not upgrading the network for new services, Coon says.
Whether it's in cutting-edge new services or just making repairs, carriers say the telecom infrastructure around New Orleans continues to improve, giving the city possibly the most resilient and redundant network in the country.
"When all is said and done . . . the residents of New Orleans are going to have a better network serving them than what they had before," BellSouth's Smith says. "There will be no question about that. Suffice it to say that the overall network in New Orleans will be a more advanced and more capable network now than it was before the storm.''
"There's always people that ask, 'Can you make your network hurricaneproof?'" Smith says. "The example I use is, if those massive concrete sections of the [Interstate] 10 bridge can just be tossed like matchsticks, it's almost inconceivable to say we can build a facility that is hurricaneproof."
While restoration work was underway, New Orleans relied on a municipal Wi-Fi mesh network as a backup, even for voice. And BellSouth also began offering pre-WiMAX service to small businesses in New Orleans after Katrina hit.
Learn more about this topicTrying to learn from disasters 03/06/06Could you or your outsourcer handle a one-two disaster punch?
09/13/05Katrina tests net service providers