Hell or high water

Following the horrific damage wrought by Hurricane Katrina last year, Network World recently sent a team to New Orleans to report on the efforts of service providers, city officials and enterprises to prepare for the next Katrina.

Last August, engineers at Michoud had a pretty important project on their plates. “We had a shuttle flight with some debris falling off of it, and we were analyzing that,” says Steve Stefancik, IS director at the company.

Then Katrina hit. While most employees evacuated, Stefancik kept a 37-man crew on site throughout the storm.

+ MORE KATRINA NEWS: Rewiring New Orleans +

The crew was able to keep the water pumps going until the last minute, and after the storm passed, Michoud was one of the only enterprises in New Orleans that didn’t flood out.

Unfortunately, the roof of the building sustained damage, and 57 of the plant’s 100 servers suffered hard-drive damage. Lockheed Martin had replacement drives delivered to the plant by hitching a ride on a National Guard convoy, and Stefancik was able to bring the internal network up by Sept. 13.

But he still couldn’t get connectivity to other sites around the country. “All of our communications going off-site went through a central switch down the road that flooded, and it took out all our wide-area network stuff,” Stefancik says.

So Lockheed Martin arranged for another NASA site, the Ames Research Lab in California, to fly in via helicopter with satellite-based WAN connectivity. “It worked pretty well,” Stefancik says, but the engineers in New Orleans needed to have some of their work processed at the Marshall Space Flight Center in Huntsville, Ala., and the satellite link didn’t provide enough bandwidth.

Because of the urgency of the project, Lockheed Martin ended up relocating the engineers and the data to Huntsville.

No WAN

Despite all of the disaster-recovery planning that businesses in New Orleans did, no one was quite prepared for the total loss of WAN connectivity.

For example, Phelps Dunbar, a New Orleans law firm, thought it had a solid disaster-recovery plan. The company, which has seven sites in the United States and one in London, had just put the finishing touches on its MPLS network shortly before Katrina struck. “That gave us any-to-any connectivity, allowing us to relocate and restore critical systems in New Orleans to our other offices,” said Chris Rigamer, director of technology.

His disaster plan was to move the company’s primary network and computing assets to its Jackson, Miss., office and recover from there. The Jackson computer room had been built three times larger than necessary so that servers and network equipment could be purchased and brought there if needed, duplicating the services in New Orleans.

And Rigamer specifically went out and added a second Internet link from a different ISP. Unfortunately, both ISPs went down when Katrina hit. “So we were doing a lot of stuff right and by the book, and we still got caught short,”he says.

Forced to improvise, he was able to take the link dedicated to the WestLaw legal research service and repurpose it for general Internet connectivity. “It was a tiny amount of bandwidth compared to what we needed, but it was a link,” he says.

Since then, the company has added another Internet connection through the MPLS network.

Lockheed Martin also now makes sure its WAN links go through two separate exchanges. “We have duplicate OC-3 lines out of here now,” Stefancik says. “If one of our exchanges goes down, the other one may not flood. We take different paths, so they’re not going through the same exchange box.”

Others say multiple carriers or dual pathing wouldn’t have made much difference in the storm’s aftermath. “With Katrina, it wouldn’t have mattered what carrier you were with – everything was down,” says Bret Jacobs, executive director of IT at Loyola University in New Orleans.

“Cell phones, local phone service, data, voice. Even the carriers lost power and ran out of diesel. Our Internet provider was forced to shut down and leave their building, because the building was commandeered. Plus every telephone pole was down, and every manhole and conduit underground was totally flooded. So there was no option really,” he says.

The flooded conduits wreaked havoc on copper wiring, Jacobs says. He converted the T-1 link that fed the campus from copper to fiber. “We didn’t get that until the end of the year,” he says. And he is considering contracting with multiple ISPs in the future. “We are looking at it as an option,” he adds.

No voice

Another major problem was that cell phone service was out, and land-line service was spotty at best, so people couldn’t communicate with each other.

Stefancik, who also is chairman of the St. Tammany Parish council, says flooded towers knocked out cell service. “All cellular towers feed to a landline, and if the land exchange floods, you’re wasting your time trying to work through a cell tower,” he says.

His parish now requires cell towers to support larger fuel capacities, as well as the ability to switch to microwave communications in the event that the AT&T or BellSouth land facilities go down. “Those are new requirements,” he says.

Because most enterprises were forced to evacuate from New Orleans as Katrina approached, many who planned to communicate via cell phones found it difficult to reach their scattered employees. “You could not dial a 985, 228 or 504 area code and get through,” Stefancik says.

Realizing the problem, many organizations bought cell phones from unaffected areas and distributed them to key employees. “We communicated primarily through e-mail, because the cell phones did not work,” Loyola’s Jacobs says. “Even my New Orleans cell phone in Houston didn’t work in Houston. My telecom director evacuated to Tampa, and I had him acquire a case of cell phones with Houston numbers. He overnighted them to me in Houston, and I distributed those to the administration so we could continue to communicate.”

Phelps Dunbar also bought new cell phones. “We outfitted the New Orleans crew that had relocated to Jackson with some cell phones from a Jackson provider in that area code, and those worked,” Rigamer says. “One of our partners just volunteered to go and get us a box of cell phones and get things going. It turned out to be very useful.”

Lockheed Martin knew that cell phone and regular phone service would be questionable, so part of its successful disaster plan was to set up a series of conference calling stations at a sister site in Denver. Each employee was given a Denver phone number and a preset time to call before evacuating.

“So every day during the disaster, our execs call in at 3 p.m. and go through our processes and our status,” Stefancik says. “That way, we can convey to people what needs to happen. Different departments have their own numbers, and they have people set up to call in at certain times. Everybody in this plant, wherever they relocate across the [United States], can also call into their number, and somebody will be on the line with them to give them some understanding of what’s going on and provide direction for the future.”

As a result of Katrina, many enterprises have made satellite phones a part of their future disaster-recovery plans. “We did not have satellite phones then, but we do now,” Rigamer says. “That’s a change we made – just for a couple of key personnel in firm management.”

Jacobs says Loyola kept about seven people on campus during the storm, and they had one satellite phone. “But we’ll probably add a couple more satellite phones for them,” he says, noting that such phones have their limitations. “You basically have to go outside and get a signal, so that could be a little dangerous during the height of the storm.”

No data

The heart of a typical disaster-recovery plan is moving data off-site so you can access it after the disaster is over. But what if you can’t?

For example, the University of New Orleans (UNO) had stored backup tapes in two separate locations: the school’s computer room and the campus security center 5 miles away. Access to the campus and the security center was impossible in the days after the storm. The school was offered space and extra server equipment by a sister school, Louisiana State University (LSU) in Baton Rouge, but getting to its data was problematic.

Eventually, school officials had to use a National Guard convoy to get back on campus and physically recover the tapes. “That’s something we do differently now,” says Jim Burgard, vice chancellor for university computing and communications at UNO. “We now store our tapes with Iron Mountain, which stores them well away from New Orleans and even beyond Baton Rouge.”

Others were luckier. “We preshipped our tapes out of town on Saturday [before the storm], because we had concerns that if it were a big hit, our off-site tape-storage provider might not be able to get the tapes out of town on request,” Loyola’s Jacobs says. “We shipped the tapes early, and now that’s part of our official plan. It was a very good call, because the facility where the tapes were stored survived, but we couldn’t have gotten to it for weeks.”

Tale of the tape

Similarly, the Innovation Group, a firm that consults on the entertainment and hospitality industry, ran into tape troubles. The company has offices in Colorado and New Jersey, and the plan was for employees to fly there and keep working. The company expected to lose use of its data center, but by taking copies of its backup tapes and CDs of its software and software licenses, planned to create a new data center in Colorado, according to Jeremiah Tangen, IT administrator at the firm.

On the Saturday before the storm, Tangen took the backup tapes, which had been refreshed the day before, and tried to evacuate to Jackson, Miss. But Jackson got hit, too, and he decided to drop the tapes off at Kinko’s to be overnighted to Colorado.

Three days later the tapes hadn’t arrived. He tracked them down, dry and safe at the Kinko’s where clerks had failed to send them out before the storm. They sent them that day, and Tangen bought new server hardware to load his applications and data so employees could work remotely using laptops equipped with Cisco VPN gear.

He also hired Iron Mountain to store them outside New Orleans at a site from which they can be delivered to Innovation within four hours. “If we see a storm coming, we can have them ship the tapes to another archive center. It gives us a lot more flexibility,” he says.

The Friday before Katrina struck, he had dropped off tapes at a safe deposit box in a bank just a half mile from Innovation Group’s offices. The tapes he eventually used to build a new data store in Colorado were fresh copies made the following week. “If I hadn’t taken those tapes with me, we wouldn’t have had access to that data for three months,” Tangen says. That’s how long it took before the bank reopened.

New plans, new technologies

Living through a disaster like Katrina has helped enterprises in New Orleans improve their contingency plans. For example, Phelps Dunbar’s Rigamer says it helped make him better aware that disaster recovery is “survival mode” and that getting critical systems back online is where he needs to devote resources.

He’s also fast-tracked a plan to consolidate the servers from his seven domestic sites into one large vendor-operated data center and plans to make good use of server and storage virtualization technologies to take advantage of economies of scale, cut down on space and power needs, and make recovering from a similar disaster far easier.

“None of our offices are quite large enough to put in a storage-area network of any size or blade server technology. Of course, going into consolidation, that’s exactly what we’ll have,” he says. “The only downside is that your dependency on the network is amplified. And with Katrina, that was a problem – everyone was hit.”

To this end, Phelps Dunbar also is working on building a second, redundant WAN. “MPLS will be our WAN A, and we’re going to have a WAN B with as much redundancy as possible,” he says. “We’re trying to engineer this to have as few common elements as possible,” which is difficult, considering the different geographical areas where the sites are situated. He says he’s considering using multiple technologies, including metropolitan Ethernet and DSL.

Lockheed Martin also is looking to address some of the larger citywide infrastructure problems it encountered during Katrina. For example, the company had no electrical power or water services from the city for months. The plant had its own large power generators on-site and had fuel stores to keep them running for as long as 60 days. “We decided to rewire our server rooms and LAN rooms to put them on the generators and get them up and running, and that worked well,” Stefancik says. “We were up well before power was restored.”

Lockheed also dug its own 600-foot well to provide water for its air-conditioning and server room chillers, and that is still used. “Even today, we provide our employees with bottled water for drinking water,” he says. “We have yet to get good clearance on our water here, almost 11 months after the storm.”

Stefancik says another change Lockheed is considering is constructing its own cell tower on the company’s 834-acre campus.

And he says his revamped disaster-recovery plan addresses the need for high-end processing. Employees now are required to take their laptops and the data on them home each night. “And we’ve acquired some high-end workstations that we will now set up at different locations if in fact a hurricane approaches, anywhere in the Gulf,” he says.

Others are considering moving to new network technologies to make them more disasterproof. For example, Burgard says UNO is looking at moving to HP blade servers to reduce the amount of space and wiring needed in LSU’s Baton Rouge data center. Currently, LSU is letting UNO use about four server racks of space, but Burgard is wary of depending on this for the future.

He says the school, which relies on Cisco for most of its network gear, is also considering a move to VoIP. “In the end, our own network was restored before BellSouth’s,” he says. “If we had VoIP, we would have been able to restore our voice communications. It’s something we’re looking into.”

Similarly, Loyola has invested in a hosted solution for a distance-learning network. Before Katrina, the school had a small distance-learning initiative in place, but has since contracted it out to Blackboard to make sure that the service is hosted away from New Orleans and remains up and running.

“We’ll use that to continue instruction, should we have to evacuate again,” Jacobs says. “That’s a change and investment for us. God forbid we have another Katrina, but we’ll probably have evacuations of three or five days. And we want to make sure that the kids can continue to receive instruction and stay connected with the school.”

Cummings is a freelance writer in North Andover, Mass. She can be reached at jocummings@comcast.net.