Three IT organizations share their storm stories.
As Hurricane Katrina swept over New Orleans, Ochsner Hospital stayed open. But it was getting hot inside. With two of three generators down, fans were brought out to cool patients in the 95-plus-degree heat after air conditioning failed. And as temperatures topped 150 degrees inside critical computer and network equipment, Kurt Induni, the hospital’s network services manager, had a call to make.
“We decided after we went down to one generator to shut all critical patient care systems down,” Induni says. “We didn’t want to take more power than we needed. It’s kind of odd to say, but we shut down the patient care systems first to protect them for after the storm was over, when we would need them most.”
To conserve power for critical patient life support and other medical gear, hospital staff shut down the mainframe that runs the patient record system, and a back-up Cisco Catalyst 6509 backbone switch. Better to have a cool spare Catalyst on hand in case the primary one failed, Induni says, instead of letting both switches fry.
Doctors and nurses moved to the back-up method of paper record-keeping during the brunt of the storm when the mainframe was offline.
Also: Volunteers rebuild Gulf Coast communications with wireless nets
Katrina’s IT aftermath
More Wider Net stories
“It was also a gamble strategy to keep the communication lines up,” Induni says. “The 504 area code became useless; e-mail became the only reliable source of communications.”
Ochsner was able to keep e-mail and Web access, because its T-3 line connecting to a back-up site in Baton Rouge still worked.
“At the peak of the storm, during the 100-mile-an-hour gusts, I walked over to the nurses’ unit, and they were keeping track of the storm online and e-mailing people,” Induni says. “If we were cut off completely, people would have gotten really nervous and would have been unable to work. It was our only lifeline to the outside world.”
Ochsner’s IT staff in New Orleans and Baton Rouge still had to scramble to keep e-mail available, even after the storm. A fiber cut in New Orleans after the storm terminated access to the hospital’s e-mail servers, located at another facility on the hospital’s metropolitan-area network in the city. Teams at the remote site quickly built Novell GroupWise boxes to serve the New Orleans facility via the T-3 link.
“All the disaster planning in the past,” which focused on the upkeep of systems and data availability, “became second fiddle to the need to maintain outside communications,” Induni says.
In keeping the technical services running at the 400-bed hospital – one of only three in the city that remained open during the storm – Induni gives all the credit to his staff.
“The human side of this is huge,” he says. “Many of our staff people have lost their homes. One person’s house is covered in oil. They’ve lost everything. We’ve had to go through that adjustment and that understanding, but we’re also expected to be productive. . . . We put a lot of thought into what we were doing and why we were here. There were a lot of opportunities for our folks to throw their hands up in the air . . . but we pulled through it.”
Courting disaster
Up north in Baton Rouge, for Louisiana’s state and local court systems, Katrina will be the harsh lesson of a lifetime on the importance of disaster-recovery planning for data backup, says Freddie Manint, CIO for the Louisiana Ninth Judicial District Court.
The state court offices in Baton Rouge are now overflowing with employees from clerks offices, and sheriffs departments from Jefferson Parish, Plaquemines Parish, Orleans Parish and others are now operating out of spare rooms, closets and any open space available in the state court offices, working from laptops tied in to the state court’s LAN.
“They’re getting everything they need to get some kind of operations running and to plan their next move,” Manint says. “The problem is, [parish court administrators] haven’t been able to assess how much they lost in terms of IT infrastructure, databases and records. A lot of [court] facilities did not have disaster recovery hot sites with immediate failover. A lot of documents were just in hard-copy form, and they’re under water now. A large portion of systems are ruined.”
Each of Louisiana’s 64 parishes operates its own court system, with its own records and files, some electronic, some not. None of these systems were linked to a centralized system for backup or disaster recovery, Manint says.
“The justice system in Louisiana is really going to waver here for a while,” Manint says. “We’ve got people in jails all over the state right now that have no paperwork and we have no way to offer them any kind of means for adjudication.”
Just last week, some court records from Orleans Parish clerk offices were recovered. The Digital Linear Tapes were en route to Baton Rouge to see what is salvageable. “Hopefully we can extrapolate something useful from that,” Manint says.
Measured against parish court facilities near the Gulf, the state court’s facilities in Baton Rouge got off easy, except for some sporadic power outages, and experienced no data loss, Manint says. The state court system uses VMware software to create virtual server images to back up its core database files and records, which are dispersed across facilities.
Manint says the state court’s over-built 10G Ethernet backbone, which includes redundant 3Com Switch 7700s, has handled the influx of new employees well with plenty of bandwidth to spare. He says 3Com was very responsive to the influx of workers into the court’s Baton Rouge offices, Manint says, with the vendor sending several hundred LAN switches and IP phones to set up the temporarily offices.
He estimates that 80% of court IT infrastructure in the New Orleans area will have to be replaced.
As for the future, Manint is hoping for more federal funding to modernize how Louisiana handles and protects court records.
“Everyone wants disaster recovery but they don’t want to fund it,” he says. “It’s not important until you don’t have it. Everyone was always talking about the big one but not doing anything about it. Now the big one hit, and everyone’s standing around saying, oh man, we got wiped out. The excuse that there’s no funding [for disaster recovery] is not an excuse anymore. . . . If you don’t have data, you don’t have a job.”
History teaches a lesson
For one electrical and telecom utility working through the storm, lessons learned during Hurricane Ivan last year and Dennis in 1999 paid off during Katrina.
The Southern Company, operator of power utilities and wireless services in Georgia, Alabama, Florida and Mississippi, began preparing for the next big storm last year by building redundant SONET rings to connect its sites across states. Right up to the first 100-mph gusts, crews were readying mobile field offices with PCs, VoIP phones and satellite uplinks. In Mississippi plants, workers were sandbagging doors to telecom rooms and wrapping PCs up in plastic wrap.
“By all measures Katrina was a much more severe storm than either Ivan or Dennis, but the impacts were less in terms of the number of sites down,” says Dan Traynor, director of IT infrastructure services.
During Ivan, Southern Company lost connectivity to several sites because its long-haul fiber was laid out in a linear topology; one fiber break meant everything beyond the cut was offline. Last year, the company rebuilt its long-haul fiber in an OC-48 ring using optical gear from Ciena. During Katrina, the company’s fiber rings broke in three areas, but data services were not interrupted, he says.
Another lesson learned from Ivan was the concept of adding satellite communications for staging areas to quickly get downed lines back up. This time, the Southern Company rented mobile offices ahead of time and outfitted them with computers, network gear, VoIP phones and a satellite uplink, which beamed voice and data to a satellite service provider and back to the company’s Atlanta headquarters. This kept field technicians online through the restoration process; 174,000 of Mississippi Power’s 195,000 customers were back online as of last week.
With Mississippi Power’s corporate offices now uninhabitable, the company is working to get the 300 employees there up and working again.
Before the storm hit Mississippi, back-up tapes of critical databases and applications were driven to alternate sites in Alabama and Georgia, where IT staff rebuilt the systems and put them back online via the OC-48 network.
Mississippi Power employees are now going to work in mobile office trailers being set up near the Southern Company’s fiber facilities, allowing workers to tap into the SONET network and use applications as if they were back in their normal office.
More amazing, Traynor says, is the attitude of the employees.
“Of the 1,200 employees, about half of them had severe damage to their homes, or their homes were completely destroyed,” Traynor says. “At the same time, these people were working on our restoration efforts; they weren’t thinking about themselves. It’s pretty inspiring.”
