Louisiana Supreme Court rebuilds its network using MPLS VPNs and hot backup.
When Katrina struck New Orleans, Peter Haas jumped into his RV with his wife, two daughters and a grandson and headed north to a campground near the town of St. Joseph’s to escape the storm.
Haas shot to the top of New Orleans’ most-wanted list not because of anything he did, but because he was director of technology for the Supreme Court of Louisiana and, without him, the court system couldn’t get back up and running.
The Supreme Court building was in an area of the city that had been overrun by looters, so his mission was to get into the courthouse, grab eight key servers and network gear, plus some important files and checkbooks (to keep them out of the hands of looters). He had 60 minutes, and then the SWAT team and the National Guard were pulling out.
It turned out the looters that had taken over that section of the city ignored the courthouse. His toughest problems were having to break windows and doors to gain access to what he needed, and the heat, which reached more than 100 degrees in the building.
Once that minor mission was accomplished, Haas had to create a new network so the seven state Supreme Court justices could get back to handling cases, as well as the important work of instructing lower courts how to respond to the emergency.
Peter Haas, director of technology, Louisiana Supreme Court:
“New Orleans could slide off the map in the middle of the night, and nobody at the remote site would know it until somebody called them and told them.’’
 |
| Peter Haas, director of technology, Louisiana Supreme Court. |
Haas discovered over the next few weeks that the court’s IT disaster-recovery plan was inadequate, and he has since rebuilt the network to eliminate single points of failure using MPLS VPNs, full replication of data in a hot-backup site, prestaged laptops and cross-training of IT staff.
The new network makes the Supreme Court system more resilient and adds other benefits, he says. For instance, when Microsoft issues a new server patch, he tries it out first in the backup data center to see whether it causes problems. When he has to do server maintenance that requires shutting down access, he switches the network over to the disaster-recovery site, and users are none the wiser.
Before the storm, the five justices who preside outside New Orleans had offices connected to the main courthouse via whatever broadband connection they could get, DSL, cable modem, T-1. When they all failed during the storm, these offices were unable to function.
This hodge-podge WAN has been replaced with a BellSouth frame relay over MPLS meshed VPN that provides instant rerouting if one site goes down.
The court has also built a second data center in an undisclosed location in the northern part of the state. “If a tidal surge reaches there, the whole nation has a big problem,” he says about the safety of the site. That center is constantly replicating with the primary center back on Royal Street in New Orleans’ French Quarter. If another hurricane knocks out Royal Street, the backup site can be switched on within an hour and all the remaining offices can connect to it via the MPLS mesh, Haas says.
“New Orleans could slide off the map in the middle of the night, and nobody at the remote [backup] site would know it until somebody called them and told them,” Haas says. The upgrade costs about $160 per month more than the roughly $1,400 per month the court had been paying for the unmeshed circuits, he says.
In addition to having duplicate servers and data stores, the court keeps records in binders of server settings for each machine, so if the primary and the backup are knocked out, someone can recreate it on new hardware. “My team can tell me every setting, every dependency on every server. If we were to lose all the gear, we could rebuild the entire environment,” he says. “We made two assumptions: One, we’re not here. Two, we don’t have any of this equipment. Someone who’s never seen the system has to build it from new equipment.”
He has also set up in-house training so formerly specialized IT staff can perform functions outside their areas of expertise. This problem came to light during Katrina when he pressed his desktop support administrator into setting up an emergency network at the 1st U.S. Circuit Court of Appeals building in Baton Rouge. He lucked out that the administrator could handle the job, and now he’s planning so he doesn’t have to rely on luck. “Our people were a problem, because each of us was a single point of failure,” Haas says.
The plan remains for the court to relocate to Baton Rouge in similar future emergencies, but this time there will be workstations ready. “We have laptops prebuilt and staged at the remote sites and in Baton Rouge, so if people can get there, there are laptops they can work on,” he says.
In addition, he has filed lists of hardware used in the data centers with the vendors he bought it from in the event equipment in both data centers fails. “All we have to do is call them and say pull our lists and start shipping,” he says.
He has also stockpiled commonly needed items — Category 5 cable, network connectors, phone wire — because during Katrina such supplies were hard to come by.
The court set up its servers on a rack in the data center of its ISP, Broadband, in Baton Rouge and connected them to a new network at the appeals court building via optical wireless links. The network was supported by a Cisco 2600 router and Cisco 3750 switches he salvaged from the New Orleans building.
A second optical wireless link connected the network in the building to a segment in two trailers dropped in the parking lot to serve as Supreme Court offices. “That got us talking to the servers, got our Web site back online and our e-mail moving,” Haas says.
With hard work and ingenuity, the court network was usable within 36 hours, but the Katrina experience taught valuable lessons that Haas has shared as a speaker at meetings of IT organizations. “I tell them if you don’t have a disaster-recovery plan, leave this room right now and start writing it. And don’t just write it, do it,” he says. “It’s not a matter of if you’ll need it, it’s when.”
