Eye of the hurricane: New Orleans prepares

Disaster-recovery plan following Katrina based on Wi-Fi, VoIP.

New Orleans prepares its disaster-recovery plan the wake of Katrina, and it's based on Wi-Fi, VoIP.

part one with link to slideshow

Residents with Wi-Fi-enabled laptops or VoIP phones would be able to access the Internet for free over EarthLink’s 15-square-mile downtown wireless mesh network, a stipulation in the company’s contract with the city that Meffert insisted upon. “We needed it to be free, a baseline free service. We need it to be there during a storm,” he says.

The recovery plan also calls for a backup data center 450 miles away — well inland — in Austin, Texas. Through an arrangement with the city of Austin, key New Orleans servers will occupy a rack in Austin’s data center. That standby recovery center will take over as soon as weather forecasters place New Orleans in the “cone” of future hurricanes, the area a storm is most likely to hit. “When we’re in the cone, we’re going to roll all our servers to Austin, and we’ll do that every storm,” Meffert says.

Because the plan relies on Internet connectivity, the city has contracts with local ISP Data Sync, plus Cox and Level 3 Communications, and each provider uses a different path to the Internet.

In addition, the municipal Wi-Fi network that covers about 4 square miles downtown, including the French Quarter, Central Business District and the Ernest N. Morial Convention Center, will connect to a tower on the tallest building in the city, One Shell Square, which will link via an optical point-to-point connection to a Level 3 Internet point of presence in neighboring St. Charles Parish, Meffert says. The city plans to install generators stocked with weeks worth of fuel to keep the municipal Wi-Fi cloud running.

Greg Meffert, New Orleans' CIO:

“If nukeproof telco switches were a way to deal with crises, the Department of Defense would never have built the Internet to begin with.You need something that can take a direct hit. [With VoIP] you could change a DNS server, an e-mail provider, and I don't need to know about it. Your communications still get through.”

Greg Meffert, New Orleans' CIO.

Lessons learned the hard way

Meffert’s decision to base his disaster-recovery plans on Wi-Fi and VoIP is a direct result of the lessons learned after Katrina, when, for example, getting a simple phone connection so the mayor could talk to the president was a major challenge.

Meffert gained national attention immediately following Katrina when he went into a looted Home Depot, grabbed a Cisco router, found a single live connection at the Hyatt Hotel where the mayor was holed up and used a Vonage client to connect the mayor with President Bush.

But that was the least of Meffert’s challenges. In Katrina’s wake, he found that there was no plan to reestablish public communications, so residents could call out to let loved ones know how they were. And the city lacked the infrastructure to process the overwhelming volume of damage assessments, claims, permits and other bureaucratic forms needed to rebuild.

Not that the municipal government was caught napping when Katrina hit. It’s just that nobody was prepared for a storm of that destructive magnitude, a storm that took down absolutely all networking, not just for days, but for weeks. “It was a shocker for me,” Meffert says. “If somebody knows a storm is coming, you expect they’ll have what they need to do their job. We didn’t even have food and water and plumbing.”

He adds, “You rely on telecom vendors to provide certain services, and you assume if they go out, they’re going to come back,” he says. “If people were working at 911 dispatch desks, they’re going to come back. If you have a diesel backup that kicks in and will run for a day and a half — it’s plenty if you’re going to get more gas.”

But nothing came back. Telecom switching offices were destroyed — at least one was washed away without a trace. Emergency workers lost their homes and didn’t return to work. Generators that weren’t flooded ran out of fuel, and there was no way to get more.

New Orleans residents wait in line to use an automated system that speeds the permitting process.

New Orleans residents wait in line to use an automated system that speeds permitting. Exterior, New Orleans City Hall

Putting it together on the fly

Awash in chaos, Meffert and his team improvised, enlisting whatever technologies they could find to patch together rudiments of a network that could immediately address some of the city’s needs.

For example, before Katrina, the city had deployed Wi-Fi access points to support police with 130 surveillance cameras scattered in high-crime districts. When power was restored after the storm, many of the access points came back to life. The city then expanded the Tropos-based mesh, linked it to the Internet and opened it for residents to make IP phone calls via VoIP software from Skype and Vonage, as well as send-and-receive e-mail and instant messaging.

“It’s obvious why it’s so much more reliable than the phone network,” he says. “We’re sitting in 8 feet of water, and we look up and what’s there? The telephone poles with the access points mounted on them.”

In future emergencies, VoIP will take the place of traditional telephony for the city of New Orleans recovery effort, Meffert says. City calls will be routed from laptop voice clients or VoIP phones through a Cisco Call Manager server, which can connect to other IP phones or to the public phone network.

This will be particularly useful when outside agencies try to contact the mayor, for instance. As long as the mayor stays near the IP phone associated with his emergency contact number and the phone stays connected to the Internet, the network will find him. With the three new Internet connections, that shouldn’t be a problem.

Because making these calls relies on the Call Manager, one priority is making sure the Call Manager server is running and accessible. During Katrina, power to the New Orleans city data center on the third floor of City Hall was out for weeks. Fortunately, key data was backed up in a California data center run by ACS, and the city’s Web portal was hosted on servers at Red Carpet Host in Dallas, allowing workers to carry out minimal city business.

But that arrangement still left public-facing applications normally accessible via the portal unreachable. They were on servers back in City Hall, without power and unconnected to the Internet, Meffert says.

Automating the permit process

Once the waters subsided, the city needed to get people to rebuild, which involved applying for federal funds and issuing permits for demolition, building and renovating, says Michael Centineo, the city’s director of safety and permits. Improvements that came out of the emergency effort to meet those demands will streamline administration permanently, he says.

Before the storm, the city had set up a Web portal to allow residents to fill out permit applications online, using an application from software vendor Accela. Online or at touch-screen appliances in the hallway outside the permit office, applicants could enter their own data before talking to city staff and, depending on what they entered, be placed in the appropriate queue for personal attention. This procedure paid off by saving people from hours of needless waiting, because applicants discovered quickly whether they lacked information they needed to retrieve.

The problem was that the maximum load on the permit system pre-Katrina was about 35 per day, and demand peaked at about 500 per day afterward, Centineo says. At the same time, budget cuts reduced permit staff from 127 to 60, he says, and only by expanding permit automation did his office keep up. “This department would have been frozen and inactive, because it would have been overwhelmed by the volume of requests,” Centineo says.

Immediately after the storm, city programmers customized the building-permit interface of the Accela software to make it easier for untrained citizens to use, says Peter Bodenheimer, a city IT project manager who oversaw the code writing. “We like to go with custom code, so we don’t have to rely on others’ timetables,” he says, “and we needed this done. We had people who were screaming.”

Meffert says the city tried to make the user experience require as close to zero training as possible, because the system simply had to work; there was no alternative. “We had to go to the extreme and have the user compatibility of Disneyland. You walk in, and you have a greeter to escort you to the kiosk box and explain how it works,” he says.

Hitting the streets with Toughbooks

Eight weeks after the storm, the waters were gone and the city had to figure out the extent of damage and what buildings were safe to repopulate, Meffert says. That called for a whole new layer of bureaucracy for which the city had no infrastructure. “We had to inspect 110,000 homes in six weeks. There was only one way: create a more simplified inspection process,” he says.

So the city wrote its own damage-assessment application for gathering information the Federal Emergency Management Agency would want and loaded it onto 34 Panasonic Toughbook laptops with wireless cards and global-positioning capabilities that inspection teams carried into devastated neighborhoods.

The laptops could call up digital records of individual properties via the Wi-Fi links, and inspectors verified and updated the data. The records had been geo-coded so records of property could be called up based on the location of the laptop as calculated by GPS. An inspector sent to a parcel of land where a building was washed away entirely, for instance, could still find the records of what had been there.

The updated database was made publicly available via the city Web portal and public kiosks. “That’s how we told people if their house was inspected or not,” Meffert says. “We had to create new [Wi-Fi] hot spots that would sync with the database.” The database was also linked to a city map to differentiate inspected sites from uninspected ones, so inspectors could see where to go next.

Residents could use online access or public kiosks to find out the damage assessment for their property, and staff was available to resolve disputes. Residents could fill in a Web form from home and get a reservation number to get them to an agent at a set time the next day.

With its success automating and Webifying building permits and the damage assessments, the city plans to add electrical, mechanical, air-conditioning, heat and gas permits to the Web process, Centineo says.

Self-service kiosks for the French Quarter

Another goal is to place weatherized kiosks outdoors in the French Quarter. “We’ve seen what self-service can do,” Bodenheimer says. Residents will be able to pay property taxes, parking tickets, research real estate transactions or find out what licenses have been issued for the construction going on next door. Whatever is available via the Web portal will be available at the kiosks. “Anything you can do that’s city business online, we want to do with a kiosk. We want to make sure every citizen is served, not just those with computers,” Bodenheimer says.

The program is being expanded to place kiosks in trailers in neighborhoods, so those without transportation or Internet access can get what they need within walking distance of home. The holdup is the trailers. “It seems like we’re always waiting on trailers,” Centineo says.

Since the storm, the city has also expanded the public interfaces to include a place to report new classes of crime, Meffert says. For instance, architectural theft — doors, mantels, windows, fences that are stolen from abandoned houses — is epidemic, he says. Residents can report this looting online to a city database. Using an administrative interface, police can log on to search reports of thefts in their districts and draw maps marking the hardest-hit neighborhoods and so better plan patrols.

As the new hurricane season plays out, Meffert says he hopes the disaster preparation the city has made proves successful, not only for New Orleans, but as an example for others. “It’s actually a symbol not of where we’ve been, but where we’re going,” he says. “People can look back and say New Orleans was leading and not just in po’ boys and beer. It was something pretty important.”

New Orleans' CIO Greg Meffert:

“VoIP and Wi-Fi are a religion to me. You need something that’s far more flexible than wired telecom networks.’’

Learn more about this topic

New Orleans companies prepare for hurricane season


Experts: Communications still a problem in disasters


Red Cross looks to IT for post-Katrina recovery


Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Now read: Getting grounded in IoT