University flees fire with network in a box -- literally

Virtualization, disaster recovery key to getting Fielding Graduate University back online fast

Current Job Listings

When fire again threatened to char Santa Barbara, Calif., last month, the IT staff at Fielding Graduate University literally raced flames out of town with its network stashed in a cardboard box safely resting on the back seat of the getaway car.

While it might sound like a dramatic rescue, it was more the result of a disaster-recovery plan, a newly virtualized network and a quick-acting network operations team.

Seven steps to disaster recovery planning

And not wanting to settle just for an escape, the team got operations back on line in a day at an off-site location where they kept the online university functioning through a nine-day ordeal.

"We joked that it was our network-in-a-box, but that is the power of virtualization and a blade environment," said Deby DeWeese, director of network services for the university.

For Fielding, the network is the university as graduate students and faculty collaborate over the Internet. The school just months prior to the fire had completed consolidating 30 servers running its Windows network onto a virtualized network based on Microsoft's Hyper-V technology and four HP ProLiant BL460c blades that included an HP MSA 1500cs SAN. In all, the network and a data protection system housed 2.4 terabytes of data.

The network-in-a-box version was born May 5, the day the Jesusita Fire broke out.

DeWeese's boss – Dan Sewell, associate provost for research and chief learning officer – returned to the data center that night when the flames came closer to the city. Over the phone from her rural Santa Barbara County home, DeWeese walked Sewell through pulling the blades and unhooking the school's servers and disk array that runs its Microsoft Data Protection Manager (DPM) system.

Fielding's networks operations team

The next day they returned, plugged everything back in and started to work.

But the fire continued to rage and by that afternoon Fielding's operations center was a block from the fire's evacuation zone.

"We could see flames from our office windows and ash was falling from the sky," DeWeese said. "We couldn't keep doing network-in-a-box, we needed to get something up and running somewhere else."

DeWeese and Sewell again did the network-in-a-box drill, while HR made sure staff was safe.

And what happened next was a combination of some nimble negotiating and finding friends in the right places that resulted in the university returning online from a new location in about 25 hours.

DeWeese managed the communications from her home, where the only Internet access is via satellite and where she still had reliable phone lines. She even helped displaced colleagues find hotel rooms around Santa Barbara. Sewell kept communication open with school leadership for decision making.

Ironically, the day the school got back online was the same day it had a scheduled data dump that would have backed up all the university's data to a site in Las Vegas run by provider SBWH, which has roots in Santa Barbara.

That was now on hold, but DeWeese called SBWH, the school's disaster-recovery vendor, and they offered engineer Paul Fisher to help clear rack space the company rents 10 miles from Santa Barbara with Tw Telecom, which just happens to be the provider of Internet lines for the university and of infrastructure for emergency services such as fire fighting.

SBWH did some redistribution of their own network hardware to create a chassis to house the blades, the disk array and DPM platform were plugged in, and the network was now safely running miles from the threat of flames that would escape containment for another seven days while consuming 8,733 acres.

E-mail was the first thing brought back online by Niv Dolgin, the director of IT services for SADA Systems, the school's remote e-mail management provider.

But a snag came when DeWeese discovered that Fielding's disaster-recovery plan neglected to account for changes to IP address and DNS updates after the move to a new location.

"That is one of the things we would have found out in the move to Vegas," DeWeese said.

The IP addresses were key for connections to the school's online library, a collection of about 1,000 databases subscribed to by the university.

Alain Dussert, the school's director of library services, led the effort to contact the database vendors so they could enter new IP addresses on their end.

"That is what took the longest to set up. Otherwise, we would have been up in an hour," DeWeese said.

The disaster recovery also included redirecting phone lines. DeWeese found a VoIP line to use with CIO Solutions, the school's virtual network administrator." CIO Senior Engineer David Ashamalla helped handle that and a majority of the move's technical work.

The phone provided a recorded message for staff and students. In addition, the school's provost used her Facebook profile to do updates, and help desk requests were routed past the help desk system and straight to the coordinator's personal in-box.

All the messages eventually funneled staff and students to the school's Epsilen.com social networking site, where all information was being aggregated.

The only network piece orphaned was an ERP system running on IBM's AIX hardware.

IBM said it would void the warranty if the school moved it, but IBM promised to come out in five days to do it themselves. The school declined and the system was offline until the IT staff returned to the office.

The return happened at noon on Monday the 11th, five days after they had fled their office. DeWeese and her staff reversed the move process and had the network up and running by Tuesday morning.

Having literally been through a fire drill, DeWeese says she is thankful that the school decided to virtualize its data center, to go to a blade system that she could direct someone to dismantle, to construct a detailed disaster-recovery plan and to choose "really good vendors."

"We deconstructed the disaster-recovery plan afterward and we found we did an exceptional job. Without it we would have been out of business for a week," she said. "When we were building the virtualized network and people were losing sleep, I had no idea it would be so satisfying to get this outcome."

There also were other lessons to stash away if there is a next time.

"And I learned I can find a cheap hotel room in Santa Barbara," DeWeese said.

Follow John on Twitter.

Learn more about this topic

Seven steps to disaster recovery planning

Security and regulatory concerns slow some server virtualization efforts

Hot data center jobs: Cloud, virtualization, green

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Now read: Getting grounded in IoT