University flees fire with network in a box -- literally

When fire again threatened to char Santa Barbara, Calif., last month, the IT staff at Fielding Graduate University literally raced flames out of town with its network stashed in a cardboard box safely resting on the back seat of the getaway car.

While it might sound like a dramatic rescue, it was more the result of a disaster-recovery plan, a newly virtualized network and a quick-acting network operations team.

Seven steps to disaster recovery planning

And not wanting to settle just for an escape, the team got operations back on line in a day at an off-site location where they kept the online university functioning through a nine-day ordeal.

"We joked that it was our network-in-a-box, but that is the power of virtualization and a blade environment," said Deby DeWeese, director of network services for the university.

For Fielding, the network is the university as graduate students and faculty collaborate over the Internet. The school just months prior to the fire had completed consolidating 30 servers running its Windows network onto a virtualized network based on Microsoft's Hyper-V technology and four HP ProLiant BL460c blades that included an HP MSA 1500cs SAN. In all, the network and a data protection system housed 2.4 terabytes of data.

The network-in-a-box version was born May 5, the day the Jesusita Fire broke out.

DeWeese's boss – Dan Sewell, associate provost for research and chief learning officer – returned to the data center that night when the flames came closer to the city. Over the phone from her rural Santa Barbara County home, DeWeese walked Sewell through pulling the blades and unhooking the school's servers and disk array that runs its Microsoft Data Protection Manager (DPM) system.

Click to see: Fielding's networks operations team

The next day they returned, plugged everything back in and started to work.

But the fire continued to rage and by that afternoon Fielding's operations center was a block from the fire's evacuation zone.
"We could see flames from our office windows and ash was falling from the sky," DeWeese said. "We couldn't keep doing network-in-a-box, we needed to get something up and running somewhere else."

DeWeese and Sewell again did the network-in-a-box drill, while HR made sure staff was safe.

And what happened next was a combination of some nimble negotiating and finding friends in the right places that resulted in the university returning online from a new location in about 25 hours.

DeWeese managed the communications from her home, where the only Internet access is via satellite and where she still had reliable phone lines. She even helped displaced colleagues find hotel rooms around Santa Barbara. Sewell kept communication open with school leadership for decision making.

Ironically, the day the school got back online was the same day it had a scheduled data dump that would have backed up all the university's data to a site in Las Vegas run by provider SBWH, which has roots in Santa Barbara.

That was now on hold, but DeWeese called SBWH, the school's disaster-recovery vendor, and they offered engineer Paul Fisher to help clear rack space the company rents 10 miles from Santa Barbara with Tw Telecom, which just happens to be the provider of Internet lines for the university and of infrastructure for emergency services such as fire fighting.