The case of the great hot-site swap

A do-it-yourself disaster-recovery plan means sharing resources and expertise. A Maine college and a California university save money with a sophisticated hot-site swap.

A Maine college and a California university are reaching across the continent to share hardware and software in a joint disaster-recovery effort that could be a model not just for other schools but also for other businesses.

You can think of it as the lobster and sushi project.

Besides creating recovery sites at each other’s campus, the networking staff at Bowdoin College, in Brunswick, Maine, and Loyola Marymount University (LMU) in Los Angeles, also are creating a set of practices that can guide other cooperative IT ventures between different organizations.

Two identical recovery sites, based on blade servers and VMware’s virtual server software, are being assembled on each campus, linked to the Internet with a secure VPN connection over a 30Mbps link. Each campus will host and manage hardware and software bought by the other institution. If a disaster or outage hits either school, the hosting campus will initialize the other’s hot site and run it for the duration of the emergency. IT staff on the stricken campus will access what is in effect their own logical data center, one that’s located on the other side of the United States.

A recent study found almost 30% of business are completely unprepared for disasters or emergencies that could cripple them.

The first step has been to run each other’s emergency Web site, Exchange e-mail and DNS servers. This summer and fall, both schools plan to roll out other virtual servers (see graphic). These new servers will support replication of Microsoft Active Directory, programs such as NTI Group’s Connect-ED, which can record an emergency voice message and distribute it via e-mail, cell phone, paging, instant messaging and other media, and enterprise applications such as learning management systems, payroll, and student information systems.

“This is a dandy example of how to do things properly,” says Michael Karp, senior analyst for Enterprise Management Associates, a technology research company based In Boulder, Colo. Karp covers storage issues, and has been advocating the idea of cooperative disaster recovery, especially for small businesses. “The IT issues are the same for both institutions, around privacy, data availability, records management and all those nitty-gritty details. And it will cost LMU about the same amount to supervise as it will cost Bowdoin, because they have the same infrastructure.”

Keeping costs low

So far, the initial work has been fairly inexpensive, according to numbers compiled by the schools, about $35,000 from June 2006 to June 2007 for each institution, mainly in staff time of about 15 to 20 hours per month. That includes a couple of cross-country plane trips for selected IT staffers from both schools. The schools are projecting about $54,000 more, for each institution, in spending for blade servers, a terabyte of network storage, software licenses, new applications, labor and other expenses.

That compares with an estimated $100,000 per month, or $1.2 million per year, for a commercially hosted disaster-recovery hot site, according to figures from the two schools. That adds up to a return on investment of over $1.1 million yearly.

Although there have been some earlier efforts at disaster-recovery collaboration, none seem to have the ambition of the Bowdoin-LMU project. Two small neighboring Massachusetts colleges, Babson College and Franklin W. Olin College of Engineering, have been sharing offsite storage and tape backup capabilities. And in the wake of the devastating 2005 hurricane season, spearheaded by colleges and universities in the Gulf region, more institutions have been trying to add long-distance into their disaster-recovery plans.

Complementary differences smooth the way

You can’t get much farther apart not only in distance but also in sensibility and culture than Bowdoin, with about 1,700 students in rural and largely white Maine, and LMU, with about 8,700 students in urban and ethnically mixed California. Yet the differences have proven to be complementary strengths rather than stumbling blocks.

Read about how the two teams co-existed and got the job done.

The seed for the idea was hatched a few summers ago when the schools’ CIOs, Mitch Davis at Bowdoin and Erin Griffin at LMU, first met at an academic computing conference. At one session on disaster recovery, Davis argued that smaller institutions could sidestep the crippling costs of disaster recovery by working together.

“We just started putting together how we could this,” Griffin says. “We were kind of ‘skunkworking’ it for a while because it didn’t have the sex appeal of other projects. Until after Katrina.”

Planning and decision making started in earnest in summer of 2006. The initial decision on using VMware as a foundation was quickly followed by other infrastructure decisions needed to support the initial round of essential services, from DNS hosting to VoIP. “That is a pretty huge undertaking right there,” Griffin says.

“It seemed really daunting….”

“It seemed really daunting when we first started,” says Dan Cooke, LMU’s director of systems administration. “But as we worked together, it started seeming much more tangible and realistic.”

One reason was the quick realization that each school brought expertise that could benefit the other in deploying new products and technologies. “They had not done anything with virtualization at all,” recalls Tim Antonowicz, systems engineer at Bowdoin. “We were over 70% virtualized, at the time. I flew out to California to get their feet wet with VMware. Now, they’re almost as involved [with it] as we are.”

When Bowdoin switched over to Exchange e-mail, so the schools would have similar e-mail infrastructure, LMU staffers were their guides and advisors. “We implemented that pretty quickly,” says Davis, the Bowdoin CIO. “When we launched Exchange, we had just eight calls to our help desk.”

And the shared experience of the infrastructure components then forms a kind of informal help desk, where managers and staff can reach out for advice, brainstorm and troubleshoot problems with their colleagues a continent away. “I can send an instant message to Dan and say ‘have you seen this [problem]?’ and he IM’s back ‘we saw this three weeks ago and here’s what we did,’” says Antonowicz. More formally, teams from both schools meet every Tuesday via videoconference to review the project, identify problems and plan the next steps. The discussions and decisions are recorded for later viewing by others.

Building a relationship based on trust

“It’s about developing a relationship, more than a business agreement,” says Davis. Relationship-building was vital in overcoming what Davis sees as the main obstacles in the project: fear and potential distrust. “All of sudden, stuff is moving off your campus and out of your control,” he says.

There is friction at times, often over speed of implementation and priorities, says Davis. “Erin wanted a particular project done right away,” he says. “For us, it sits on the next stage below. We’ve had some strong talk about doing this faster. She has a project manager who’s always telling us, ‘you have to get moving on this.’ And that’s not a bad thing.”

“This is not something that pops up and happens,” says Davis. “It seems kind of organic [in development] but everyone was thinking about how to make things better for the client [the other school] and optimize it to be easier for IT to support.”

Davis and Griffin see this project as a template for future cooperation, and not just in higher education. “We’re building a methodology for approaching other collaborative IT projects,” says Griffin. “That outcome is as important as the hot site itself.”