TiVo's disaster recovery plan

TiVo, a company with more than 600 employees, 700 servers and 100TB of data, is based in Alviso, Calif., an area prone to earthquakes and other natural disasters. With that in mind, the company recently set up a disaster recovery site in Las Vegas, co-located with the service provider Switch Communications. Richard Rothschild, director of IT at the DVR maker and service provider, says TiVo chose Las Vegas for three main reasons:

1. Driving distance. Las Vegas is outside California’s earthquake-prone borders but still within driving distance of Alviso, meaning TiVo staffers can reach their disaster recovery site easily in the event of a disaster near the home office.

2. Network- and power savvy. The casinos -- Las Vegas’s main business residents -- are highly dependent on networks and power, which means that the city is well served by both.

3. Few natural disasters. Las Vegas is relatively free from earthquakes, floods, hurricanes and other challenges to business continuity.

Read about how TiVo deals with compliance issues

The biggest challenge TiVo has in the disaster recovery site is ensuring that it remains in sync with production-level data, especially in terms of software revisions and so on.

“Version control and keeping software up to date is a big problem,” Rothschild says. “We use Opsware there, because that’s what it’s designed to do. It packages up the software versions and pushes it out in an efficient manner. It really helps us solve that problem.”

TiVo is also considering moving to virtualization as a way to simplify disaster recovery. “If a service has a problem, you can just apply a virtual image of it onto another server or servers and very quickly recover,” he says. “It really shortens your time to recovery, which is the thing everyone is always searching for when they have a disaster recovery facility.”

TiVo is moving slowly because while some services work well when virtualized, others can’t quite make the leap. “Services that have a lot of [transactional] activity or where changes to the OS or configuration happen frequently don’t work as well,” he says. “There you have to be much more careful of virtualization because you’re banking on some of the software, like the OS, to remain fairly constant. And tracking it can be a real nightmare.”

Still, virtualization is on his road map. “I’d say within the next month or so, we’ll have virtualization across quite a few of our services,” he says.