Adventures in managing virtualization

Evan Jafa, CIO of First American, says getting your virtualized servers set up and running right really is just the start of any IT leader's virtualization work. And that if you don't think holistically about virtualization, you're in for a rude surprise.

"I think lots of folks are going to find out that virtualization poses a whole new set of concerns for security, networks and applications," says Jafa.

In fact, he calls management a "make or break issue" for virtualization. "Monitoring and management becomes absolutely critical in the virtual environment," says Jafa, who started work about three years ago on a large project to consolidate First American's data centers and standardize its storage and networking technologies.

To that end, his organization is directing much of its energy now at management processes and tools, wide application of ITIL process to the virtualized environment, and even reshaping the IT staff.

Jafa hopes that in the future virtualization will play into better management of his networking technologies as well. He's already testing some new technologies from vendors including Cisco to advance that goal. If you're trying to do holistic virtualization planning, Jafa's advice may prove particularly interesting.

Big Provisioning and DR Wins

First American's business, which posted US$8.2 billion in 2007 revenues, delivers information such as mortgage, real estate and financial data to clients in those industries and others. So while many IT leaders closely monitor virtualization projects to ensure that application service levels don't slow down for internal users, Jafa's team also keenly focuses on application performance for external customers.

The company today has two data center locations, one in California and another in Texas, with about 4,500 physical servers, soon to be 6,500 (due to newly added business units.) It serves 40,000 to 50,000 concurrent users during peak business hours; its network carries 10 to 21 terabytes of data per hour, averaging about 15 terabytes per hour, Jafa says.

To keep all that data flowing smoothly, Jafa says his tool chest currently includes offerings from VMware, Microsoft, BEA (WebLogic) and "lots of Linux". His team is also using HP's SiteScope tools for hardware performance monitoring.

Jafa says he defines virtualization as "achieving the highest efficiency of any given system at any give time". In servers, virtualization is netting him 50 to 60 percent utilization rates, compared to 10 to 15 percent utilization rates on standard servers, he says. The team also sees a 30 to 35 percent reduction in energy consumption per virtual machine compared to its physical counterpart, Jafa notes.

To date, IT has virtualized 25 to 26 percent of First American's servers, Jafa says. These include many production servers, plus test and development and staging boxes. In some cases, whole business units now have had all their servers virtualized, he says. "My goal is to reach 50 percent virtualized this year," he adds.