Why San Diego city workers expect apps up and running in 30 minutes or less

Major VMware project makes application deployment a snap while easing the IT budget

Deploying a new application used to be a month-long headache and a budget-drainer at the San Diego Data Processing Corporation. Now it takes 30 minutes and costs next to nothing, thanks to an extensive server virtualization project.

Rick Scherer

Deploying a new application used to be a month-long headache and a budget-drainer at the San Diego Data Processing Corp. Now the process takes as little as 30 minutes and costs next to nothing, thanks to the extensive use of server virtualization.

The SDDPC, a private nonprofit that handles IT for the San Diego municipal government and its more than 10,000 city workers, embraced virtualization with a vengeance three years ago -- and has been reaping substantial rewards ever since.

7 tips for succeeding with virtualization..

7 things NOT do with virtualization

The first goal was server consolidation, says Rick Scherer, a Unix systems administrator who spearheaded the SDDPC virtualization project. "We were just like everyone else: We had tons of x86 machines that were only being 10% utilized," he says. "I presented to our directors that we could virtualize 20 machines into one box, get a better ROI and save tons of money plus data center space." Easier application deployment followed, which meant huge gains for business users, he adds.

Before using VMware's ESX Server, every application upgrade or new deployment meant the SDDPC had to buy -- then install -- a new server. Such is the nature of the Windows operating system, Scherer says. "Anytime there was a new application, we couldn't put it on a box with another application, because either those applications wouldn't work properly or the [application] vendor didn't support that."

For example, when the city needed to upgrade a purchase-order system for outside contractors, SDDPC had to push the project out three to four weeks to get the infrastructure ready, Scherer says. The same three- to four-week wait was in store when the organization needed to boost processing power for a Citrix Systems Presentation Server deployment. Besides the annoyance, each new HP ProLiant server cost somewhere around $10,000, he says.

When SDDPC started with server virtualization, users were surprised at how speedily IT could turn up applications, Scherer says. "But what's funny is now that we've been doing it so long, they expect it. It has put a damper on management," he says. Users are disappointed now "if they put a request in for a server and it's not up in a half-hour," he adds. One of the only things preventing further virtualization right now is the time Scherer and his colleagues must devote to day-to-day tasks and other projects.

VMware loyalty

Before deploying server virtualization, the SDDPC had about 500 physical x86 machines, largely from HP. With VMware, the organization can consolidate as many as 35 virtual machines onto one physical server. Such density has allowed the organization to power off 150 physical servers; it now runs 292 virtual machines on 22 physical x86 servers -- leaving plenty of room for expansion. "A lot of those hosts aren't even being used; they're just for future growth," Scherer says.

SDDPC also uses Sun's virtualization technology on Sun Sparc servers, and now runs 120 logical servers on 90 boxes.

The goal is to virtualize as much as possible: "We've set an initiative: For any new application or service that needs to be deployed in our data center, we're going to do everything we can to virtualize first. If there's no way to virtualize it, we'll look at physical hardware," Scherer says, noting that the organization also is aggressively moving the city's existing applications, as appropriate, to the virtual infrastructure.

VMware remains the server-virtualization tool of choice, even though products from rivals Citrix and Microsoft are now available and cost quite a bit less. Using VMware, SDDPC pays $10,000 to virtualize a four-socket machine, but that one physical host can support well over 20 virtual machines, Scherer says. If you buy 20 physical servers at $10,000 a pop, you're shelling out $200,000, he says.

Numerous management advantages, disaster-recovery capabilities and security also make the investment well worth it, Scherer says. The SDDPC operates two data centers and contracts with a co-location vendor in Chicago for disaster recovery. Initially, the organization included only mission-critical applications in its disaster-recovery plan, but it's beginning to account for many other applications -- those it considers crucial though not necessarily mission-critical -- because virtualization makes it feasible to do so from a cost perspective, he says.

In addition, SDDPC makes extensive use of VMware's VMotion, a live-migration function that moves virtual machines from one physical server to another without any downtime. VMotion comes in handy for balancing loads among servers, as well as for routine maintenance, Scherer says. "We can migrate virtual machines on a specific host to others within the cluster; this allows us to perform hardware and software upgrades with zero downtime and no impact to the customer," he says.

As for the security benefits, if a virtual server becomes compromised, it's easy to shut it down or isolate it into a separate network group. "If we [want to] have a Web zone, an application zone and a database zone, we can accomplish that with a lot less hardware. We're virtualizing our network as well. Instead of having separate physical switches we have to pay for, maintain and manage, all of it can be done within the [VMware] ESX host just by creating separate virtual switches," Scherer says.

SDDPC now is looking forward to its next big project, Scherer says -- virtualizing desktops. It's testing several thin-client devices to replace many of the city's 8,500 desktops, and plans to use VMware's Virtual Desktop Manager to provision and manage clients. This software "includes the ability to create a work-from-home scenario. A user can go home, not necessarily have thin-client at home, but through a Web-site connect to his desktop," he says. "That could potentially eliminate the need for Citrix, which is a significant licensing cost to the city every year."

Nevertheless, desktop virtualization will not happen immediately. The city refreshed most of its desktops in the past year, so the desktop virtualization project won't kick off for the next two or three years, Scherer says.

Sticky points

Despite realizing benefits, Scherer has run into some roadblocks and challenges related to virtualization. His goal is to virtualize nearly everything, but vendor licensing setups are holding him back. The SDDPC has virtualized Citrix's application-delivery software, SQL servers and all its Web services on VMware boxes. Vendor support issues, however, meant it could run SAP software only on virtual servers in its development environment, and run Exchange nowhere on a virtual infrastructure. SAP and Microsoft now support VMware, so Scherer will revisit those earlier decisions, he says. "Until virtualization is adopted by every single software company, you're going to run into those issues," he adds.

The potential for server overuse is another issue. Because it's so easy to deploy new virtual machines, an overzealous IT pro runs the risk of overcommitting resources. It may be tempting to allocate nearly 100% of a server's resources, but it's better to leave room for future growth.

In addition, virtualization introduces a potential single point of failure, because dozens of applications and virtual machines may reside on one physical server. "Your biggest risk is having a hardware failure and then no place for those virtual machines to run," Scherer says. That's why it's important to use such disaster-recovery features as live migration, and build out a strong network and storage system to support virtual servers, he says.

The SDDPC relies on about 250 terabytes of NetApp storage, and uses such features as data deduplication and thin-provisioning to maximize storage space and make sure each application has enough storage dedicated to it. Devoting too many physical and virtual machines to one storage array can be problematic.

"That recently happened: We had another application that wasn't virtualized and was on the same array as our virtual-machine environment," Scherer says. "Resources spiked up and caused contention on our virtual-machine farm. Luckily, we caught it in the early stage and moved the data off, but it was definitely a lesson learned," he adds.

Beyond storage, IT pros who embrace virtualization need to design full multipath networks, Scherer says. "You want to make sure that if a link fails, you have full redundancy," he says. "If all your virtual machines are running on a data store that has only one path and the link dies, that's the same thing as unplugging all your ESX hosts, and you're completely dead."

< Previous story: 7 things NOT to do with virtualization | Next story: IT security: What's hot, what's not

Copyright © 2008 IDG Communications, Inc.

The 10 most powerful companies in enterprise networking 2022