How to ID and fix 10 security threats on virtual servers

In 2007, the big question about virtualization in data centers was "How much money and time will this save us?" In 2008, the big question will be "How secure are we?"

It's an extremely tough question to answer. A slew of vendors and consultants trying to sell security products and services have conflicting opinions about the risks and how to prevent them. Simultaneously, some security researchers are hyping theoretical risks such as the possible emergence of malware targeted at hypervisors (a threat that has yet to appear in the real world). "There's a lot of noise out there on virtualization," says Chris Wolf, senior analyst for market research firm Burton Group. "It can be distracting."

Adding fuel to the hype is that fact that many IT organizations say they prioritized operational speed over most other factors, including security planning, when they started creating hundreds of new VMs in 2007. (That's not surprising, when you consider that most enterprises started with virtualization on their testing and application development boxes, not their servers running core business apps.)

"We're finding security is the forgotten stepchild in the virtualization build out," says Stephen Elliott, IDC's research director for enterprise systems management software. "That's scary when you think about the number of production-level VMs." According to IDC, 75% of companies with 1,000 or more employees are employing virtualization today.

And through 2009, 60% of production VMs will be less secure than their physical counterparts, Gartner VP Neil MacDonald predicted in a presentation at Gartner's October 2007 Symposium/ITxpo.

But much of the discussion about virtualization security has been flawed to date, says security expert Chris Hoff, because people often frame the discussion by asking whether virtual servers are more or less secure than physical ones.

That's the wrong question, says Hoff, who blogs frequently on this topic and serves as chief architect for security innovation at Unisys. The right question, he says, is "Are you applying what you already know about security to your virtualized environment?"

"People get wound up about theoreticals...when in reality there's a clear set of things you can do today," Hoff says. Certainly, virtualization does introduce some new security concerns, but first things first, he says: "We have to be pragmatic. Let's make sure we architect the virtual network as well as we architect the physical networking."

As an example, he points to a virtualization management tool such as VMware's VMotion, which is helpful for moving VMs around in times of machine trouble, but which can also allow someone with admin rights to combine two VMs that, in the physical world, would have been carefully separated in terms of network traffic for security reasons.

Some IT organizations are making a fundamental mistake right now: They're letting the server group run the virtualization effort almost single-handedly-leaving the IT team's security, storage and networking experts out of the loop. This can create security problems that have nothing to do with inherent weaknesses of the virtualization technology or products. "This is a perfect opportunity to bring the teams together," Hoff says.

"Virtualization is 90% planning," says Burton Group's Wolf. "The planning has to include the whole team, including the network, security and storage teams."

But the fact is, most IT teams ran fast with virtualization and now must play catch-up. What if you missed that opportunity to plan with all your experts, and you're starting to worry more as you expand your number of VMs and put higher-profile apps on those VMs?

"To catch up, start with a good audit of your virtual infrastructure," using tools or consultants, Wolf says. "Then you really have to work backwards." (Wolf suggests checking out audit tools from CiRBA and PlateSpin for this purpose.)

Here are 10 positive steps enterprises can take now to tighten virtualization security:

1. Get VM Sprawl Under Control

CIOs such as Michael Abbene, who runs IT for Arch Coal, understand the problem of VM sprawl full well: VMs take minutes to create. They're great for isolating certain computing jobs. But the more VMs you have, the more security risk you have. And you'd better be able to keep track of all those VMs.

"We started by virtualizing very low-profile test and development boxes," Abbene says. "Then we moved some low-profile application servers. We've been moving up as we've been successful. We understand we're increasing our risk profile as we do that." The company currently has about 45 production VMs, he notes, including Active Directory servers, and some application and web servers.

How do you control server sprawl? One approach: Make creating virtualized servers and VMs as disciplined as creating physical ones. At Arch Coal, the IT team is rigorous about allowing new VMs: "People have to go through the same process to get a server, whether it's physical or virtual," says Tom Carter, Arch Coal's Microsoft Systems Administrator, who works for Abbene.

For this purpose, Arch Coal IT uses a change control board (made up of a cross-section of IT staffers from disciplines liker servers and storage, serving on a rotating basis) to say yes or no to new virtualized server requests. This means, for example, that people in the applications group can't just build a VMware server and start creating VMs, Abbene says-though he's had developers ask to do just that.

VMware's VirtualCenter management tools as well as tools from Vizioncore can also help manage VM sprawl.

Ignore VM sprawl at your own peril, says IDC's Elliott: "VM sprawl is a huge problem, causing lag times in the ability to manage, maintain performance and provision," he says. Also, unexpected management costs will arise if your number of VMs gets out of hand, he adds.

2. Apply Your Existing Processes to the Virtual Machines

Perhaps the sexiest aspect of virtualization is its speed: You can create VMs in minutes, move them around easily, and deliver new computing power to the business side in a day instead of weeks. It's fun to drive fast. But slow down long enough to think about making virtualization part of your existing IT processes, and you will prevent security problems in the first place, says IDC's Elliott. You will also save some management headaches later.

"Process is important," he says. "Think about virtualization not just from a technology standpoint but from a process one." If you're using ITIL to guide your IT processes, for example, think about how virtualization fits into that process framework, Elliott advises. If you're using other IT best practices, look at how virtualization fits into those processes.

One example: "If you have a server-hardening document (prescribing a standard set of security and setup rules for a new server)," Hoff says, "you should do the same set of things to a virtual server as to a physical one."

At Arch Coal, Abbene's IT team does just that: "We take our best practices for securing a physical server and apply those to every VM on the box," Abbene says. Steps like hardening the OS, running antivirus on every VM and ensuring patch management, keep those virtual boxes in tune with the same procedures used on physical ones, he says.

3. Start With Your Existing Security Tools, But Be Critical

Do you need a whole new suite of security and management tools for your virtualized environment? No. Starting with your existing set of security tools for the physical server and network world and applying them to the virtual environment makes sense, says Hoff. But do press your vendors to tell you how they're keeping up with virtualization risks, and how they'll integrate with other products going forward.

"There's a false sense of security in relation to adopting physical tools for the virtual environment," IDC's Elliott says. At the same time, he adds: "It's very early in the market," for new security tools designed with virtualization in mind. That means you must press your legacy and potential startup vendors a little harder than usual.

"Don't assume the platform-level tools (such as VMware's tools) are good enough for you," Elliott says. "Look at the startups and the legacy management vendors. Press those legacy vendors to do more, and provide guidance for them."

Jim DiMarzio, CIO at Mazda North America, follows this strategy in his enterprise. Like Arch Coal, Mazda NA runs VMware's ESX Server 3 software at the core of its virtualized servers and has been ramping up its number of VMs recently. DiMarzio says he expects to have about 150 production VMs running by March 2008. He's using the virtualized servers for Active Directory servers, print servers, CRM application servers and Web servers-the last being a mission-critical app since Mazda uses these Web apps to serve information to all its dealers, DiMarzio says.

To secure these VMs, DiMarzio decided to continue with his existing firewall and security products, including IBM's Tivoli Access Manager, Cisco firewall tools, and Symantec's IDS monitoring tools.

At Arch Coal, Abbene and his team are sticking with the security tools they're already using, while also investigating tools from startups BlueLane and Reflex Security. "The [legacy] security and change vendors are trying to work hard to catch up and they're behind," Abbene says.

BlueLane's VirtualShield product for VMware, for instance, claims that it can protect virtual machines even in cases where certain patches are out of date, as well as automatically scanning for possible problems, updating problem areas, and protecting against some remote threats.

Reflex Security's Virtual Security Appliance (VSA), which Hoff describes along with BlueLane's software as one of the few emerging products worth attention right now, essentially serves a virtual intrusion detection system (IDS), adding a layer of security policies inside the physical boxes where the VMs live. It could help block a hypervisor attack, among other possible future troubles, Abbene's team figures.

Abbene says his IT group has also discussed adding a second internal firewall to further isolate the VMs, but he's concerned there might be a performance impact on the virtualized applications.

IDC's Elliott cites a few other virtualization security tools worth examining: PlateSpin, known for physical-to-virtual workload conversion tools and workload management tools; Vizioncore, known for file-level backup tools; Akorri, known for performance management and workload balancing tools; and storage firm EqualLogic, recently acquired by Dell and known for iSCSI storage-area network (SAN) products optimized for virtualization.

4. Understand the Value of an Embedded Hypervisor

Maybe you've read about "embedded" hypervisors already, but if you haven't, it's a term that IT leaders should understand. The hypervisor layer on a server serves as a foundation for housing the VMs. VMware's recently-announced ESX Server 3i hypervisor, designed to be very slim (32MB) for security reasons, uniquely does not include a general purpose OS. (And no OS means no OS maintenance chores.)

Some hardware vendors such as Dell and HP have recently said that they'll ship embedded versions of this VMware hypervisor on their physical servers. In basic terms, an embedded hypervisor is safer because it's smaller, says IDC's Elliott. "The larger the code base, the larger the opportunity for breaches," he says. "This becomes part of your architecture decision.

Embedded hypervisors will be a big trend going forward, Elliott says, and you can expect to see them from most server vendors, as well as some companies that haven't played in this space before. Phoenix Technologies, a market leader in the BIOS software field, recently announced that it's getting into the hypervisor game, starting with a product called HyperCore: It's a hypervisor for desktop and laptop PCs that will let users turn on the machine and use a basic Web browser and e-mail client without waiting to boot Windows. (HyperCore will be embedded in the machine BIOS.)

Competition and innovation in the hypervisor market would be good for enterprises, Hoff says. The end result could be companies slugging it out to deliver the slimmest, smartest hypervisor software.

"Whether it's Phoenix or someone else, there's a very interesting battle of these hypervisors becoming the next great OS," Hoff says.

Related:
1 2 Page 1
Page 1 of 2
IT Salary Survey: The results are in