How to ID and fix 10 security threats on virtual servers

In 2007, the big question about virtualization in data centers was “How much money and time will this save us?” In 2008, the big question will be “How secure are we?”

It’s an extremely tough question to answer. A slew of vendors and consultants trying to sell security products and services have conflicting opinions about the risks and how to prevent them. Simultaneously, some security researchers are hyping theoretical risks such as the possible emergence of malware targeted at hypervisors (a threat that has yet to appear in the real world). “There’s a lot of noise out there on virtualization,” says Chris Wolf, senior analyst for market research firm Burton Group. “It can be distracting.”

Adding fuel to the hype is that fact that many IT organizations say they prioritized operational speed over most other factors, including security planning, when they started creating hundreds of new VMs in 2007. (That’s not surprising, when you consider that most enterprises started with virtualization on their testing and application development boxes, not their servers running core business apps.)

“We’re finding security is the forgotten stepchild in the virtualization build out,” says Stephen Elliott, IDC’s research director for enterprise systems management software. “That’s scary when you think about the number of production-level VMs.” According to IDC, 75% of companies with 1,000 or more employees are employing virtualization today.

And through 2009, 60% of production VMs will be less secure than their physical counterparts, Gartner VP Neil MacDonald predicted in a presentation at Gartner’s October 2007 Symposium/ITxpo.

But much of the discussion about virtualization security has been flawed to date, says security expert Chris Hoff, because people often frame the discussion by asking whether virtual servers are more or less secure than physical ones.

That’s the wrong question, says Hoff, who blogs frequently on this topic and serves as chief architect for security innovation at Unisys. The right question, he says, is “Are you applying what you already know about security to your virtualized environment?”

“People get wound up about theoreticals…when in reality there’s a clear set of things you can do today,” Hoff says. Certainly, virtualization does introduce some new security concerns, but first things first, he says: “We have to be pragmatic. Let’s make sure we architect the virtual network as well as we architect the physical networking.”

As an example, he points to a virtualization management tool such as VMware’s VMotion, which is helpful for moving VMs around in times of machine trouble, but which can also allow someone with admin rights to combine two VMs that, in the physical world, would have been carefully separated in terms of network traffic for security reasons.

Some IT organizations are making a fundamental mistake right now: They’re letting the server group run the virtualization effort almost single-handedly-leaving the IT team’s security, storage and networking experts out of the loop. This can create security problems that have nothing to do with inherent weaknesses of the virtualization technology or products. “This is a perfect opportunity to bring the teams together,” Hoff says.

“Virtualization is 90% planning,” says Burton Group’s Wolf. “The planning has to include the whole team, including the network, security and storage teams.”

But the fact is, most IT teams ran fast with virtualization and now must play catch-up. What if you missed that opportunity to plan with all your experts, and you’re starting to worry more as you expand your number of VMs and put higher-profile apps on those VMs?

“To catch up, start with a good audit of your virtual infrastructure,” using tools or consultants, Wolf says. “Then you really have to work backwards.” (Wolf suggests checking out audit tools from CiRBA and PlateSpin for this purpose.)

Here are 10 positive steps enterprises can take now to tighten virtualization security:

1. Get VM Sprawl Under Control

CIOs such as Michael Abbene, who runs IT for Arch Coal, understand the problem of VM sprawl full well: VMs take minutes to create. They’re great for isolating certain computing jobs. But the more VMs you have, the more security risk you have. And you’d better be able to keep track of all those VMs.

“We started by virtualizing very low-profile test and development boxes,” Abbene says. “Then we moved some low-profile application servers. We’ve been moving up as we’ve been successful. We understand we’re increasing our risk profile as we do that.” The company currently has about 45 production VMs, he notes, including Active Directory servers, and some application and web servers.

How do you control server sprawl? One approach: Make creating virtualized servers and VMs as disciplined as creating physical ones. At Arch Coal, the IT team is rigorous about allowing new VMs: “People have to go through the same process to get a server, whether it’s physical or virtual,” says Tom Carter, Arch Coal’s Microsoft Systems Administrator, who works for Abbene.

For this purpose, Arch Coal IT uses a change control board (made up of a cross-section of IT staffers from disciplines liker servers and storage, serving on a rotating basis) to say yes or no to new virtualized server requests. This means, for example, that people in the applications group can’t just build a VMware server and start creating VMs, Abbene says-though he’s had developers ask to do just that.

VMware’s VirtualCenter management tools as well as tools from Vizioncore can also help manage VM sprawl.

Ignore VM sprawl at your own peril, says IDC’s Elliott: “VM sprawl is a huge problem, causing lag times in the ability to manage, maintain performance and provision,” he says. Also, unexpected management costs will arise if your number of VMs gets out of hand, he adds.

2. Apply Your Existing Processes to the Virtual Machines

Perhaps the sexiest aspect of virtualization is its speed: You can create VMs in minutes, move them around easily, and deliver new computing power to the business side in a day instead of weeks. It’s fun to drive fast. But slow down long enough to think about making virtualization part of your existing IT processes, and you will prevent security problems in the first place, says IDC’s Elliott. You will also save some management headaches later.

“Process is important,” he says. “Think about virtualization not just from a technology standpoint but from a process one.” If you’re using ITIL to guide your IT processes, for example, think about how virtualization fits into that process framework, Elliott advises. If you’re using other IT best practices, look at how virtualization fits into those processes.

One example: “If you have a server-hardening document (prescribing a standard set of security and setup rules for a new server),” Hoff says, “you should do the same set of things to a virtual server as to a physical one.”

At Arch Coal, Abbene’s IT team does just that: “We take our best practices for securing a physical server and apply those to every VM on the box,” Abbene says. Steps like hardening the OS, running antivirus on every VM and ensuring patch management, keep those virtual boxes in tune with the same procedures used on physical ones, he says.

3. Start With Your Existing Security Tools, But Be Critical

Do you need a whole new suite of security and management tools for your virtualized environment? No. Starting with your existing set of security tools for the physical server and network world and applying them to the virtual environment makes sense, says Hoff. But do press your vendors to tell you how they’re keeping up with virtualization risks, and how they’ll integrate with other products going forward.

“There’s a false sense of security in relation to adopting physical tools for the virtual environment,” IDC’s Elliott says. At the same time, he adds: “It’s very early in the market,” for new security tools designed with virtualization in mind. That means you must press your legacy and potential startup vendors a little harder than usual.

“Don’t assume the platform-level tools (such as VMware’s tools) are good enough for you,” Elliott says. “Look at the startups and the legacy management vendors. Press those legacy vendors to do more, and provide guidance for them.”

Jim DiMarzio, CIO at Mazda North America, follows this strategy in his enterprise. Like Arch Coal, Mazda NA runs VMware’s ESX Server 3 software at the core of its virtualized servers and has been ramping up its number of VMs recently. DiMarzio says he expects to have about 150 production VMs running by March 2008. He’s using the virtualized servers for Active Directory servers, print servers, CRM application servers and Web servers-the last being a mission-critical app since Mazda uses these Web apps to serve information to all its dealers, DiMarzio says.

To secure these VMs, DiMarzio decided to continue with his existing firewall and security products, including IBM’s Tivoli Access Manager, Cisco firewall tools, and Symantec’s IDS monitoring tools.

At Arch Coal, Abbene and his team are sticking with the security tools they’re already using, while also investigating tools from startups BlueLane and Reflex Security. “The [legacy] security and change vendors are trying to work hard to catch up and they’re behind,” Abbene says.

BlueLane’s VirtualShield product for VMware, for instance, claims that it can protect virtual machines even in cases where certain patches are out of date, as well as automatically scanning for possible problems, updating problem areas, and protecting against some remote threats.

Reflex Security’s Virtual Security Appliance (VSA), which Hoff describes along with BlueLane’s software as one of the few emerging products worth attention right now, essentially serves a virtual intrusion detection system (IDS), adding a layer of security policies inside the physical boxes where the VMs live. It could help block a hypervisor attack, among other possible future troubles, Abbene’s team figures.

Abbene says his IT group has also discussed adding a second internal firewall to further isolate the VMs, but he’s concerned there might be a performance impact on the virtualized applications.

IDC’s Elliott cites a few other virtualization security tools worth examining: PlateSpin, known for physical-to-virtual workload conversion tools and workload management tools; Vizioncore, known for file-level backup tools; Akorri, known for performance management and workload balancing tools; and storage firm EqualLogic, recently acquired by Dell and known for iSCSI storage-area network (SAN) products optimized for virtualization.

4. Understand the Value of an Embedded Hypervisor

Maybe you’ve read about “embedded” hypervisors already, but if you haven’t, it’s a term that IT leaders should understand. The hypervisor layer on a server serves as a foundation for housing the VMs. VMware’s recently-announced ESX Server 3i hypervisor, designed to be very slim (32MB) for security reasons, uniquely does not include a general purpose OS. (And no OS means no OS maintenance chores.)

Some hardware vendors such as Dell and HP have recently said that they’ll ship embedded versions of this VMware hypervisor on their physical servers. In basic terms, an embedded hypervisor is safer because it’s smaller, says IDC’s Elliott. “The larger the code base, the larger the opportunity for breaches,” he says. “This becomes part of your architecture decision.

Embedded hypervisors will be a big trend going forward, Elliott says, and you can expect to see them from most server vendors, as well as some companies that haven’t played in this space before. Phoenix Technologies, a market leader in the BIOS software field, recently announced that it’s getting into the hypervisor game, starting with a product called HyperCore: It’s a hypervisor for desktop and laptop PCs that will let users turn on the machine and use a basic Web browser and e-mail client without waiting to boot Windows. (HyperCore will be embedded in the machine BIOS.)

Competition and innovation in the hypervisor market would be good for enterprises, Hoff says. The end result could be companies slugging it out to deliver the slimmest, smartest hypervisor software.

“Whether it’s Phoenix or someone else, there’s a very interesting battle of these hypervisors becoming the next great OS,” Hoff says.

A smaller attack surface isn’t the only benefit of an embedded hypervisor. Mazda’s IT group is looking forward to upcoming Dell servers with embedded hypervisors for VMware ESX server, says Kai Sookwongse, IT systems manager, LAN/Server for DiMarzio at Mazda. “One of the features we’re waiting for with Dell’s embedded ESX is all the VM images can be on the SAN,” Sookwongse says. “When we start up the server, it can boot up from the image on the SAN.” This centralized administration and security and also means Mazda could order a server without a disk if it wants, for physical security concerns, he notes.

5. Don’t Over-Assign Rights to VMs

Remember that when you give admin-level access to a VM, you give access to all the data on that VM. Think critically about what kind of accounts and access your staffers in charge of backup tasks need, Burton Group’s Wolf advises. Compounding the problem, some third-party vendors will actually give outdated advice with regards to VM security around storage and backup issues, Wolf adds. “Some vendors are not even following VMware’s best practices for VMware Consolidated Backup themselves,” he says.

Arch Coal makes it a point to limit admin access to its VMs overall, says Paul Telle, information security administrator, noting that his security colleague Tom Carter and Carter’s boss are among a very small group with those rights.

Application developers get minimal access. “Our application people have access to a share, or the minimum access…not access to the OS,” Carter says. This helps control VM sprawl while increasing security.

6. Watch How You Provision Storage

Some enterprises are over-provisioning storage on SANs today, says Wolf. It’s not that you’re provisioning too much storage overall; it’s that you may be letting the wrong VM’s share a part of the SAN, he says.

If you’re working with VMotion, VMware’s tool for moving VMs around, you’re assigning some zoned storage in SANs. But you may want to make that storage assignment more granular, as you would in the physical world, Wolf advises. Looking forward, endpoint ID virtualization-a technique that lets IT assign storage to just one VM-is an option worth investigating, Wolf says.

7. Ensure Good Isolation Across Network Segments

As enterprises go virtual, they shouldn’t ignore security-related network traffic risks. But some of these risks can inadvertently be overlooked, especially if IT leaders fail to bring networking and security staffers to the table while doing virtualization planning. “A lot of organizations simply use performance as the metric of how to consolidate,” Wolf says. (When evaluating which application servers to co-locate as VMs on one physical box, IT teams tend to first focus on how performance-hungry those application servers will be, since you want to avoid asking any one physical box to bear too much load.) “They forget because of security restrictions on network traffic that they shouldn’t locate these VMs together,” Wolf says.

For example, some CIOs are deciding not to allow any virtualized servers in the DMZ (also known as demilitarized zone, the subnetwork that houses external services to the Internet, like e-commerce servers, adding a buffer between the Net and the LAN).

If you do have some VMs in the DMZ, you may want them on physically separate network segments from some of your other systems, say a critical Oracle database server, Wolf says.

At Arch Coal, the IT team thought about the DMZ from the start, Abbene says.

They’ve deployed virtual servers on the internal LAN but nowhere public facing. “That was a key early decision,” Abbene says. For example, the company has some secure FTP servers and some servers doing lightweight electronic commerce in the DMZ; it has no plans to introduce VMs there, he says.

8. Worry About Switches

When is a switch not a switch? “Some virtual switches behave like a hub today: Every port is mirrored to all the other ports on the virtual switch,” Burton Group’s Wolf says. Microsoft Virtual Server, in particular today, presents this problem, Wolf says. VMware’s ESX Sserver does not, nor does Citrix XenServer. “People hear the term ‘switch’ and think isolation exists. It really varies by vendor,” Wolf says.

Microsoft has said the switch issue will be addressed in Microsoft’s upcoming Viridian server virtualization software product, Wolf adds.

9. Monitor for “Rogue” VMs on Desktops and Laptops

Servers are not your only worry. “The greatest threat is on the client side-rogue VMs,” Burton Group’s Wolf says. What’s a rogue VM? Remember, Wolf says, your users can download and use a free program like VMware Player, which lets a desktop or laptop PC user run any VM created by VMware Workstation, Server or ESX Server.

Many users now like to use VMs on a desktop or laptop to separate pieces of work, or work and home-related activities. Some people use VMware Player to run multiple OSes on the machine; say using Linux as a base OS but creating a VM for running Windows apps. (IT teams also can also use VM Player to evaluate virtual appliances-software products shipping configured as a VM.)

“Often times, those VMs are not even at the right patch level,” Wolf says. “Those systems get exposed to your network. And now all of these unmanaged OSes can float around.”

“There’s a lot of risk you’re adding there,” Wolf says, noting that the machines running rogue VMs could spread viruses-or worse-to your physical network. For example, he says, it would be very easy for someone to load up a DHTP server to give out fake IP addresses. That’s effectively a denial of service attack, he notes. At the very least, you’re going to waste IT resources trying to track down the problem, he says. “It may even be simple user error introducing services to the production network.”

How can you prevent against rogue VMs? You should have controls around who gets VMware Workstation, for starters (since it’s needed to create the VMs). IT can also use a group security policy to prevent certain executables from running, such as those needed to install VM player, Wolf notes. Another option: Do periodic auditing of user hard drives. “You want to look for machines with VMs and flag them for follow up by IT,” he says.

Has this become yet another point of contention between users and IT, where savvy users want to use VMs at work the same as they’re doing at home? Not yet, Wolf says. “IT departments for the most part have ignored it,” Wolf says.

If you do want to allow VMs on user machines, tools such as VMware’s Lab Manager and other management tools can help IT control and monitor those VMs, he notes.

10. Remember Virtualization Security at Budget Planning Time

“Make sure to allocate budget for virtualization security and management,” IDC’s Elliott says. You may not need to break it out in your security budget, Arch Coal’s Abbene notes, but your security budget overall had better have enough funds for it.

Also, be careful of security costs as you do virtualization ROI calculations. “You may not see a reduced spend in security,” just by virtualizing more and more servers, Hoff notes, because you will need to apply some of your existing security tools to every VM that you create. If you don’t anticipate this expense, it could eat into your ROI.

According to Gartner, it’s a common mistake right now. Through 2009, some 90 percent of virtualization deployments will have unanticipated costs, such as security costs, affecting ROI, according to a presentation by Gartner VP Neil MacDonald at Gartner’s October 2007 Symposium/ITxpo.