Cordoning off physical servers from one another in a data center is difficult enough. Trying to isolate virtual servers running on a single hardware box is even more complicated.

Mercy Medical Center in Baltimore ran into the problem when it deployed network access control (NAC) gear from ConSentry Networks. As part of that move, the company ripped out hardware servers that were running multiple applications and replaced them with virtual servers, each running one application. While this made the NAC implementation more application-centric than server-centric, it introduced security complications.

If you take into account the prerequisites of a good server-to-server firewall -- in particular, the needs for speed and for intrusion-prevention-system (IPS) support -- deploying security in a virtual environment might make more sense. Some vendors have found a niche there. Reflex Security, for example, offers a virtual firewall-IPS appliance packaged as a virtual machine for controlling and securing communications among virtual servers. Because the servers and the firewall communicate within the box, they don't pay the overhead of going out to an external switch. That reduces the IPS performance hit.

Read related story: Four virtualization security companies to watch

In addition, because these features are provided in software on a virtual machine, it costs less to deploy than a hardware-based firewall-IPS. "You don't need to start moving wires around every time you want to secure a different server," says Nand Mulchandani, senior product director at VMware. "It's all done in software, at a much lower cost."

Other companies, such as Blue Lane Technologies, also make virtual-machine-based security appliances, and even firewall stalwart Check Point Software says it has not ruled out the idea of building its own virtual firewall-IPS appliance.

Virtual security appliances are a step in the right direction, but some users say the performance is still not acceptable. "We've been playing with Reflex and others, but they can't seem to pass packets fast enough," Rein says. What's really needed, he says, is for security to run "somewhere within the hypervisor itself."

To that end, VMware is opening the hypervisor to trusted security partners via a new program called VMsafe. Security vendors will get the same visibility as the hypervisor into the operation of a virtual machine, including the memory, CPU, disk and I/O systems, VMware says. That way, they can identify and eliminate malware, such as viruses, Trojans and keyloggers, before it can harm the machine or steal data.