Patching critical servers is Russian Roulette

One surprising finding from Nemertes’ recent security research is that, the more critical a server, the longer it takes to get patched. Not only are most critical servers patched manually (slower but safer) but patches also need to be subjected to rigorous testing so as not to cause disruption. As a result, security professionals are faced with an uncomfortable dilemma: leave the server exposed to hackers or expose it to potentially damaging patches.

The more thoroughly patches are tested for conflicts, the longer the servers remain exposed. This is not a good risk-mitigation strategy or an easy choice to make.

Virtualization and inline patching may be solutions to this dilemma.

Vendors are obviously loath to release a patch that may cause conflicts or that crashes on a server. But interactions between different components of a server or between multiple servers can create infinite configurations to test – an impossible task for vendors. Not an easy task for corporate IT either, as an unexpected interaction from some piece of software could bring down a production server. For system administrators, the best way to do this is to create an exact replica of the production system to run tests on. Possibly a very expensive solution, if the critical server is on expensive hardware.

This is where virtual servers might be a helpful tool. With VMware, Xen or Microsoft’s Virtual Server, servers can be cloned and deployed on a virtual platform to test the patches. This allows administrators to perform tests on an identical system with no risk to the production server. Of course, this requires that the critical server be on a virtual machine to start with; otherwise the virtual test machine configuration will be different from the non-virtual critical server. Also, a system under stress from production traffic behaves differently than an idle system, so if the patch causes performance problems, these might go unnoticed.

To test more-complex environments, you’d have to simulate lots of servers and user machines in a large virtual network. IT managers can build such a “lab in a box” using virtual servers and management software from the virtualization vendors mentioned above. A complementary approach is offered by Akimbi’s Virtual Lab Automation system, which allows IT to manage and deploy dozens of virtual machines in pre-configured virtual networks.

Another approach is to not patch the critical servers immediately, but instead to protect them via an inline security device on the network. If such a device can eliminate attacks against the vulnerability on the server, administrators can take more time testing patches without fear of exposure. One approach would be to use intrusion prevention systems and firewalls with custom signatures and access control lists that protect a specific un-patched exposure. Creating the rules and signatures requires highly specialized skills, though. The BlueLane PatchPoint appliance applies inline patching to network traffic.

Whatever approach you take there is one thing for certain – leaving servers un-patched or patching in haste will only lead to greater problems.

NOTE: Don’t forget to check out the Data Center World spring conference in Atlanta, March 19-23. The conference is organized by AFCOM, the leading association for data center professionals.