Automated tools help, but don't cure all patch ills

Successful patch management requires planning, processes, testing, integration of multiple security tools.

Davidson Healthcare in Lexington, N.C., got a wake-up call recently when a vulnerability scan discovered that the network was missing more than 4,000 cumulative patches on its 30 servers and 500 workstations.

"I had this bad feeling about those patches," says Kevin Buchanan, who runs Davidson's 10-person IT department. "The problem was that we couldn't keep up with the volume of Microsoft's patches. They were releasing them way too often, there were too many of them, and a staff like ours had no way to manage this. Yet if we didn't, we knew we'd be at risk."

Buchanan quickly secured funding for automated patch management software from Shavlik Technologies.

Adam Hansen, manager of security at Chicago law firm Sonnenschein Nath & Rosenthal, has a similar tale. With nearly 2,000 servers, desktops and laptops spread across 11 U.S. offices, Hansen knew his firm had to get an automated patching solution, and quick.

"About a year and a half ago, this all came to a head," Hansen says. "We did a vulnerability assessment and found we were only about 15% in compliance in terms of patching." He began looking for something that could automate patching and provide real-time reporting.

Sonnenschein Nath & Rosenthal went with PatchLink, another of the pure-play patch vendors that were first to market with automated patch tools.

"The old way of doing things, deploying patch by patch, is not effective in the long term," Hansen says.

There's widespread agreement on that point among enterprise IT executives, analysts and vendors. Any lag between detecting a vulnerability and correcting it leaves an organization open to attack. And with their own set of automated tools, hackers can strike almost as soon as new vulnerabilities are discovered.

But users are finding that even with automated patch management tools, patching can be a complicated, laborious and often problem-causing process because patches have been known to break applications.

First things first

According to Meta Group analyst Peter Firstbrook, the first step is a network assessment. "My biggest piece of advice to customers evaluating patch management solutions is to take a step back and evaluate your own organization first," Firstbrook says. "So much of what has to be done is process and procedure."

Even the best tools won't save you if you don't have the right processes in place and the people and computing resources to back them up, he adds.

Davidson used to patch machines when they needed to be serviced or when a new image was pushed out. Now the patching is an ongoing part of IT maintenance, typically performed in off-peak hours, Buchanan says.

The patching path not taken

The issue for IT executives is not whether to automate, but which product to buy. There are the pure-play patch products from Shavlik, PatchLink and others. There are patch tools that are part of larger software suites that include life-cycle management, change management, security management and configuration management. Plus, there are Windows-only tools from Microsoft.

Angela Triola, an infrastructure analyst at ENT Federal Credit Union in Colorado Springs, tackles patching with Enterprise Configuration Manager (ECM) from Configuresoft. ECM performs a variety of functions, including vulnerability assessment, change management, compliance, remediation and patching. Triola says everything from finding patches to testing to deployment to verification has now become manageable.

"We rely heavily on Microsoft," Triola adds, "so the fact that Configuresoft works well with Microsoft was very important to us." She says that Microsoft's own patching advances, including the forthcoming Windows Update Services (WUS) patch management software, will not change the need for third-party tools.

Other IT professionals agree. "Essentially, even with [Software Update Service], WUS or [Systems Management Server], you're still taking a patch-by-patch approach," Hansen says. "You still need the entire platform to see this through from start to finish."

Hercules provides patching power

At Sonnenschein Nath & Rosenthal, Hansen has migrated from PatchLink to Citadel's Hercules, which performs network discovery, vulnerability assessments and ongoing compliance audits.

"What I like about Hercules is that it frees you from worrying about the fine-grained details of patching. You define a baseline, and it becomes the product's responsibility to get your network devices in step," Hansen says. "We needed a solution that did more than just pushing patches, which is what a lot of products on the market do."

Hansen says he likes Hercules' strong reporting and policy-enforcement features, which he argues are essential to any successful patching strategy. "Even if a properly patched image is pushed out to a new server or desktop, who's to say that something doesn't get changed along the way, a vulnerable port opened, an unpredictable service turned back on?" Hansen asks.

An agent-based solution, Hercules keeps tabs on those devices, noting any deviation and, if the vulnerability warrants it, quarantining devices that are not compliant. With more mobile devices entering the network, this feature is critical.

Hansen says Hercules enables a lightning-quick response to vulnerabilities, with most of the patches tested and pushed out in less than 24 hours. Moreover, even with a large mobile user base, the firm's patch-compliance ratio is close to 80%, with the network getting more airtight each day.

The process of integrating Hercules wasn't problem-free, however. "We had issues with getting the Hercules agent to work with certain images, but Citadel's customer support was responsive and worked with us to resolve those issues," Hansen says.

EDS opts for Opsware

Larry Lozon, the vice president of hosting services at Electronic Data Systems (EDS), has 70,000 distributed servers and a massive number of other devices. Not only that, but EDS also now requires that any device that enters its network must have end-to-end configuration and patch management.

EDS turned to data center automation vendor Opsware, which provides server management, provisioning, configuration, change management and patch management. Lozon adds that the next problem on EDS' patch and configuration management radar screen is mobile devices.

Patch payoffs

The payoff from patch management tools can show up almost immediately. According to Davidson, it realized ROI with a single patch cycle. "If you consider our previous process, where we had two technicians walking from machine to machine and patching manually, the cost of the Shavlik product was equal to what their salaries would have been during that period of time," Buchanan says.

Another Shavlik customer, Indiana University, agrees. "The cost savings was so immediate and obvious, that we didn't even bother to quantify it in order to justify the purchase," says Jim Kippenbrok, manager of local support provider services for the Bloomington school.

Hansen estimates that Citadel's Hercules saved about the equivalent of a midtier full-time IT employee's salary, or about $60,000. "And this doesn't include the savings in terms of reduced risk," Hansen adds.

Automation is the way to go

Customers say they're generally happy with their patching tools, but there are still issues of concern, such as the ever-shrinking window between when a patch is issued and when it needs to be deployed, the need for good internal testing, interoperability issues, the importance of nailing down a workable patch cycle and the occasional customer service problem.

Because many companies have homegrown or niche applications running in a mission-critical capacity, the patching vendor must work with them to ensure that the patches won't cause more harm than good.

However, when assessing the risk of not patching, most customers agree that patches have become so stable that the risk of leaving a vulnerability unchecked vastly outweighs any interoperability risks.

"Let me go out on a limb here and say that if you are an IT manager and you ignore patch management, you are negligent," Buchanan says.

7 steps for an effective patch management process
1. Learn what is on your network via a scanning or network-discovery tool (which are often provided in patch management suites).
2. Prioritize. Understand the criticality of patching in terms of business risks. That is, if assets aren’t terribly critical on a particular desktop, address that device only after your mission-critical assets have been protected.
3. Allocate the computing resources and people needed to effectively handle the task.
4. Determine an appropriate workflow for patching; that is, attempt to patch during off-peak hours when normal business flow won’t be disrupted.
5. Find an automated method for pushing out patches to your network devices.
6. Utilize reporting and validation tools. If you can’t verify that your devices have been properly patched, you can’t be sure you are secure.
7. Repeat this process as part of your larger IT security and management process.

Learn more about this topic

Vance is a freelance technology writer and the president of Sandstorm Media (www.sandstormmedia.net). He focuses on trends in wireless communications, next-generation networking, security and Internet infrastructure. He can be reached at jeff@sandstormmedia.net.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2005 IDG Communications, Inc.