Practical patch management

Managing security patches leads to many a stormy day for IT personnel — and while Microsoft is a natural lightning rod because of Windows' ubiquity, the problem touches on all operating systems and applications.

Patch management quickly can flatten IT departments as they struggle to find the time and resources needed to get the problem under control. The cost implications are huge. Aberdeen Group estimates the tab for patch management for U.S. businesses at $2 billion per year.

"Amazing" is how one technologist characterized the frustration level he has had in dealing with Windows patches. "Getting a good collective knowledge of where to find patches was a challenge," says Andrew Nielsen, a senior technologist who works for Raytheon Technical Services in support of NASA Ames Research Center's Federal Information Processing Service contract.

Once Nielsen began using a patch-tracking tool from Shavlik Technologies, the time, labor and frustration that went into managing Windows patches "dropped by orders of magnitude," he says. "Patching tasks that would take two systems administrators the better part of a day are now accomplished in 30 minutes."

Without patch-management tools, many network administrators essentially tracked patch status in their heads, fixing holes on the fly. But in the past two years, the sheer complexity of networks and number of patches have rendered this approach ineffective.

Ironically, says Eric Hemmendinger, an Aberdeen analyst, "The more time you spend on patch management, the less you focus on security."

Now that a slew of patch-management products and services are available, more systematic approaches are at hand. Users, analysts and vendors who share practical advice on how to investigate, prioritize and deploy patches start with this basic fact: Patch management is a subset of both change and risk management. Understanding that is critical, they say.

Risk and change

Before establishing a patch-management plan, you must meet with business executives and decide the corporate risk tolerance regarding patches. For example, to maximize security at the expense of productivity, you would shut down production systems every time a vendor issued a relevant software patch. You wouldn't bring your systems online again until you had downloaded the patch, tested it extensively and deployed it. Of course, in practice such extreme measures are unheard of. The point is, you must view patch management as one component of corporate risk management.

Ditto for change management, which is a systematic approach to keeping track of IT changes. For companies with strong change-management processes, patch management is simply one more pigeonhole.

Bill Anderson, lead product manager for Microsoft's Systems Management Server, recounts how one customer drastically cut the time it took to test patches by revising change-management processes. "A customer told me the other day it took him four days to test a patch on all his Windows configurations. That's too long. They were able to look at their [change-management] processes and cut that to one day," he says.

Still, Anderson notes, one day isn't satisfactory either. "The real goal is to get that to three or four hours," he says.

The problem is too few companies have such processes in place. CERT says as many as 80% of corporate computer outages are caused because servers have been improperly configured.

But patch management provides a good entry to broader change management, experts say. Consider creating a patch-management plan as a dress rehearsal for a full-bore change-management policy, they suggest.

Invite to inventory

Users, analysts and vendors agree that a crucial step in taming the patch-management storm is to take an inventory of your entire IT infrastructure — a daunting task, no doubt, especially for large businesses. Still, it's necessary. "If you don't have your entire environment mapped out, you can't know what you need," says Ralph Logan, director of the VigilEnt Intelligence Threat Analysis Laboratories at PentaSafe Security Technologies, an integrated security management company.

Plus, mapping your infrastructure will help uncover who's patching what, says Terry Grogan, manager of information systems security at Lancaster General Hospital in Pennsylvania. "You find different departments do different things with their patching," she says.

Your inventory should tell you:

• The systems that make up your environment.
• Their operating systems and applications, including version.
• What patches have been applied.
• Ownership and contact information (important in large and far-flung companies).
• Any known but unpatched threats to your systems and vulnerabilities in them.

Once you've gathered this data, update it frequently and make it available to all who might need it — network executives, security managers and system administrators. And once that Herculean task of inventorying the enterprise infrastructure is completed, run quarterly updates of divisions or business units on a rolling basis, analysts advise. This will keep the workload manageable.

Triage and test

With a firm grip on your IT infrastructure, you're ready to devise a patch-management policy or improve an existing one. The first step is to better assess and triage patch notifications. One way is to rely on in-house experts for assessments, users say.

When Lancaster General receives patch updates for its Windows, Unix and IBM AS/400 systems from PentaSafe, Grogan says she confers with her LAN group for server-related patches or PC group for client patches. The group helps her determine whether the hospital needs the patch and, if so, how urgently.

Another way is to keep your cool. Don't act before careful consideration of the blanket recommendations issued by vendors and organizations such as CERT and Sans.org. Groups such as these tend to issue broad recommendations, erring on the side of safety.

When news spread in August of a vulnerability in many vendor implementations of the SNMP — a widely deployed protocol used to monitor and manage network devices — many corporate users went scrambling to fix their systems. The initial reaction was that all Windows machines needed the patch that Microsoft released, says Randy Streu, vice president of product management at Configuresoft, a configuration-management and software-patch vendor. But a careful reading of the bulletin showed the patch was necessary only on machines with SNMP running, he adds. That reduced the number of vulnerable machines to as few as 5%.

"You've got to take a hard look at the patch and decide if it applies to you. In the case of the SNMP [vulnerability], I just blocked SNMP traffic coming into my firewall," Grogan says, explaining this decision spared the hospital the time and expense of applying a patch to hundreds of Windows desktops.

Once you decide to apply a patch, testing is a must. Streu recommends trying a patch in a dedicated lab first, then rolling it out to 10% of your least-critical servers if it succeeds and doesn't introduce other problems. "That way, if it fails, it's relatively easy to roll back," he says.

While conceding that the 10% idea is nice, some users say time pressures and real-world workloads force them to move patches straight from limited tests to the production environment. Nielsen suggests setting up a patch test domain on a private network. If the patch is successful in the test environment and no issues arise, it can be rolled out to a production environment, he says.

An IT manager at a large insurance company who requested anonymity says his organization likes to run patches by its developers before putting them on the corporate network. "We push out hotfixes to our development environment to make sure they're stable. Next they go to [quality assurance]. Then they go to production," he says.

Caution is warranted. Because of its youth, the discipline of patch management has raised as many security issues as it has solved. "Vendors are generating quick solutions, and quick solutions are the enemy of security. Vulnerabilities will be introduced to the world as a result," says Ray Wagner, a Gartner analyst.

Lancaster General's Grogan agrees but accepts the inevitable. "There's no perfect solution until we don't need to put them on," she says. "But for now, I still need to manually patch my systems, and I need to know what's out there."

Ulfelder is a freelance technology writer. He can be reached at sulfelder@yahoo.com.

Buyer's Guide: Vulnerability-assessment tools
Vulnerability-assessment tools edge toward usefulness in large networks.
Network World, 02/04/02

Defending the extended enterprise
Are firewalls enough? Should you invest in a bundled security product? What type of security planning should you undertake? We explore what it takes to secure your changing enterprise in this special report.
Network World, 07/29/02

Security research center
Get the latest news, alerts, opinions, reviews and more.