How to handle patch management

The list of all-too-familiar names - Nachi, Klez, Lovsan, SoBig, 
BugBear, Swen, Blaster and Yaha - represents only a sampling of the most prevalent worms and viruses that slithered into corporate networks this fall. But they all have one thing in common: Patches were readily available before most damage had been done.

So why do these intruders continue to wreak such havoc?

Because patch management is tough.

It's tough because there are too many patches and not enough time, and because exploits to announced vulnerabilities are materializing faster. (Blaster appeared only 26 days after Microsoft reported the vulnerability.)

It's tough because clients are becoming the attack targets as much as servers, fueling faster propagation and the threat of re-infection from mobile workers reconnecting to the network.

And it's not just Microsoft vulnerabilities. Although Windows seems to get the bulk of the exploits and end-user animosity, the list of targets includes routers, switches, firewalls; Unix and Linux, too.

Patching chores likely will never go away, experts say, but there are ways to address the task proactively to minimize exposure.

"Patching is the physical process," says James Williams, information delivery manager for RBC Centura Bank in Rocky Mount, N.C. "But you have to manage that process, and to do that you need some structure."

Centura has an 11-person staff as part of a computer security incident response team that maintains what Williams calls a "very systematic and very organized" patch management process. That process utilizes inventory, change-control practices and automated deployment supported by tools from Ecora, IBM/Tivoli and others.

"I might not have enough staff, but I have processes and organization that help me cover that issue," he says.

How to patch

"We see people looking for a tool that will solve all their problems, but what you need is a process; it's not just about the tool," says Felicia Nicastro, senior network systems consultant for International Network Services, a consulting firm that kicked off a patch management service in September. Nicastro says the biggest mistake companies make is leaving out the processes, such as diligent monitoring for new patches coupled with detailed evaluation, testing, deployment and validation that a team or individual manages.

"This typically isn't a task for one person. It has to involve the security group, the operations group and the developers," she says. "So what also makes patching tough is a lack of resources."

Nicastro says companies need to have several pieces in place before a patch management process can be installed: network inventory, change management, configuration management, asset management, formalized record keeping, an understanding of costs, prioritization guidelines, and maintenance and communications plans.

"Getting a process in place can be difficult if you don't have all these pieces together," she says.

Inventory, or documenting what machines run what software, is the first step.

"This might be your biggest cost," Nicastro says. "Inventory can take some time."

Inventory ties into asset, change and configuration management. "If you track configuration then you know what's changed, and that can help with future patching," she says.

The process starts, Nicastro says, with monitoring for new vulnerabilities and available patches for everything in inventory. Once a vulnerability is identified and determined to be a threat, teams of IT, data and operations managers must work together to usher a patch through the established rollout process. A course of action and a timetable for execution, including lab testing, should be established.

"Many times companies don't have the money to support a lab or duplicate environment, but at a minimum you should try to duplicate business-critical systems, say a Web server with a database back end," Nicastro says.

After testing, distribution of the patch, implementation, exception handling, tracking and reporting need to be done.

Software and services for such tasks are available from vendors such as Altiris, BigFix, Computer Associates, ConfigureSoft, Ecora, HP, IBM, Loudcloud, Microsoft, Novell, PatchLink, Shavlik Technologies and St. Bernard Software.

Nicastro says in times when patching becomes a fire-fighting exercise, companies should quarantine the worm or virus on network segments and patch using their documented processes.

"The number of vulnerabilities, their exploits and the serious damage that they can do is why having a process is so important," she says.