Patch management burdens customers

When it comes to patch management, there’s no one-size-fits-all approach to doing a job no one wants to do: update software for new features, or the more troubling task, fixing a security hole before a hacker or computer worm finds it.

By all accounts, patching software is a disruptive, time-consuming process requiring IT departments to test any new software patch before applying it, scheduling downtime for machines to apply the patch and ensuring it doesn’t “break” applications. Patching for security purposes also means managers have to be on constant alert for news of any new holes found in vendor products. This thankless task monopolizes large chunks of IT staff time, in spite of a growing array of products and services that can track machines that need patches and automate patch downloads from vendor sites.

At any rate, many organizations say they don’t need commercial patch-management products to do the job.

“We have our own system for this,” says Anthony McBride, IT network security analyst at financial services firm Principal Financial Group in Des Moines, Iowa. “It’s a homemade system with a database of the server and applications we use for Windows, Solaris and Linux and what’s been patched. And we monitor a list of open sources, like BugTrak, for information.”

Because there are so many patches released by vendors, Principal Financial Group evaluates each one according to a risk category to determine which need to be applied immediately and which can wait for the next quarterly scheduled software maintenance. “You have to weigh the risk, and get into a lab and test that patch,” McBride says.

Commercial patch management products can either be stand-alone patch products like those from BigFix, PatchLink, St. Bernard Software and Shavlik Technologies, or the patch component of systems management products from ConfigureSoft, Ecora, IBM Tivoli and LANDesk Software.

In any event, the idea of deferring patching based on risk is a common practice, according to network executives. That’s because the number of vendor patch releases is skyrocketing as the number of newly discovered vulnerabilities increases dramatically.

“The number of software vulnerabilities has doubled every year since 1999,” says Casey Dunlevy, manager of the CERT Analysis Center at Carnegie-Mellon University, which tracks this data as part of its ongoing effort in issuing the closely watched CERT security alerts.

“Last year it was 4,200 different vulnerabilities in software products, the year before it was 2,100,” Dunlevy says. “And it looks like we’ll double it again this year.”

In organizations where Microsoft server and desktop products predominate, there’s frustration that Microsoft hasn’t made the patch management job easier.

“We need to put pressure on companies like Microsoft to build security in as a requirement,” says Steve Malphrus, CIO at the Board of Governors and director of management at the Federal Reserve System. Malphrus expressed dismay about Microsoft’s all-too-steady stream of software patches. The Federal Reserve doesn’t use commercial patch management products, but the staff monitors vendors directly and patches according to risk when a new vulnerability is discovered.

Microsoft offers three basic ways to patch its own server and desktop applications: through its Server Management System (SMS) console for pushing software updates; via System Update Services, which lets the desktop “call home” to the Microsoft Web site for patches; and through the Microsoft Baseline Security Analyzer, a free online tool for vulnerability assessment.

But customers generally aren’t impressed with Microsoft’s tools. Microsoft’s separate software teams for operating systems and applications churn out patches differently, so there’s no consistency in how they’re to be applied or how the patch is identified. Some customers say they have had rough experiences using SMS for patch management.

“We tried SMS but just couldn’t get it to work,” says Denny Cannon, PC specialist at Farm Credit Services of America in Omaha, Nebraska. “SMS wouldn’t apply the patches at the right time when we wanted it to. When we called Microsoft, they just said, “Well, that’s SMS time. It does it when it’s ready to, when it doesn’t see traffic on the network.'”

FCSA instead is relying on the LANDesk Management Suite, which keeps track of the software installed on the organization’s 950 desktops and 150 servers – all Microsoft – for patching.

At present, FCSA manually has to find the right patch at a vendor site and download it to distribute it via LANDesk to the right machines. But by year-end, LANDesk, which supports Windows, Unix, Linux and Macintosh, will add vulnerability assessment to its suite to scan for patch needs and automate the patch-download process as part of a new security service.

IT managers say complete automation to instantly download a new patch and distribute it would be the ideal, but experience leads them to believe it carries too much risk because patches can cause unexpected disruptions in applications.

“I haven’t found any tool yet that I’d be comfortable with to allow complete automation of patch management,” says Garett Redelings, systems administrator at Bio-RAD Laboratories in Hercules, Calif. The lab uses St. Bernard UpdateExpert to get word of new patches and have them downloaded and pushed out to 20 Windows servers and 120 workstations.

Redelings tests every patch first because he’s found that Microsoft patches might work fine with Microsoft applications but can cause non-Microsoft applications to shut down.

Time-Warner Cable in Raleigh, N.C., uses the Security Update component in ConfigureSoft Enterprise Configuration Manager software to tackle the patch process. George Geddis, the IT department business analyst there, says procedures and policies need to be in place for patch management because some patches, particularly from Microsoft, don’t work correctly when first issued.

Some patches also “undo” the effect of patches applied before. “Unless you take the time to do regressive testing, you could find new problems,” Geddis says.

“I’m not a big fan of automated patch updates because it can break applications,” says Pete White, security architect at M.D. Anderson Cancer Center at the University of Texas in Houston, which has a mixed server environment of Windows, Linux and Solaris. “Microsoft is not known for getting the patch right the first time.”

The hospital has deployed host-based software from Symantec called Enterprise Security Manager across 200 servers to enforce security policy, and White says the ability of ESM to take a “snapshot” of each server gives him a way to keep track of what needs patching. The hospital also uses the Symantec DeepSight Alert Service to get notification of vulnerabilities that might necessitate an immediate patch.

Although managers remain wary of automated patch installation, some patch management vendors say they are intent on providing automation as an option.

In the fall Shavlik Technologies plans to release an automatic patch management agent for Microsoft desktops and servers. Shavlik, which currently targets only Microsoft products, plans to expand support to include Oracle and Apache.

A growing number of IT managers say patching, especially to fix security holes, isn’t a tenable process and that new approaches are needed.

“You hold your breath when you’re applying patches to see if they’re breaking anything,” says Eric Beasley, senior network administrator at Baker Hill in Indianapolis, which provides hosted loan-processing applications for the banking industry. “We handle patching at the service-pack level, not the hot-fix level. Patches have to be done in a specific order or you can undo some of the patches from before.”