Experts divided over rootkit detection and removal

The detection and eradication of rootkits — the software code increasingly used to hide malware or adware — is either fairly simple or nearly impossible, depending on which security expert is bringing up the topic.

This often striking difference of opinion is certain to confuse corporate security managers and systems administrators who have an interest in defending against rootkits hiding on desktops, servers and databases. While there are few software products promising rootkit detection and removal today, more vendors are stepping up to take a swing at it.

Even the more optimistic security firms offering tools for rootkit detection and eradication caution it can be a little tricky wiping out stealth code that can hook into the operating system to hide backdoors, worms or running processes.

“Some people say, in order to eradicate a rootkit, you should reinstall the whole system," says Mike Stahlberg, research manager at F-Secure, one of the few security vendors to offer a desktop rootkit detection and removal tool.

F-Secure considers a system purge unnecessary because its Windows-based tool, called BlackLight, detects and removes rootkits in worms and spyware.

“The majority of rootkit cases out there can be disinfected using BlackLight by renaming the rootkit files," Stahlberg says in describing BlackLight’s disinfecting technique.

Disinfect, at a cost

The main difficulty in using BlackLight — offered as a free beta tool or as part of the commercial F-Secure Internet Security 2006 suite — is that people sometimes have a hard time renaming the files. That’s because rootkits can hide operating system files and users could rename the wrong files, Stahlberg says.

BlackLight isn’t 100% perfect, Stahlberg acknowledges, and if people have trouble using it, F-Secure will help them find a rootkit manually. If that doesn’t work, then rebuilding the system because of a rootkit infection will probably necessary.

Other researchers say rootkit detection may be viable but removal is not. Once rootkits have hooked into operating systems, the stealth code will likely be impractical to remove because doing so will damage the operating system.

“The inline function hooks [in rootkits] are very similar to Microsoft’s hotpatching," says James Butler, CTO at start-up Komoku, which is developing software-protection products aimed at combating the rootkit menace. “Part of the original function is overwritten with an instruction that causes a change in execution."

Butler, who spoke on the topic at the recent Black Hat conference, says Komoku’s research has identified several types of hooks — system call hooks, IDT hooks, IRP table hooks — and trying to eradicate a rootkit from an infected computer is often impossible.

A whole new problem

In any event, removing a rootkit “may mean opening up a new hole," Butler says. “A lot of these rootkits basically put the machine into a very bizarre state."

One thing that researchers do agree on is that the cloaking capability of rootkits is a growing threat as rootkit functionality increasingly shows up as part of spyware, backdoors and Trojans such as Haxdoor, Ginwui, HaxSpy, Gurong, Maslan and many more.

At Komoku, “we came up with the word 'rootware’ to describe rootkits and spyware combined," Butler says. “When a rootkit is hooked into a worm, you could lose your network pretty quickly."

Rootkit techniques can be used to replace system drives, create specialized registers and layered drivers. A total hijacking of the machine can be done through virtualization, which security firm Coseinc's researcher Joanna Rutkowska demonstrated in her Blue Pill rootkit for Vista at Black Hat. No one has yet claimed a way to even detect Blue Pill — not even its inventor, Rutkowska.

Ambitious protection

Some of the traditional antivirus software vendors are becoming more ambitious in taking on rootkits. BitDefender introduced a Rootkit Removal Beta last month, and McAfee plans rootkit detection and removal in its enterprise antivirus/antispyware software before year-end.

BitDefender spokeswoman Carmen Nita says the BitDefender Rootkit Removal tool is designed to detect files and processes that have been hidden by rootkits.

“Rootkits might hide viruses, Trojans, backdoors, spyware and other types of malware," she says. “The BitDefender tool can clean the infected computer by renaming the hidden files, thus un-hiding them."

She said BitDefender’s antirootkit tool should be used in conjunction with the BitDefender Antivirus and Antispyware modules by performing an on-client scan of the respective system after the files have been uncovered. The BitDefender antirootkit tool will be included in all BitDefender desktop products, starting next month.

David Marcus, security researcher and communications manager for McAfee’s Avert Labs division, says McAfee’s current slate of antimalware software can stop and eradicate rootkit-based worms and spyware through scans before they’ve embedded into the operating system.

But the McAfee products today can’t reliably detect and eradicate rootkits after they’ve hooked into the system APIs, Marcus says. “This is much more difficult on the running system," he says.

“Later this year we’ll release antirootkit software as part of our enterprise antivirus," marcus says. The successful detection and eradication of rootkits “is an area in which we’re definitely the most challenged," he adds.

Symantec also said it plans to add rootkit-detection capability to its Norton antivirus products to look for rootkit-hidden malware.

Oliver Friedrichs, director at Symantec Security Response described how this would work: "We use our own file system driver to bypass the operating system APIs," said Friedrichs. If the security software discovers what would appear to be a rootkit-hidden malware, it will send a copy of it back as a sample to the Symantec lab for analysis. If the sample is determined to be malware that should be eradicated -- and can be eradicated safely -- Symantec will send out a detection and eradication signature to its customer base.

"We can't just go deleting files and removing them," said Friedrichs. "It could end up damaging the system."

While rootkits are more commonly associated with desktops than databases, some security experts caution that savvy attackers install rootkits on databases, too.

“A hacker can hide his presence in the database," said Alexander Kornbrust, CEO of Red-Database-Security, which specializes in Oracle security, speaking on the topic during the Black Hat conference. “An attacker can hide database jobs, creating a database job running at midnight."

Kornbrust said he viewed the use of checksum tools, such as Tripwire, as the best means to identify rootkits. “They’re difficult to find," he says.

Copyright © 2006 IDG Communications, Inc.

The 10 most powerful companies in enterprise networking 2022