Experts divided over rootkit detection and removal

The detection and eradication of rootkits — the software code increasingly used to hide malware or adware — is either fairly simple or nearly impossible, depending on which security expert is bringing up the topic.

This often striking difference of opinion is certain to confuse corporate security managers and systems administrators who have an interest in defending against rootkits hiding on desktops, servers and databases. While there are few software products promising rootkit detection and removal today, more vendors are stepping up to take a swing at it.

Even the more optimistic security firms offering tools for rootkit detection and eradication caution it can be a little tricky wiping out stealth code that can hook into the operating system to hide backdoors, worms or running processes.

“Some people say, in order to eradicate a rootkit, you should reinstall the whole system," says Mike Stahlberg, research manager at F-Secure, one of the few security vendors to offer a desktop rootkit detection and removal tool.

F-Secure considers a system purge unnecessary because its Windows-based tool, called BlackLight, detects and removes rootkits in worms and spyware.

“The majority of rootkit cases out there can be disinfected using BlackLight by renaming the rootkit files," Stahlberg says in describing BlackLight’s disinfecting technique.

Disinfect, at a cost

The main difficulty in using BlackLight — offered as a free beta tool or as part of the commercial F-Secure Internet Security 2006 suite — is that people sometimes have a hard time renaming the files. That’s because rootkits can hide operating system files and users could rename the wrong files, Stahlberg says.

BlackLight isn’t 100% perfect, Stahlberg acknowledges, and if people have trouble using it, F-Secure will help them find a rootkit manually. If that doesn’t work, then rebuilding the system because of a rootkit infection will probably necessary.

Other researchers say rootkit detection may be viable but removal is not. Once rootkits have hooked into operating systems, the stealth code will likely be impractical to remove because doing so will damage the operating system.

“The inline function hooks [in rootkits] are very similar to Microsoft’s hotpatching," says James Butler, CTO at start-up Komoku, which is developing software-protection products aimed at combating the rootkit menace. “Part of the original function is overwritten with an instruction that causes a change in execution."

Butler, who spoke on the topic at the recent Black Hat conference, says Komoku’s research has identified several types of hooks — system call hooks, IDT hooks, IRP table hooks — and trying to eradicate a rootkit from an infected computer is often impossible.

A whole new problem

In any event, removing a rootkit “may mean opening up a new hole," Butler says. “A lot of these rootkits basically put the machine into a very bizarre state."

One thing that researchers do agree on is that the cloaking capability of rootkits is a growing threat as rootkit functionality increasingly shows up as part of spyware, backdoors and Trojans such as Haxdoor, Ginwui, HaxSpy, Gurong, Maslan and many more.