Security experts now believe that trojan, spam and malware protection software cannot adequately prevent system compromise by increasingly sophisticated rootkits.

Rootkits are used to conceal the presence of trojans, hacker backdoors, and botnets by cloaking their files and processes through modifying the output of common operating system routines. They grant administrator access to a system after a hacker installs them typically through obtaining user level access by exploiting known vulnerabilities.

Intelligent Security Research Services analyst James Turner said rootkits will be increasingly used in highly targeted attacks as they become more sophisticated and form a critical part of hacker arsenals.

"We are going to see rootkits used in highly targeted attacks where hackers will source, for example, a CFO's operating system and the typical applications they use, and then find a specific vulnerability based on these which allows a rootkit to be inserted," Turner said.

According to Turner, information security infrastructure is heating up through increased education and simulations of information security warfare, however he said the biggest problem is getting people who have been hacked to warn the public about it.

Rootkits can be classified as; kernel-mode, which intercept kernel interface calls and alter OS kernel data to conceal rootkits from process lists; persistent, which use the system registry to execute on boot; user-mode, which can use keyloggers and infect or masquerade as OS commands; and memory-based, which rely on manual user execution to operate.

The most critical exist in unpatched exploits in common applications, according to Chris Gatford, senior security analyst at penetration testing firm Pure Hacking.

"Microsoft Word has an unspecific exploit that has been unpatched for 47 days; if I were a hacker I would certainly target these kinds of exploits because the scope is so wide," Gatford said.

"Hackers are using the same spyware model but are distributing them with the next-level of rootkits."

Security firm Markets-Alert director Jeff McGeorge said rootkits are the frontline arsenal of hackers and are too sophisticated for rootkit revealers, and virus and spam protection software to combat.

For more enterprise computing news, visit Computerworld. Story copyright Computerworld, Inc.