After security researcher Joanna Rutkowska Thursday demonstrated how it’s possible to circumvent security in Microsoft’s Vista beta software and install a rootkit called Blue Pill, Microsoft said it intends to find ways to stop both potential threats before Vista ships.

At the Black Hat conference, Rutkowska, security researcher at Singapore-based firm COSEINC, showed that she found a way to bypass the Vista integrity-checking process for loading unsigned code into the Vista kernel. Then she presented Blue Pill, a rootkit she created based on Advanced Micro Devices (AMD) Secure Virtual Machine, Pacifica.

On bypassing the signature-checking mechanism for device drivers that Microsoft is including in Vista to prevent loading of malware or unauthorized software, Rutkowska said, “The fact that this mechanism was bypassed does not mean that Vista is completely insecure. It’s just not as secure as advertised.” She added: “It’s very difficult to implement a 100% efficient kernel protection in any general-purpose operating system.”

In the second part of her presentation at Black Hat, Rutkowska, a noted expert on rootkits, detailed Blue Pill, which can hijack a computer and serve as a backdoor for attackers. While developed for Vista, Blue Pill could be adapted to other platforms as well.

Blue Pill so far has proven impossible to detect, said Rutkowska, although she was continuing research to uncover a means of detecting it. She said software-based detection mechanisms may not be effective though a hardware-based approach in the computer may provide a means of detecting or preventing a Blue Pill-based attack.

Microsoft’s director of the Windows client group, Austin Wilson, said Microsoft considers Rutkowska’s findings “legitimate” and is looking at the problem.

Wilson pointed out that Rutkowska’s code-signing bypass attack requires the attacker to start as the machine’s administrator in Vista. “If you’re running as a standard user, this wouldn’t work,” he noted. “But we’re still looking at blocking this type of attack.”

In her presentation, Rutkowska suggested a few ways Microsoft might address the code-signing bypass issue, and Microsoft intends to review them. However, Microsoft probably won’t hold up shipping Vista if it can’t resolve the issue in time.