The way Microsoft is protecting the operating system kernel in the upcoming 64-bit version of Windows Vista continues to be a sore spot with some security vendors who claim it impairs the effectiveness of their intrusion protection and antimalware products.

The Vista Kernal Patch Protection feature, known as PatchGuard, is intended by Microsoft to prevent modifying system structures for purpose of intercepting system calls, said Bruce McCorkendale, Symantec’s distinguished engineer. But at Symantec, whose host-based intrusion-prevention and antimalware software works by sometimes using undocumented methods not formally recognized by Microsoft to combat spyware or ward off attacks, the PatchGuard restrictions in Vista will hamper Symantec’s effectiveness.

“The behavior blocking, intrusion prevention and tamper protection in our products today will be somewhat degraded by PatchGuard,” McCorkendale says. That’s because Symantec products have been designed “to use whatever means necessary,” he pointed out, to detect and eradicate malware and block attacks that by their nature also use any means possible to undermine Windows security.

“Sometimes when attackers are doing certain things, we turn to ‘kernel patching’,” McCorkendale says. “This runs afoul of the PatchGuard policy.”

“There are legitimate reasons for protecting the kernel and we are not asking Microsoft to disable PatchGuard,” McCorkendale says. But he said the security industry would benefit from added APIs for 64-bit Vista that would allow for documented ways to accomplish technical processes such as image-load filtering, memory-management filtering and named-object event filtering to name a few.

“We brought this to the attention of Microsoft 1½ years ago,” McCorkendale says. PatchGuard is not a feature in the 32-bit version of Vista, however.

McAfee, a main competitor to Symantec in the traditional antivirus market, last week wouldn’t define exactly what the impact of PatchGuard feature might have on its line of security products. But it has also been sparring with Microsoft over the issue and lobbying the Redmond giant for more APIs. In addition, a smaller security vendor, Authentium, says it has found a way to disable PatchGuard, load its own antivirus and antispyware software, and turn PatchGuard back on again.