Microsoft released a critical software update Tuesday to patch a security hole in a common Windows component that could allow malicious hackers to place and run their own code on machines running the Windows operating system.

The security hole, in a Windows component called the ASN.1 library, affects a wide range of Windows features and software, from file sharing between Windows machines, to software applications that use digital certificates, said Microsoft and eEye Digital Security, which discovered the problem.

ASN, or Abstract Syntax Notation, is an international standard for representing different types of binary data such as numbers or strings of text, eEye said.

The ASN.1 library allows different software applications running in Windows to identify the types of data they are passing back and forth, allowing each system to properly interpret the data it receives. It has been a standard Windows component since the release of Windows NT Version 4.0, said Marc Maiffret, chief hacking officer at eEye.

An unchecked buffer in the ASN.1 Library could allow remote attackers to cause a buffer overflow and take control of a vulnerable Windows system, Microsoft said Tuesday.

In buffer overflow attacks, hackers use flaws in a software program's underlying code to overwrite areas of the computer's memory, replacing legitimate computer instructions with bad data or other instructions.

The Microsoft security bulletin, MS04-007, patches the ASN.1 Library on affected Windows systems, ranging from Windows NT Workstation 4.0 Service Pack 6a to the latest version of Windows Server 2003.

The patch fixes vulnerabilities eEye discovered in the ASN.1 Library and informed Microsoft about in July. However, the company warned that Microsoft's implementation of ASN is "fraught with integer overflows." Speaking Tuesday, Maiffret said it is likely that other security flaws will be found in the library.

"Typically, if there's one set of vulnerabilities, there will be more found. Everything we know of was fixed today, but usually where there's one there's many," he said.

That was the case with Microsoft's implementation of Distributed Component Object Model (DCOM), a protocol that allows software programs to communicate over a network. A security vulnerability in the DCOM interface that handled remote procedure call traffic spawned the Blaster worm, as well as other attacks.

The IDG News Service is a Network World affiliate.