Zero-day exploits: Consider the OS

Attackers wielding zero-day exploits are one of the most significant threats facing enterprise networks today. While plenty of vendors promote zero-day protection mechanisms, if they don’t address the entire operating system, they leave the door open for attack.

Today’s operating systems are designed to provide varying layers of access to resources. Hierarchical protection domains — often referred to as privileged rings — protect the operating system from faults and general instability. Arranged from most privileged or most trusted (usually zero) to least privileged or least trusted (usually the highest number), these domains provide the ability to enforce security in the operating system.

Applications execute in the least trusted or least privileged domain (also known as user space), while the operating system executes in the most trusted or most privileged domain (also known as kernel space). This separation enables the operating system to distribute resources and shield against undesirable behaviors that might otherwise have a rippling effect. Without this barrier, viruses and other malicious software could easily replicate across each process and run rampant. Protected behind the barrier, the operating system requires each application to request permission to access various system resources or to have more privileged operations carried out on its behalf.

Click to see: Rings of protection

Microsoft and a host of security vendors have invested a tremendous amount of time and effort into developing enhanced security features to protect customers. These enhancements typically deal with kernel space, monitoring the resource requests made by applications in user space. The enhancements, for example, prevent write access to critical structures in memory, monitor inbound and outbound packets for known exploits, and analyze application behavior to ensure that a word-processing application isn’t suddenly and inexplicably sending out confidential data.

Additionally, a variety of other methods are commonly used by host-based security products to shield applications from vulnerabilities lurking beneath the surface. This can include marking stack and heap memory addresses as nonexecutable, or randomizing memory addresses returned by memory-allocation routines. This increases the level of difficulty — and in some cases might make it impossible — to exploit buffer-overflow vulnerabilities.

Another type of host-based protection, commonly known as behavior analysis, intercepts and inspects the various system calls requested by applications to enforce restrictions based on policies. A variation to this approach involves loading an application in a virtual machine emulator, which allows instructions rather than system calls to be intercepted and analyzed before execution. This approach can be used to analyze every instruction executed by the application, rather than relying on system calls with limited visibility to piece the bigger picture together.

Despite host-based protection mechanisms being notoriously difficult to configure and use in enterprise networks, they represent some of the best approaches available. But organizations implementing such protections remain at risk, because these security products provide zero-day protection only at the application layer and not the operating system kernel.