Attackers wielding zero-day exploits are one of the most significant threats facing enterprise networks today. While plenty of vendors promote zero-day protection mechanisms, if they don’t address the entire operating system, they leave the door open for attack.
Today’s operating systems are designed to provide varying layers of access to resources. Hierarchical protection domains — often referred to as privileged rings — protect the operating system from faults and general instability. Arranged from most privileged or most trusted (usually zero) to least privileged or least trusted (usually the highest number), these domains provide the ability to enforce security in the operating system.
Applications execute in the least trusted or least privileged domain (also known as user space), while the operating system executes in the most trusted or most privileged domain (also known as kernel space). This separation enables the operating system to distribute resources and shield against undesirable behaviors that might otherwise have a rippling effect. Without this barrier, viruses and other malicious software could easily replicate across each process and run rampant. Protected behind the barrier, the operating system requires each application to request permission to access various system resources or to have more privileged operations carried out on its behalf.
Microsoft and a host of security vendors have invested a tremendous amount of time and effort into developing enhanced security features to protect customers. These enhancements typically deal with kernel space, monitoring the resource requests made by applications in user space. The enhancements, for example, prevent write access to critical structures in memory, monitor inbound and outbound packets for known exploits, and analyze application behavior to ensure that a word-processing application isn’t suddenly and inexplicably sending out confidential data.
Additionally, a variety of other methods are commonly used by host-based security products to shield applications from vulnerabilities lurking beneath the surface. This can include marking stack and heap memory addresses as nonexecutable, or randomizing memory addresses returned by memory-allocation routines. This increases the level of difficulty — and in some cases might make it impossible — to exploit buffer-overflow vulnerabilities.
Another type of host-based protection, commonly known as behavior analysis, intercepts and inspects the various system calls requested by applications to enforce restrictions based on policies. A variation to this approach involves loading an application in a virtual machine emulator, which allows instructions rather than system calls to be intercepted and analyzed before execution. This approach can be used to analyze every instruction executed by the application, rather than relying on system calls with limited visibility to piece the bigger picture together.
Despite host-based protection mechanisms being notoriously difficult to configure and use in enterprise networks, they represent some of the best approaches available. But organizations implementing such protections remain at risk, because these security products provide zero-day protection only at the application layer and not the operating system kernel.
This leaves customers with a false sense of security. A skilled adversary can gain access to the network by exploiting vulnerabilities in the kernel.
Any product touting zero-day protection and application-layer security must extend the same level of security to the operating system.
Although no security product on the market today can protect the entire operating system from every vulnerability, recent virtualization and hardware advances make it possible to build the next generation of security technology directly into virtual machines. More important, by creating a trusted and security-conscious virtual-machine monitor or hypervisor, it will be possible to achieve greatly increased levels of visibility.
As one of many possible scenarios, virtualization appliances could be built and deployed that allow one or more servers to operate concurrently while the security software operates beneath and provides the protection necessary. With the security software positioned lower than the operating system, vulnerabilities in the operating system can no longer compromise or circumvent the security product’s ability to function. Furthermore, these vulnerabilities could be detected easily and prevented.
A solution such as this would provide greater protection than what is currently offered, achieve cost savings by consolidating critical servers and eliminate the need for additional security software. Although this approach demonstrates one way of securing enterprise servers, similar concepts could be applied to protecting the user as well.
Zielinski is a security engineer and member of Arbor Networks’ Security Engineering & Response Team.
