Malware-detecting 'sandboxing' technology no silver bullet

University of California researcher says malware authors are aware of sandboxing and are in an 'arms race' to stay ahead of it

The security technology called "sandboxing" aims at detecting malware code by subjecting it to run in a computer-based system of one type of another to analyze it for behavior and traits indicative of malware. Sandboxing -- one alternative to traditional signature-based malware defense -- is seen as a way to spot zero-day malware and stealthy attacks in particular. While this technique often effective, it's hardly foolproof, warns a security researcher who helped establish the sandboxing technology used by startup Lastline.

When it comes to malware detection, "a sandbox shouldn't be considered a silver bullet," says Christopher Kruegel, associate professor, computer science department at University of California at Santa Barbara, who is also chief scientist at Lastline, which has its own sandboxing techniques. His admonition comes at a time when the sandbox approach, typically applied to email, is getting more attention as a way to uncover stealthy zero-day attacks intended to compromise organizations and steal data.

[ IN THE NEWS: McAfee acquires ValidEdge sandboxing technology ]

FireEye, Trend Micro, Palo Alto with its WildFire service, GFI, AhnLab, Damballa, Norman and Sourcefire are among the security firms with some form of sandboxing; McAfee recently acquired ValidEdge to expand its own approach.

But malware authors are aware of sandboxing and they come up with various ways to evade sandbox detection, Krugel warns, in what he says is an "arms race." Among malware evasion methods that are part of this growing trend are:

- Stalling code. According to Lastline, "this new evasive technique, which we refer to as stalling code, delays the execution of malicious cade so that a sandbox times out. However, to do this, the malware does not simply sleep. Instead, the malware performs some (useless) computation that gives the appearance of activity."

The stalling technique by the malware works because it "simply executes, and from the point of view of the malware analysis system, everything is normal." According to Lastline's research, it can be part of the sandbox "blind spot":

- A "blind spot" in the sandbox implementation. To monitor malware, "a sandbox introduces hooks," Lastline says. "These hooks can be inserted directly into a program to get notifications (callbacks) for function or library calls. The problem with direct hooks is that the program code needs to be modified, and this can be detected by malware or interfere with dynamic code generation (unpacking)." But the main problem with hooking system calls is that "the sandbox cannot see any instruction that the malware executes between calls. This is a significant blind spot that malware authors can target; and they do so with stalling code, which is code that runs between system calls."

Another evasive method is done through environmental checks, Lastline points out.

- Malware authors can add novel, zero-day "environmental checks" related to the operating system and "manipulate the return value" as an evasive maneuver that forces vendors to "patch" their sandbox to catch it, according to Lastline.

Lastline seeks to address these sandbox-evasion tricks in its Previct appliance it offers, but Kruegel acknowledges "there is no 100% security."

Some information-security managers say they appreciate sandboxing as a defensive technology but don't seem to have any illusions that it is going to be perfect in detecting and stopping malware.

"Sandboxing will get some of it," says Brad Stroeh, senior network security engineer at First Financial Bank, a Sourcefire customer, in discussing a wide variety of security approaches and the credence he places in them. It's worthwhile subjecting malware when possible to a sandbox test, and using it as part of the overall defensive process. But since malware could bypass sandbox checks, it only makes sense to use other malware-detection methods as well.

Ellen Messmer is senior editor at Network World, an IDG publication and website, where she covers news and technology trends related to information security. Twitter: @MessmerE. Email: emessmer@nww.com.

From CSO: 7 security mistakes people make with their mobile device
Join the discussion
Be the first to comment on this article. Our Commenting Policies