Anomaly detection is the best way to prevent virus, worm attacks

Two experts debate the effectiveness of a new security technology.

To stop worms and malware, first you must know about them. In today's rapidly evolving networks, where attackers are often one step ahead of the products designed to thwart them, anomaly detection is an important innovation. Many vendors rely on signature detection to find network-borne threats. Customers often have to wait days to get a working signature for a new worm, leaving their networks vulnerable in the most critical period during a worm's release.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

To stop worms and malware, first you must know about them. In today's rapidly evolving networks, where attackers are often one step ahead of the products designed to thwart them, anomaly detection is an important innovation. Many vendors rely on signature detection to find network-borne threats. Customers often have to wait days to get a working signature for a new worm, leaving their networks vulnerable in the most critical period during a worm's release.

Network behavior analysis is one of the most robust and scalable security technologies classified recently by Gartner. At the core of network behavior analysis are anomaly-based algorithms used to identify emerging threats. Three types of anomaly detection are used in network behavior analysis:

• Protocol - detects packets that are too short, have ambiguous options or violate specific application layer protocols. Most useful for detecting host-level attacks.
• Rate-based - detects floods in traffic using a time-based model of normal traffic volumes. Most useful for detecting denial-of-service attacks.
• Relational or behavioral - detects changes in how individual or groups of hosts interact with one another on a network. For example, a normally quiet host that starts connecting to hundreds of hosts per second on the SQL port indicates a worm. Useful for a variety of threats, from worms and malware to insider misuse.

By applying anomaly algorithms best suited to the attacks they are designed to detect, anomaly detection can proactively identify zero-day worms, malware, acceptable-use policy violations and insider misuse. Because anomaly detection looks for substantial changes in network behavior, it is less prone to false positives, and requires less configuration and ongoing maintenance than many other security methods.

However, network behavior analysis doesn't end with detection. Once a threat has been identified, this technology allows operators to visualize good and bad or suspicious traffic, and contextualize it in relation to other traffic and historical roles.

A network behavior-analysis system can take preemptive action, blocking a port on a switch, quarantining traffic to a separate virtual LAN, or applying a filter or access control list to lock down propagation. Behavior modeling is equally applicable to mitigation and detection.

For example, network behavior analysis systems can generate a filter to block traffic on a suspect port and simultaneously generate a white list for known good traffic, freezing the network in a known good state.

There's both an art and a science to applying anomaly detection. Effective use of the technology by security vendors requires deep experience with networks, threats and the appropriate anomaly-detection algorithms for a given threat model. When done well, anomaly detection is extremely effective in finding and foiling network-borne threats and should be part of everyone's security tool set.

Morville is director of product management at Arbor Networks. He can be reached at paul@arbor.net.

Read more about security in Network World's Security section.