Gigabit intrusion-detection systems

Tests show network IDS products have a ways to go to get accurate detection at gigabit speeds.

In our tests of five leading network intrusion-detection systems and the popular open source Snort, performance was spotty during baseline testing and degraded by as much as 50% on some products when we opened the throttle to gigabit speeds.

Our first step was to run 28 well-known attacks against each product in an untuned state on a wire that had no other traffic running on it. Most products detected only about half the attacks.

When the systems were tuned, most products caught an additional two or three attacks, but still missed a good number of them.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

In our tests of five leading network intrusion-detection systems and the popular open source Snort, performance was spotty during baseline testing and degraded by as much as 50% on some products when we opened the throttle to gigabit speeds.

Our first step was to run 28 well-known attacks against each product in an untuned state on a wire that had no other traffic running on it. Most products detected only about half the attacks.

When the systems were tuned, most products caught an additional two or three attacks, but still missed a good number of them.

IntruVert's IntruShield 4000 was a bright spot. It detected the greatest number of attacks in every test (see the performance chart), and wins the Network World Blue Ribbon Award. A newcomer to this market, IntruShield is a well-designed, and feature-rich.

Internet Security Systems' RealSecure Gigabit Network Sensor Version 7.0 didn't detect as many attacks as IntruVert's product overall, (16 out of 28 at baseline with no tuning and 25 with tuning), but deserves the runner-up prize because its ability to detect attacks did not change at gigabit speeds. The other three commercial products tested at Miercom's lab facility in Princeton Junction, N.J., were Dragon IDS Server Appliance and Dragon IDS Sensor Appliance; Intrusion's Intrusion SecureNet; and, Symantec's (formerly Recourse) ManHunt Version 2.11. We also tested the open source package, Snort on Acid.

Performance tests

Our primary focus was to determine how well these products performed under a gigabit traffic load, which was 970M bit/sec in our tests. We ran the tests at slightly less than a full gigabit load to ensure that the link was not overutilized and all our attacks could get through (see How we did it).

In our baseline tests with no traffic, we did not tune the systems in any way, but we did turn on all signatures and protocol anomalies. We delivered 28 attacks to each system, including commonly known denial-of-service, surveillance and probe attacks, and attacks, such as Stick and Fragrouter, designed to evade an IDS system (see Attack List).

IntruShield 4000 detected the highest number of attacks - 24 out of 28. Dragon, RealSecure and Snort each caught 16 of the 28 attacks. ManHunt detected 14 attacks, and SecureNet caught 11.

A key factor in IntruVert's strong showing is a good implementation of signature-based attack detection (in which packets' contents are compared against a database of known attack patterns), and protocol anomaly detection (PAD) (in which the product verifies that a traffic flow is not violating its defined protocol - signaling suspicious activity.). Except for Snort, all the products supported both techniques, but IntruVert married the two technologies especially well.

Overall, the products caught about half the attacks. What accounts for this lackluster showing on a nontuned system is that some vendors turn off signatures to heighten performance. Vendors also make this trade-off so that administrators are not overwhelmed by the many false-positive alarms they receive before systems are tuned (see review).

The extent of an IDS's signature database also is a factor. The more attack signatures a product supports, the better its rate of detection without system tuning. For example, although ManHunt supports a signature database and PAD, it relies more on the latter. If an exploit or attack follows protocol then it's not detected unless the product has a signature to catch it. ManHunt supports a small signature database and doesn't do as well in this type of test.

We next ran the set of attacks that each product detected at baseline traffic levels against these still untuned products, but filled the pipe. RealSecure caught 16 out of the 16 attacks, ManHunt caught 13 out of 14 attacks, and IntruShield 4000 caught 21 out of 24 attacks. Dragon and Snort had the poorest overall showing, catching only 3 out of 16 attacks and 6 out of 16 attacks, respectively.

Tuning helps, but not much

This same set of tests was conducted again on tuned systems. Tuning meant that the vendor could tweak any signature code included in the product's database to let it catch or identify an attack correctly. The vendor could change User Datagram Protocol (UDP) to TCP, or vice versa to catch a Back Orifice attack. Or it might decrease thresholds for the number of TCP connects to catch an NMAP attack. It also might turn off processor-intensive engines, signatures and features to enhance performance.

Some products were tuned in an hour; some took much longer. How familiar the vendor's technical representative is with the system and tuning it plays an important role - something end users should consider. What type of assistance does the vendor provide for system tuning is a good question to ask.

Vendors were not allowed to add new signatures to a database, customize existing signatures or download signatures from the Snort database to use during testing. In a customer deployment, customers presumably would take advantage of all of these tuning options. We did not allow this to happen in the lab because we would then be assessing how well a vendor's on-site technician could tune a system rather than evaluating gigabit performance, which was our intent.

Overall, tuning helped the products detect a few more attacks, but the increase was not dramatic. At baseline traffic levels and tuned, the IntruShield 4000 caught all 28 attacks. RealSecure benefited the most from tuning, detecting 25 out of 28 attacks - compared with only 16 without tuning. Snort improved from 16 to 18 and Dragon went from 16 to 17. Tuning had no effect on ManHunt, which detected 14 attacks in both rounds.

When we threw gigabit traffic at these tuned boxes, again using only attacks the boxes could catch at baseline, RealSecure caught 25 of 25 attacks delivered at gigabit. IntruShield 4000 caught 27 out of the full 28. ManHunt caught the same number of attacks (13 out of 14) it had on an untuned system.

SecureNet's performance was much improved by tuning. In the baseline tests on a untuned system, SecureNet caught only four out of 11 attacks at gigabit speeds, but caught 14 out of 15 attacks when tuned.