False positives remain a major problem

But IDSs are getting better at managing large volumes of alerts.

Last year, our IDS review concluded that false alarms would drown any network manager who tried to use these devices. The level of alerts managed to drown the devices: Several couldn’t handle the load of our modest test network.

This year, we took a different slant in our testing, looking at how security analysts would use these devices in specific scenarios, but false alarms remain a major problem. As the virus and worm incidents during our test caused massive “bad” traffic across the Internet, we ran into serious problems with the volume of alerts. Even though we monitored significantly fewer systems sitting behind these IDSs than last year and significantly less traffic, 100,000 copies of the same alert each day made the systems sluggish and ill-behaved. In the case of Barbedwire Technologies, the systems became unusable. Cisco and Internet Security Systems (ISS) also filled up their disks, showing the importance of proactive management of alert information.

But while the volume of false alarms remains high, the products have gotten better in their ability to manage that information. Products from Cisco, ISS and NFR Security all showed significant improvement in how they present alert information to the operator. With flexible grouping and display options, and automated upgrade and downgrade of alert information, we could make our way though the thousands of alerts we got each day. Although tuning remains a major task – which each of the products could simplify – the event management tools gave us a better handle on things.

We also observed that while the attack signatures seem to be not much smarter than the last time we tested, IDS products are getting better at managing the output of these signatures. We got better information on the estimated severity and likelihood of an attack.

Still, there is a huge element of trust: You don’t get to actually see the offending packet (except in the case of Barbedwire). Over the months of testing, these products didn’t earn that trust very well. For each attack we detected, we were unable to say, for certain, how it happened. We only could come up with a candidate list of possibilities, each of which had to be researched individually.