How we did it

Our real-world test put the five network intrusion-detection systems through the wringer at three locations.

Our real-world test put network intrusion-detection systems through the wringer at three locations. Our goal was to mix elements of a multi-site enterprise network with the inherent randomness of the Internet to see how network IDS products would support a professional security analyst.

We started by installing sensors for each product at two locations, in the Los Angeles and San Jose areas. At each site, all sensors saw the same traffic at the same time, giving us the opportunity to compare reactions to the same events across all five products. We also installed HP ProLiant DL330 hosts running unpatched versions of Unix and Windows versions as well as Cisco IOS – these were our “sacrificial lambs.” We started unpatched, but had to apply some patches to avoid being taken over within seconds by the self-propagating worms which were roaming the Internet during our test. We monitored each system and reloaded using Symantec Ghost as each was cracked or hacked. To ensure maximum availability, we attached all sensors and sacrificial lambs to SmartUPS XL5000 uninterruptible power supply from American Power Conversion.

We hoped to reproduce our environment from last year, where each host was broken into several times during the months of testing. Because this was not a performance test, we allowed the normal Internet traffic level at both sites, about 3M bit/sec combined, to be seen by each sensor.

For management, each vendor was invited to send its management system to our network operations center, in Tucson, Ariz. Barbedwired Technologies was unique in having a combination sensor/management console, so the company arbitrarily picked one of its sensors to also be a management system. In most cases, the management system also was accompanied by a client application for analysis and forensics research.

All IDS sensors have at least two interfaces: one (or more) for sensing, and one for reporting and management. We hooked the management interfaces on the Los Angeles and San Jose sensors to an IP Security VPN built using Nokia VPN appliances between all three sites so that they could have secure protected access between sensors and management systems. The management systems also were given Internet access (through a NetScreen firewall) so they could download signature updates and patches as necessary.

We then invited each vendor to monitor and tune their sensors to focus on our test sacrificial lamb systems and to reduce the false positive and false alarm rate. Vendors had 10 days to work on their own products, after which we cut them off from access to their products. All vendors but Barbedwired also chose to send a systems engineer to assist in the final installation, tuning, and training of our review team.

We ran the test without interruption for eight weeks from July 21 through Sept. 18.