IPS performance tests show products must slow down for safety

High-end intrusion-prevention systems (IPS) move traffic at multigigabit rates and keep exploits out of the enterprise. The problem is they might not do both at the same time.

In lab tests of top-of-the-line IPS systems from six vendors - Ambiron TrustWave (formerly Lucid Security), Demarc Threat Protection Solutions, Fortinet, NFR Security; TippingPoint, a 3Com company; and Top Layer Networks - we encountered numerous trade-offs between performance and security.

Several devices we tested offered line-rate throughput and impressively low latency, but also leaked exploit traffic at these high rates. With other devices, we saw rates drop to zero as IPS systems struggled to fend off attacks.

In our initial round of testing, all IPS systems missed at least one variant of an exploit we expected they'd easily catch - one that causes vulnerable Cisco routers and switches to reboot. While most vendors plugged the hole by our second or third rounds of testing (and 3Com's TippingPoint 5000E spotted all but the most obscure version the first time out), we were surprised that so many vendors missed this simple, well-publicized and potentially devastating attack (see Can anyone stop this exploit?).

These issues make it difficult to pick a winner this time around (see link to NetResults graphic, below). If high performance is the most important criterion in choosing an IPS, the TippingPoint 5000E and Top Layer Networks' IPS 5500 are the clear leaders. They were the fastest boxes on the test bed, posting throughput and latency results more commonly seen in Ethernet switches than in IPS systems.

Product ipAngel-2500 Sentarus Network Security Sensor FortiGate-3600 Sentivist Smart Sensor ES1000 TippingPoint 5000E IPS 5500-1000 Vendor Ambiron TrustWave Demarc Threat Protection Solutions Fortinet NFR Security TippingPoint Top Layer Networks Price $100,000 Sensor $37,000; Sentarus Threat Protection System management application starts at $25 per node. $30,000 Sentivist Smart Sensor ES1000, $75,000; Sentivist Management Platform, $10,000. TippingPoint 5000E, $170,000; Security Management System, $10,000. $80,000. Pros Blocked all exploits in final tests; innovative, vulnerability-based configuration system. Blocked all exploits in final tests; vendor contributes signatures to open source Snort community; fastest to develop missing Cisco SNMP signature; well-designed dashboard gives instant status. Blocked all exploits in final tests. Blocked all exploits in final tests; very fine-grained control over traffic detection and response. Fastest performer for good (non-exploit) traffic; choice of fail-open and fail-closed modes; outstanding management interface overall. Strong performer with one or two port-pairs; good anti-denial-of-service protection features; rate-based management tools are top of the pack. Cons Modest performance from beta hardware and drivers; initially missed Cisco SNMP exploit; weak forensics and alerting capabilites. Relatively modest performer; searching for signatures is difficult; comprehensive forensics and analysis tools; weak IPS configuration, forensics and reporting. Lower port density than other products in this test; some software versions flooded exploit traffic (fixed in final version supplied by vendor); initially missed Cisco SNMP exploit; integration of IPS into UTM Firewall lacks features and manageability. Relatively modest performance; initially missed Cisco SNMP exploit; complexity of interface not for the casual user. Forwarded exploit traffic under heavy load; disables logging when overloaded. Forwarded some exploit traffic (possibly because of vendor misconfigura-tion); initially missed Cisco SNMP exploit; weak forensics capabilities. One port-pair configurations The breakdown Top Layer Ambiron TrustWave TippingPoint Fortinet Demarc NFR Baseline forwarding rate 10% 5 1.25 5 2.5 5 3.75 Forwarding rate under attack 15% 5 5 4.25 4 3.25 1 Baseline latency 15% 3.25 3.75 3.5 4 3.5 5 Latency under attack 15% 5 5 3.25 3.5 1.5 1 Protection from attack 25% 3 4 3 4 4 4 Usability 20% 3.5 2.8 4.1 2 2.7 3.9 TOTAL SCORE 3.94 3.75 3.72 3.38 3.28 3.21 Two port-pair configurations The breakdown Top Layer TippingPoint Ambiron TrustWave NFR Demarc Baseline forwarding rate 10% 5 5 1 1 2 Forwarding rate under attack 15% 4 3.75 1 1 1 Baseline latency 15% 2.75 5 2.75 4.25 3.5 Latency under attack 15% 5 2 5 1 1.5 Protection from attack 25% 3 3 4 4 4 Usability 20% 3.5 4.1 2.8 3.9 2.7 TOTAL SCORE 3.71 3.68 2.97 2.82 2.64 Four port-pair configurations The breakdown TippingPoint Ambiron TrustWave NFR Demarc Baseline forwarding rate 10% 2.5 1 1 1 Forwarding rate under attack 15% 2.75 1.5 1 1 Baseline latency 15% 5 4.5 4.75 2.5 Latency under attack 15% 3.25 4 1 2.5 Protection from attack 25% 3 4 4 4 Usability 20% 4.1 2.8 3.9 2.7 TOTAL SCORE 3.47 3.16 2.89 2.54 Scoring Key: 5: Exceptional; 4: Very good; 3: Average; 2: Below average; 1: Subpar or not available

Click to see: NetResults