Why we didn't test performance

Our intrusion-protection system review methodology was open for comment for four months before testing began. We openly solicited vendor input regarding what to test and how to test it. Three vendors pushed hard to include performance testing.

The problem was, all three vendors suggested very different methodologies. The basic metrics - throughput and latency - were the same. And we agreed with these points because if IPS vendors want network professionals to put these devices in-line in production networks, the devices have to act as fast and as reliable as the switches and routers they replace.

But there was no agreement beyond that because IPSs differ in some of their most basic characteristics. Some appear on the network as hubs, others look like switches, and some operate as routers. Performance tests for Layer 2 switches and hubs are very different from those for Layer 3 routers.

Let's suppose for the moment that we tested IPSs the way we test Layer 2 or Layer 3 devices. It is possible to get valid numbers on throughput, latency, jitter and the like. The problem with such measurements is that they'd tell us absolutely nothing about the way these systems behave as IPSs. Performance tests don't measure security. They might not even measure performance in a meaningful way. After all, system behavior might differ radically when we configure IPSs for Layer 7 inspection rather than for Layer 2/3 forwarding.

Additionally, different vendors' IPS devices go into different places in the network. A few are meant to sit at the absolute outside edge, right next to an Internet-connected firewall. Others are more generic, engineered to go closer to the core or right in front of some set of protected hosts or subnets.

It's not rational to compare a device designed to support a DS3 Internet circuit with one engineered to replace a core 100M bit/sec or even 1G bit/sec switch.

The biggest roadblock to comparable, repeatable performance statistics was that no two vendors agree on the definition of IPS. The most basic test would be to simply push traffic through these devices and see how they behaved. That might have been repeatable, but it wouldn't have been useful. We don't care how these devices behave when they're simply passing traffic; we care about how they behave in the presence of attacks.

And that brings up two more difficult performance questions: What traffic should be passed? And what attacks should be blocked? As attacks begin to build up through an IPS, you can expect different kinds of behavior. The IPS might choose to drop packets randomly to protect the systems or networks behind it. Or it might delay packets, hoping to deter a SYN flood. Because putting attacks onto an IPS is going to cause it to vary its characteristics, the question again arises: What exactly would we be measuring?

If we expect the IPS to differentiate between good and bad traffic, we'd have to first define what good and bad are. Different products by design define those parameters differently, depending on what kind of attack they're seeing.