How we tested UTM firewalls

We invited all major firewall vendors to participate in this Enterprise UTM Firewall test last June. To prepare for the test, we wrote a test methodology, which we circulated to enterprise network managers, other Network World testers and some contacts in the vendor community. Based on their feedback, we constructed a final test plan (.pdf) that accompanied the invitation.

We asked vendors to submit devices that could handle about 1Gbps of throughput, and we warned them that sending “overpowered” devices would not necessarily be to their advantage, because of pricing and other considerations. We asked for a high-availability pair of devices so we could access those options. We also asked each vendor to sent its central management toolkit, whether on a dedicated appliance or as an individual software applications.

In our methodology and invitation, we noted that we would be primarily testing antivirus and intrusion-prevention features, so we asked for devices that could handle one or both of these common UTM features, along with other enterprise features that might be separately licensed, such as dynamic routing.

For each set of devices, we used a combination of commercial test tools from Spirent and Mu Security, standard electrical-engineering measurement products, as well as our own custom-written tests to evaluate the products in 10 categories. We plugged each device into an infrastructure that included a core 10/100/1000 Ethernet switch from Enterasys Networks, KVM switching devices from Avocent and Intel-based servers running VMware server.