How we tested Check Point UTM-1 firewall.
Fortunately, we were able to save and restore all our policy settings, though we did run into some rough spots discussed in the test article. We also used the UTM-1 as a firewall with Point’s SmartDashboard.
We did not test the firewall, because we were concentrating on what elements the new appliance hardware and the UTM-1 appliance-management system brought to the table rather than capabilities of the longstanding VPN-1 applications, which we have tested in the past, running on the appliance.
For testing of performance and goodput — the actual usable bandwidth, not counting lost or retransmitted data — we used Spirent’s WebAvalanche and WebReflector to generate HTTP traffic and push it through the UTM-1. To measure throughput capacity, we created a profile of HTTP transactions ranging in size from 2.2Kbits to 1300Kbits. We chose this profile to approximate the observed HTTP traffic on a production network with many users doing Web browsing. We connected two of the Gigabit Ethernet ports on the UTM-1 to the Spirent systems and used this profile.
We used the simulated users tests provided with the Spirent systems to simulate 250 and 500 users repeatedly and continually browsing Web pages based on the defined profile. The numbers we reported in our test are for 500 simultaneous users, though they do not statistically differ from the 250 user tests. These simulated users drove the UTM-1 to the limit of its goodput capacity.
When we tried to push more than 400Mbps of HTTP traffic through the UTM-1 by increasing the number of users or the size of the HTTP objects, we were not able to do so without significant numbers of failing transactions. Thus, we stuck with 400Mbps and 500 users as a very stressful test environment for the UTM-1.
For our tests, we wanted to see how much the UTM features effected performance, so we started with a basic firewall configuration with no other UTM features turned on. Our firewall rule set was modest and simply allowed HTTP and ICMP traffic through the firewall for our test network.
We did not vary the firewall rules at all in our performance testing. We also did not enable logging on the firewall “accept” (pass) rules. We believe that a typical UTM-1 buyer would normally log “deny” (drop or refuse) rules, but not Web-browsing pass rules. Because our UTM-1 logged locally, enabling full logging of all rules could have a dramatic effect on performance. With basic testing of firewall performance done, we enabled the first UTM feature, Check Point’s Smart Defense, a type of intrusion-prevention system.
We tested firewall plus several levels of Smart Defense, including the default settings, default with some client-side protections enabled and finally all Smart Defense settings enabled. With those test results in place, we disabled Smart Defense (IPS) and enabled antivirus scanning for HTTP traffic, the second UTM feature on top of the basic VPN-1 firewall.
Finally, we tried antivirus UTM and Smart Defense UTM at the same time. Our full results are in the table below.
