Because the hardware platforms differed from firewall to firewall, we could not conduct consistent performance testing. However, we did run a basic series of tests, mixing single and multiple traffic streams using HTTP and FTP protocols, both sending and receiving data. We configured all systems with the same security policy or as close as we could get with each product's interface. Where content filtering was available, we disabled it.
We ran most of the firewalls on 350-MHz Pentium II systems with dual 10/100 Ethernet cards. Sonic Systems and Secure Computing provided their own hardware. In the case of Secure Computing, we used their 450-MHz Pentium II system; Sonic's hardware was proprietary.
A perfect score would have yielded the same throughput through the firewall as we achieved through our Cisco 3640 router: approximately 22M bit/sec. In our tests, only Secure Computing's Sidewinder came close (see table). Although Sidewinder - a Unix product - ran on a slightly faster box, it did better than double the performance of the fastest NT system. SonicWALL Pro also turned in excellent numbers with its proprietary kernel and a total hardware-plus-software price of less than $1,500.
We looked at the other products on Windows NT, although they are also available on Unix. After seeing their subpar results, we'd love to try the same tests on Unix.
| Throughput without a firewall: | 22M bit/sec |
| Secure Computing's Sidewinder: | 19.8M bit/sec |
| Internet Dynamics' Conclave 2: | 8.9M bit/sec |
| Sonic Systems' SonicWALL Pro: | 8.1M bit/sec |
| Axent's Raptor 6.0: | 2.8M bit/sec |
| Network Associates' Gauntlet: | 0.5M bit/sec |
| IBM's eNetwork Firewall 3.3: | 0.5M bit/sec |
Performance testing details
We conducted our performance tests on a bridged Ethernet network with no other traffic. We connected all the systems to the same Nortel Networks' Baystack 350T switch.
TEST 1
We downloaded a 10M-byte file from the Internet 10 times using FTP, through the firewall and to the client system, and noted throughput figures reported by the FTP client simulator. Traffic was a single stream from one BSD-kernel Alpha 500-MHz system to another using 1,518-byte Ethernet packets and the default TCP window size. We averaged the times of our 10 trials; the resulting number made up 60% of our performance figure.
TEST 2
We next repeated the first test using three streams running simultaneously from three clients to three servers. We ran the tests serially without interruption and averaged the throughput times. That number made up 20% of our performance figure.
TEST 3
Finally, we used an HTTP client simulator to retrieve 10 20K-byte documents simultaneously. The requests came from Internet users to a Web server located inside the firewall. We started each retrieval as soon as the previous TCP open had completed. We measured throughput based on elapsed time, meaning we included the total latency for TCP opens. The resulting number made up 20% of our performance figure.
RELATED LINKS
Review: Firewalls
Raptor Firewall 6.0 takes top honors in our testing. Network World, 7/19/99.
Firewall RFP
See how vendors responded to our RFP. Includes links to all the RFP responses.
Issues and trends
Where the firewall market is headed and what to look for. Network World, 7/19/99.
Interactive buyer's guide
Detailed specs on 52 models. Find the one that meets your criteria or compare two or more models on different specs.
Forum: Firewalls
Post your firewalls questions and discuss their use in this forum.
Firewalls to the rescue
Interviews with firewall users. Network World Fusion, 7/19/99.
