Pushing firewall performance
Our new benchmarks show Cisco, CyberGuard and NetScreen to be top performers.
 
|
|||
 | |||
|
The terms "firewall" and "high performance" aren't often used in the same sentence. But with Internet access speeds of 45M bit/sec and higher setting the norm these days inside many enterprise networks, performance becomes a factor when buying a firewall.
How we did it
Sizing up price performance
Firewalls forum
Read user reaction to this review and Snyder's responses.
See our related links
Archive of Network World reviews
Subscribe to the Product Review newsletter
The results bring excellent news for network managers who want a firewall to separate multiple 100M bit/sec LANs. Products from NetScreen, Cisco and CyberGuard fit the bill for the highest throughput, with NetScreen offering overall outstanding performance with consistently high numbers across all our tests. To measure how these firewalls behave under different - but typical - conditions, we devised three test suites. The first suite measured raw throughput. This test closely simulated traffic conditions a firewall might encounter while being used at the core of an enterprise network or in an Internet environment where activities such as file-sharing must pass through it. Although real Internet traffic is a mix of packet sizes, we focused on the two ends of the spectrum: long packets (1,400 bytes) and short packets (64 bytes). Although long packets put more of a load on the network, short packets are the nemesis of most network equipment, as the per-packet-processing overhead begins to dominate network performance. An application that is well-behaved and sends lots of data in long packets over a small number of connections (such as a disk back-up system) will see the best performance with a firewall that does well in this particular test. Nine firewalls handled the long-packet test without dropping significant amounts of traffic. But only three products - those from TopLayer, Nokia and NetScreen - were able to handle more than 50% load (100M bit/sec throughput) in our small-packet test. These numbers show that finding a firewall to handle well-designed applications is not a difficult task. However, when it comes to the worst-case traffic pattern, you have to be a lot pickier. Six products overall turned in outstanding performance in the raw throughput test. Boxes from TopLayer, Nokia and NetScreen each clocked at more than 120M bit/sec throughput with short packets and 200M bit/sec with long packets. TopLayer and Nokia edged out NetScreen slightly in peak performance with small packets, with a peak of 130M bit/sec each vs. NetScreen's 120M bit/sec peak performance. NetScreen turned the tables on TopLayer and Nokia when it came to peak latency, however. TopLayer's peak latency was seven times and Nokia's more than six times that of NetScreen's. The higher the firewall latency, the slower the interactive performance. The next three finishers in the raw throughput sweepstakes were Cisco, Lucent and CyberGuard, with throughput rates in our short packet tests between 50 to 70M bit/sec. These products also achieved 200M bit/sec throughput on the easier long-packet tests. CyberGuard was an especially impressive performer because of its platform: SCO Unix running on fairly generic PC hardware. We didn't expect to see CyberGuard edge out some of the other dedicated hardware vendors. Our tests showed a significant drop-off in performance after the six top performers. If you are expecting a high volume of internal traffic for applications such as backup, database and file services, we suggest you select one of the top six products. Some products that handled 200M bit/sec in the long-packet tests, including products from Enternet, Secure Computing and WatchGuard, turned in near-zero results when it came to short-packet tests. This shows the dramatic difference packet size can make in product performance and how cautious network managers have to be when evaluating benchmark data. When reviewing these test results, remember raw throughput testing isn't indicative of typical Internet traffic. For a typical company connected via a DS-3 (45M bit/sec) circuit, the only products that turned in suspiciously low numbers were from Network-1 and Novell. These firewall packages would need additional tuning or faster hardware.
Massive connections
Our second test suite evaluated each firewall's ability to maintain a high TCP connection rate without losing packets or having to retransmit them. This is an especially critical feature for a firewall used in a data center where it may be protecting multiple Web servers. Due to TCP's design, packet loss or delay during connection establishment severely degrades the end-user experience. This is particularly true in Web-based applications in which a single page can have dozens of elements, each of which requires a separate connection to the Web server. Our tests set up varying numbers of connections, from 25,000 to 120,000, at rates from 100 to 15,000 per second. A Web server accepting 100 connection/sec over an eight-hour business day will handle almost three million connections that day. Seventy-five percent of the products tested came in with acceptable performance rates up to 500 connection/sec. These measurements were for bidirectional traffic, meaning incoming and outgoing, connections. We also tested at 100 and 1,000 connection/sec, outgoing only, which we used to tune the products to ensure everything was working properly. Only about half of the products tested could compete at a rate of 10,000 bidirectional connection/sec. Of course, 10,000 connection/sec is a huge number. For a Web server, that represents more than 25 billion hits per month - well beyond what most firms need. Even at 500 connection/sec, one billion hits per month is plenty for the average company. Two-thirds of the products tested could hit that mark with reasonable levels of success. In the firewall world, a connection is a precious and expensive thing. For example, when a firewall is acting as a packet filter, a connection is a lightweight operation - every packet looks like every other packet. However, in a firewall that conducts stateful packet inspection, each connection means an allocation of overhead resources. At the slow end, firewalls that are also acting as full proxy servers, such as the CyberGuard product when running in proxy mode, have a huge amount of work to do. These machines typically have to work with the operating system to set up a matched set of TCP connections and maintain state tables. In our connection-based test suite, each product must be able to handle multiple firewall rules. We installed at least 100 rules in each firewall. For some firewalls, such as TopLayer and Check Point, the rules didn't make much of a difference because they create "fast path" dynamic rules for connections that speed processing once a TCP connection has been established. For packet-filter-based firewalls that don't dynamically modify rules, such as those from Novell and CyberGuard, adding rules created a bigger performance hit. Our tests showed that hitting the 100 connection/sec mark was not a big deal for most products. Only a few - products from Symantec and Secure Computing - couldn't hack it at all, as they could only establish a small percentage of the offered connections at that speed. Other products were limited in other ways. Network Associates' product had a hard-wired internal limit of 4,096 connections, which means it protects a network of 500 users or so - even though it can set up those 4,096 connections at a rate of 10,000 connection/sec. Another substantial difference came in CyberGuard's firewall, which offers full proxy and packet filter modes of operation. When running as a packet filter, this product could keep up even at rates of 10,000 connection/sec. However, when we turned on the proxies, performance wasn't even close, as it maxed out at 100 connection/sec. At rates of 10,000 connection/sec, we could really see the differences that raw performance brought to the table. That's an obscenely high connection rate. There aren't many data centers that see that level of traffic, and even those that do probably don't funnel all the connects through a single firewall. Still we pushed hard and saw nearly perfect performance from Cisco's, CyberGuard's, Enternet's and NetScreen's firewalls.Real users
Our third suite of tests measured how each firewall behaved under peak stressful connections. We wanted to see how each product behaves when put in the role of protecting a building full of users who were all surfing the Web. Using wire-speed switches from Extreme Networks, we placed 20 test systems on either side of the test firewall. Because multiple systems were funneled down into a single port on each side of the firewall, we created contention, retransmissions and general chaos on the network. It was up to each firewall to keep it all straight, buffering and ordering the packets for best performance. We created many simultaneous connections (up to 800 in the hardest test) and started data transfer on all those connections simultaneously. To summarize the megabytes of data that resulted from this test, we divided the results into three categories: low load (100 to 300 simultaneous data transfers), moderate load (400 to 600 simultaneous data transfers) and high load (700 to 800 simultaneous data transfers). Because we pre-established the connections and concentrated on simultaneous data transfer, the high load created enough traffic to saturate the full-duplex Fast Ethernet connections to each firewall. We used the same set of 100 rules mentioned earlier to simulate a typical corporate security policy implementation. Our final score for each load level was a combination of the network throughput for each low, moderate and high category. We applied a penalty for data that had to be retransmitted. A firewall that didn't care whether it was seeing one or 1,000 connections provided roughly equal performance for low, moderate and high loads. Products from Cisco, CyberGuard, NetScreen, Enternet and Network-1 fell into this category. All achieved the same performance, plus or minus 10%, at every load we threw at them. However, the overall performance of each of these products varied widely. At high speeds, Nokia, NetScreen, CheckPoint and Cisco gave the best total performance. Other firewalls didn't scale quite so well - they might have done well at low numbers of connections, but didn't handle the higher levels at the same pace. At the lowest load, Secure Computing's firewall was the fastest firewall in the bunch. But as the connection count rose, it handled fewer packets, giving lower throughput. Likewise Novell's firewall turned in respectable numbers at the low load level, but when we pushed up the connect load, everything froze. If you're protecting a building fed by a DS-3 (45M bit/sec circuit) with 1,000 users or less, any product with at least a 100 score for the low load would look the same to you. If you are planning to feed your firewall anything faster, however, you'll want to look more carefully at products from Cisco, CyberGuard, Check Point, NetScreen, Network Associates, Nokia and Enternet.What's best for me?
Choosing the best firewall is more than looking at raw performance numbers. You must also consider management, flexibility, support, and reporting and documentation, as we have done in previous firewall reviews. That said, once you start designing high-speed Internet connections for your firm or look to establish internal LAN-to-LAN connections that require a high-performing firewall, our test brings definite performance differences to light. It's critical to characterize your load before drawing conclusions from our testing. For example, if you were protecting a building full of end users, Check Point's product would be on our short list for performance, while TopLayer's firewall would not. But if you were running a Web hosting farm, the reverse would be true. Only three vendors consistently made it into our top 5 performance lists: Cisco, CyberGuard and NetScreen. These three represent a spectrum for the network manager. Cisco turned in the best aggregate performance score across all tests and is always a safe bet. CyberGuard, one of the oldest players in this business, has an incredibly mature product, but recent financial troubles make it a riskier bet. NetScreen, the pre-IPO bad boys of firewalls, have a lot of momentum going, but it's still a small company and has a long way to go before its product line reaches the maturity of Cisco's or CyberGuard's.Click the images below for a larger view
RELATED LINKS
More firewall info:
Firewall fact or fiction?
Who's missing?
Is it secure, or is it an illusion?
Firewalls forum
Read user reaction to this review and Snyder's responses.
Global Test Alliance
Snyder is also a member of the Network World Global Test Alliance, a cooperative of the premier reviewers in the network industry, each bringing to bear years of practical experience on every review. For more Test Alliance information, including what it takes to become a member, go to www.nwfusion.com/alliance.
