Fast-forwarding firewall faceoff

Next-generation firewalls claim to identify application-layer attacks and enforce application-specific policies while delivering top-notch performance, even with advanced security features turned on.

In the first installment of this two-part Clear Choice test, we tackle the performance issue, evaluating NGFWs from Barracuda, Check Point, Fortinet and SonicWall (recently acquired by Dell). On May 7, we'll present Joel Snyder's analysis of the features and functionality of these same devices.

Our overall conclusion is that next-gen firewalls are getting faster, and the tradeoff between speed and security is definitely getting smaller, but it's still there.

While all devices moved traffic at multi-gigabit rates while doing application inspection - the feature that separates a next-gen firewall from a traditional firewall - forwarding rates fell when we offered SSL traffic, and plummeted when we turned on SSL decryption.

Palo Alto earns short list status

Next-gen firewalls off to a good start

In our tests, SonicWall's SuperMassive, the most expensive of the four products, moved traffic the fastest, even when forwarding SSL traffic. In multiple cases it maxed out the capabilities of our test bed. For example, when doing application inspection of clear-text traffic, it moved traffic at or near 20Gbps. That's even faster than Palo Alto's PA-5060, which hit 17Gbps in a test we conducted last year. 

Fortinet's FortiGate 3950B also pushed the limits of our test bed and finished a close second to SonicWall in tests involving clear-text traffic. It also handled slightly more TCP connections than the SonicWall device.

There was no performance slowdown with either the SonicWall or Fortinet devices when IPS and unified threat management (UTM) features were turned on. Conversely, turning on IPS and unified threat management (UTM) in the Barracuda and Check Point systems carried a heavy performance cost.

Check Point ran away with our toughest test. The Check Point 12610 proved by far the fastest at SSL decryption across all device configurations and was the only system tested to break the 1Gbps barrier (The SonicWall device ran faster still, but only when we changed our test configuration to offer more flows (see sidebar).

Barracuda, the lowest-cost device in our test, delivered a solid 12Gbps when we measured clear-text throughput using mixed content types.

Mixed-content loads: SonicWall leads the way

We measured forwarding rates for mixed and static-length HTTP and SSL content (both using HTTP and SSL); forwarding rates with SSL decryption enabled; and TCP scalability (see "How We Did It"). Of these, we put the greatest emphasis on the mixed HTTP tests, because they most closely approximate the loads handled by firewalls in enterprise networks.

A key goal of these tests was to compare results with those of the Palo Alto PA-5060 we evaluated in 2011 using the same methodology. 

The mixed-content tests involved a variety of object sizes, like enterprise traffic, ranging from 1KB to 1.536MB, and a variety of content types, including .jpeg images, PDF documents, binary files and text objects.

We set up the Spirent Avalanche traffic generator to offer this mixed-content load to each NGFW in three different modes: As a firewall only; as a firewall and IPS; and as a UTM device with all functions enabled (firewall, IPS, anti-spyware, and antivirus [anti-bot in Check Point's case]). For all three modes, we offered both cleartext Web and SSL traffic. We also ran separate tests involving decrypted SSL traffic, to be discussed later.

These NGFWs always had application inspection enabled. As we've discussed in previous tests, the ability to classify traffic and make forwarding decisions at the application layer is what distinguishes NGFWs from previous-generation firewalls, IPSs and other security devices.

NGFWs generally run fastest when they function as straight firewalls handling unencrypted traffic (see Figure 1 below). In terms of combined forwarding rate (adding incoming and outgoing traffic rates), SonicWall's SuperMassive was fastest, followed closely by Fortinet's FortiGate 3950B. Both products moved cleartext traffic at or near 20Gbps, the highest rate possible in one direction on our test bed. (All systems had four 10G Ethernet interfaces, with servers on one side and clients on the other.)

Both the SonicWall and Fortinet devices came close to maxing out the test bed's network capacity not only in the firewall-only tests but also when configured with IPS and antivirus/anti-spyware features enabled.

These numbers also compare favorably with the ones posted last year by Palo Alto's PA-5060, which topped out at around 17Gbps as a firewall, but fell to 5.3Gbps in IPS mode and IPS plus UTM modes.

SSL rates were generally lower than those for cleartext traffic. This isn't surprising given that even without decryption, an application inspection engine may work harder to identify the seemingly random patterns in an SSL stream.

However, there were some exceptions: Check Point's 12610 moved SSL traffic faster than straight HTTP, and in one case so did Barracuda's NG Firewall F900. The most likely explanation is that once the devices identified traffic as SSL (easy to do since SSL headers themselves aren't encrypted), they stopped any further attempts at traffic classification.

One configuration gotcha surprised at least two vendors' test engineers: When the Check Point and Fortinet systems had both SSL firewall rules and application inspection enabled, the inspection logic kicked in twice, causing SSL rates to be around half what each vendor expected to see.

The Check Point and Fortinet results were obtained without a specific SSL firewall rule, since the application inspection feature can identify SSL traffic and block or forward it as necessary. If this configuration issue can trip up firewall vendors' own engineers, it's definitely something for enterprise network managers to look out for.

Moving across the different configurations, the Barracuda firewall's forwarding rates dropped sharply when we enabled IPS and then all UTM features. Check Point's 12610 also moved cleartext traffic more slowly with antivirus and anti-bot features enabled; its SSL performance was about the same in all three configurations, again suggesting the device stopped inspection upon identifying a flow as SSL.

Static object tests

Tests of static 100KB and 512KB objects produced results similar to those involving mixed content. Devices generally moved static objects far faster over HTTP than SSL (see Figure 2 below).

The Fortinet and SonicWall firewalls again moved cleartext HTTP objects at or near the network limits of our test. SonicWall's SuperMassive came close to maxing out the SSL capabilities of our test bed. With no DUT in place, the Avalanche traffic generators moved 100KB and 512KB objects over SSL at 17.1Gbps and 14.4Gbps respectively. The SuperMassive moved SSL traffic near those rates, regardless of configuration. The performance degradation was more noticeable for Fortinet's FortiGate 3950B.

Also, as in the mixed-object tests, both the Fortinet and SonicWall devices moved traffic faster than Palo Alto's PA-5060 did in last year's tests. As a straight firewall, the PA-5060's top speed was 18.7Gbps with 512KB objects. That rate fell to 6.1Gbps in IPS mode and 6.3Gbps in UTM mode.

Conversely, the Barracuda and Check Point firewalls generally moved SSL traffic faster than plain HTTP, in one case - for Check Point - more than three times faster. Once again, both devices probably stopped inspecting traffic after classifying it as SSL.

When IPS or UTM modes were turned on, both the Barracuda and Check Point firewalls slowed down, but the Fortinet and SonicWall devices moved traffic at roughly the same rate regardless of device configuration.

(Sharp-eyed readers will notice we used 100KB objects here, compared with 10KB objects referenced in the previous Palo Alto test. Due to a configuration error on our part, the Spirent Avalanche traffic generator actually used 100KB objects in both tests. We stuck with the 100KB object size here for comparison, but in future tests we'll probably use 10KB objects because this is around the average HTTP object size for many enterprises.)

SSL decryption

SSL traffic poses a dual problem for NGFWs: If traffic is encrypted, applications cannot be inspected, but if traffic is decrypted there may be a very high performance cost. In fact, the SSL decryption tests turned out to be the biggest differentiator in this comparison, and for SonicWall the most controversial issue.

When doing SSL decryption, a firewall acts as a proxy, intercepting client requests and replacing the server's certificate with its own. Since users seldom inspect the replaced "server" certificate, they think they're dealing directly with the origin server. The firewall, meanwhile, decrypts and inspects traffic contents.

The Barracuda firewall's current software works as a non-transparent proxy, requiring reconfiguration of all client browsers for decryption to work. Barracuda says a forthcoming software release will support transparent proxying. The other three devices all functioned as transparent proxies when doing SSL decryption.

Also, the Barracuda and Fortinet devices only perform SSL decryption when antivirus inspection is enabled. The results given here reflect that; even though our methodology called for decryption in firewall-only and firewall-plus-UTM modes, the firewall-only numbers for the Barracuda and Fortinet firewalls were obtained with antivirus inspection enabled.

In these tests, Check Point's 12610 proved by far the fastest at SSL decryption across all device configurations. It also was the only system tested to break the 1Gbps barrier (see Figure 3 below). While that's nowhere near the 12610's forwarding rates without decryption, it's still a lot faster than the others tested.

Neither the Fortinet nor SonicWall devices decrypted SSL traffic at rates anywhere close to their rates without SSL decryption. Decryption rates for Fortinet's FortiGate 3950B ranged between 191Mbps and 472Mbps, far slower than its 3.6Gbps to 6.0Gbps range of rates without decryption.

Decryption rates fell even more precipitously for SonicWall's SuperMassive, but the vendor disputed our methodology. In our tests, the SuperMassive moved SSL traffic at 11.3Gbps without decryption, even with UTM features enabled; with decryption, the same load moved at just 83Mbps, slower than the 108Mbps low-water mark seen in the previous Palo Alto PA-5060 test. The rates were slower still, down to 49Mbps, with static 100KB objects, compared with 626Mbps for the PA-5060 in last year's test.

SonicWall says the SuperMassive also can decrypt traffic at far higher rates, provided it's pushed harder. The vendor noted that its device's CPU utilization during these tests was only around 2%, suggesting it was capable of doing around 50 times more work.

To put that assertion to the test, we conducted one-off tests with 50 times more flows, and found that SuperMassive decrypted traffic at rates of up to 4.8Gbps (see sidebar: "Scaling Up With SonicWall's SuperMassive"). We also tried the same large-flow-count tests with the other firewalls, but none could operate at this level without some failed transactions.

Even though the results clearly show a big performance hit for all devices with SSL decryption, things actually could be much worse. We used the relatively weak RC4-MD5 cipher in these tests. While that's the cipher in use at many e-commerce sites, most banks and other financial institutions use much stronger ciphers, such as AES256-SHA1, that are far more compute-intensive and presumably would result in still lower forwarding rates.

TCP scalability: Fortinet handles the most connections

The final set of tests examined TCP scalability in two ways: in terms of capacity (the maximum number of concurrent connections each device could sustain without timeouts or other failures) and rate (the maximum speed at which each device could set up and tear down new connections, again with zero failures).

In the connection capacity tests, we configured Spirent Avalanche to build up successively larger connection counts by having each existing connection make one new HTTP request every 60 seconds. Fortinet's FortiGate 3950B took top honors here, handling more than 10 million connections (see Figure 4). SonicWall's SuperMassive was close behind, successfully fielding 9.9 million connections. The Check Point and Barracuda systems handled far fewer concurrent connections, at 900,000 and 320,000 respectively.

To measure connection setup rate, we configured Spirent Avalanche to use the older HTTP 1.0 specification, which requires a new TCP connection for each new transaction. SonicWall's SuperMassive was the clear leader, setting up 290,000 connections per second (cps). Check Point's firewall was next, setting up 57,039 cps, while the Barracuda and Fortinet firewalls set up connections at 47,043 and 42,911 cps respectively. The SuperMassive's highly parallelized architecture (using 96 CPU cores) clearly favors a test like this.

We concluded last year's review of the Palo Alto PA-5060 saying there's room for improvement when it comes to NGFW performance. The vendors in this review have taken note: Forwarding rates are generally higher, as is TCP scalability. Further, some devices decrypt SSL traffic far faster than in previous tests. While there's still a security/performance tradeoff - a big one - when decrypting SSL traffic, it's clear there are now more choices for high-speed application inspection and control.

Thanks

Network World gratefully acknowledges the assistance of Spirent Communications, which supplied its Spirent Avalanche 3100 GT traffic appliances for this project. Spirent's Michelle Rhines, Jeff Brown and Paul Downs also provided engineering support. Thanks also to Arista Networks for supplying its 7124S 10G top-of-rack switch that tied together all systems on the test bed.

Newman is a member of the Network World Lab Alliance and president of Network Test, an independent test lab and engineering services consultancy. He can be reached at dnewman@networktest.com.