Wanted: Safety plus simplicity

The six firewalls we tested act their age, for the most part. The seasoned firewalls rely on their finely tuned security features and a range of built-in proxy support. The firewall rookies have narrower feature sets but enviably easy management tools.

With so many firewalls on the market, how did we decide which to review? See The final roster for details on our selection process.

Our top scorer is Axent Technologies' Raptor Firewall 6.0, an established product in the firewall marketplace. Raptor's architecture isn't particularly exciting: It uses application-level proxies, some packet-filtering options and a graphical management interface. Though its performance could be faster, Raptor is an excellent high-end firewall.

What's striking about Raptor is the depth of the product. For example, implementing dual domain name system (DNS) servers is one area in which firewall vendors have done a lousy job for years. Security experts insist that keeping public and private DNS information separate is vital; firewall vendors have devised a variety of obscure and difficult-to-maintain configurations to do this. With Raptor, building dual DNS servers for a midsize site is a simple matter, requiring only a few clicks and fields to fill in.

How we did it We set up each firewall using our corporate security policy: a combination of internal and external access, authentication, and some blocking and filtering. We verified that the products could implement the basic policy we needed and checked for obvious errors in configuration using external verification tools. Because we wanted to do performance testing, we asked all the vendors to provide software on Windows NT if available. Because not all the vendors were able to accommodate this and because the hardware varied from vendor to vendor, we conducted only minimal performance testing. The International Computer Security Association has certified all these products, so we didn't duplicate that organization's extensive security attacks. Our hardware platform for testing most firewalls was a 350-MHz Pentium II server with dual 10/100M bit/sec Ethernet cards, and 384M bytes of RAM running Windows NT Server 4.0 with Service Pack 3. Secure Computing provided a 450-MHz Pentium II system, and Sonic Systems provided its own proprietary hardware.

Our second-place finisher, Sonic Systems' SonicWall Pro 3.3.1, is a dedicated hardware package that uses stateful packet filtering. Well-suited for branch-office environments, it's small, quiet and easy to configure using a Web browser. Rather than spend a lot of time building proxies with little added value, the Sonic folks have pushed for maximal functionality in the few proxies they do provide, which are mostly for Web browsing via HTTP. In addition, Sonic's dedicated hardware turned in excellent scores in our performance testing .

On the down side, however, SonicWall Pro limits your ability to set security policies. For example, SonicWall Pro was the only product we tested that could not implement the demonstration policy we set for the firewalls because it could not finely control outside users' access to LAN services. With SonicWall Pro, if you let an Internet user in, then you let him in everywhere.

Another firewall high on our list of interesting products is Internet Dynamics' NT-based Conclave 2. The architecture Internet Dynamics offers is beautiful: You manage policy in one place and download it into access filters everywhere.

Policy managers act as custodians of the security policy; this is where the administrative graphical user interface (GUI) sits, along with administrative tools such as report writers. Access filters are the gateways between resources and the rest of the world. Access filters, in theory, don't even have to be provided by Internet Dynamics - we could imagine an IP router providing access filter services, or perhaps a product developer could add code to a database or e-mail server.

While you can express complex policies very easily in Conclave, the firewall responsible for implementing those policies isn't very sophisticated. For example, one of the most basic features of a File Transfer Protocol (FTP) proxy is the ability to control file uploads and downloads, a feature that Conclave doesn't support.

In contrast, Secure Computing's Sidewinder Security Server 4.1 is right out of the old school. Secure Computing has added access-control features to an Intel-based Unix platform. On top of the hardened kernel, Sidewinder includes a vanilla set of packet-filtering rules plus a large set of TCP/IP proxies and services.

Sidewinder is one of the few products that still encourages you to run services on the firewall itself, taking advantage of the company's "tough kernel." Even if someone manages to get control of the FTP partition, for example, the intruder cannot jump over and affect e-mail. Sidewinder also shows its age in some of the proxies it supports. For example, none of the other products we tested has an X.500 proxy nor the massive configuration options for X.500 that Sidewinder does.

Although Sidewinder won our performance race hands down with double the throughput of the nearest competitor, it's not a firewall for the faint of heart. There's a mutated Unix operating system under that GUI, and you better know how to manage it.

Network Associates' Gauntlet Firewall 5.0 is the direct descendent of one of the oldest firewalls around, Trusted Information Systems' freeware tool kit. Gauntlet offers packet filtering for services that need it and proxies for everything else. With a full-featured HTTP proxy (including support for Cyber Patrol URL filtering), Gauntlet can support most simple configurations.

Gauntlet consists of a lot of pieces linked together with a GUI pasted on top. The Gauntlet team has done a good job at report generation and rolling over log files, but Gauntlet was the slowest firewall in our performance tests, and its GUI made it difficult to get a complete picture of our configuration.

IBM's eNetwork Firewall 3.3, like Sidewinder, is out of an earlier era of firewalls, but it lacks the depth of Sidewinder. IBM has done little to take this firewall forward, other than giving it a GUI and adding an NT version. IBM is one of the few firewall vendors to still support NEC's SOCKS protocol for proxying users through the firewall.

Inside, eNetwork Firewall consists of a few proxies combined with packet filters, though the proxies that are included are not particularly intelligent. For example, even though eNetwork has what IBM calls "transparent FTP," the process is anything but transparent - users have to change their FTP syntax to properly accommodate the firewall. What IBM does do well, though, is motivate third parties to integrate their products with IBM's product. While eNetwork Firewall comes on one CD-ROM, you'll receive a stack of 10 others with the pieces that IBM doesn't include, such as HTTP filtering and management and virus scanning.

One of the nice things about IBM's eNetwork Firewall is that it takes user authentication seriously. Unlike most vendors who simply tell you that it's a really bad idea to allow re-usable passwords, IBM actually gives you a copy of Security Dynamics' ACE/ Server and a couple of hardware tokens. It's not enough for a full deployment, but putting the parts in front of you is a big step forward.

Security matters

When we started evaluating these products, we first focused on their security features. There are certain features every firewall should have: proxies for control of some applications; packet filtering for speed or where proxies aren't needed; hooks to user authentication systems; Network Address Translation; software for virtual private networks (VPN) via IP Security if possible; and some way of handling difficult protocols such as DNS.

DNS was a trouble spot for many of these products. IBM's idea of handling DNS with three separate servers requires a high level of DNS expertise - and it depends on Microsoft's BIND-based DNS server running on the firewall, with all of the known security problems of running an old version of BIND.

Gauntlet didn't do much better, also bringing in Microsoft's DNS server. On the other hand, Conclave and SonicWall Pro didn't have a DNS server that can run on the firewall.

Similarly, proxy capabilities varied. Every product had an HTTP proxy, but only Gauntlet had a Lightweight Directory Access Protocol (LDAP) proxy. On the other hand, the Gauntlet LDAP proxy didn't do anything - you might as well have gone with packet filtering for all the functionality it added.

Within proxies we also found substantial differences. The most important proxy to most organizations is the HTTP proxy. Some want to control access; others are more concerned with viruses. SonicWall Pro had good support on the control side: you could block out ActiveX, Java and cookies, and run any Web, Network News Transfer Protocol, FTP or Gopher traffic through a content filter list. SonicWall Pro also let us vary filtering based on time of day. Conclave's HTTP proxy gave us even more flexible options for blocking traffic, and it has built-in virus scanning for HTTP traffic.

On the other hand, while Sidewinder's proxy can perform content filtering, it can't identify HTTP parts such as ActiveX or certain kinds of Java. IBM didn't even take a credible swipe at advanced HTTP proxy services; you have to use one of the third-party products IBM includes.

As for other security features, only SonicWall Pro offers stateful packet filtering, and only Sidewinder lets packet filtering trigger an audit event.

IBM's interface for designing packet filters was so awful that we gave up any hope of building our special packet-filter rule and manually edited the configuration file - a method the documentation discouraged. We were also disappointed in IBM's and Sonic Systems' VPN software because neither supports Internet Key Exchange protocol (IKE), greatly increasing the management burden while decreasing security.

Most of these firewalls have awful mail proxies, stripping mail down by leaving out delivery status notification, ESMTP extensions, encryption and authentication in a way that actually reduces security. Sidewinder offers the most powerful mail proxy, with sophisticated and well-designed filtering capabilities, but we weren't impressed with its implementation. Out of the box, Sidewinder comes configured as a promiscuous mail relay that makes you an unknowing vector for spammers to distribute their garbage all over the Internet. To fix this, you have to venture into the wild world of sendmail.cf files. In every case, you'd be better off transparently handing your mail through the firewall to a real mail relay.

Overall, we found Raptor delivered the most comprehensive security features. Gauntlet and Sidewinder were close behind.

Configuration and management

The weakest aspect of firewall products has long been their configuration utilities. The ideal management interface lets an unsophisticated user build a simple configuration that supports an organization's security policy, while at the same time letting a security guru do all the tinkering and fine-tuning required.

We found IBM's configuration utility overly complicated. Making sense of overlapping services, rules and connections in huge Java-built windows wasn't easy. It took us longer to construct our test configuration in eNetwork Firewall than in any other firewall, largely because of its GUI.

We also found Gauntlet's configuration unsettling. While you may think you know what each individual piece is doing, when you combine Gauntlet transparency (which has one default behavior for incoming data and another for outgoing), proxies, policies, destination matching and packet filtering, you may find that it's difficult to get a good overview of your entire configuration.

We found some features, such as remote management, available in all the products we looked at. However, when we went looking for separate configuration and monitoring functions, we found them only in Conclave.

Our favorite advanced feature - the ability to delegate different configuration tasks to different network managers - is available in Conclave, Sidewinder and eNetwork Firewall. However, IBM's restrictions are only built into the GUI and not enforced by the firewall itself.

A good test of a configuration interface is how difficult it is to build insecure configurations. We found the most error-proof GUIs in Conclave and SonicWall Pro, though they are miles apart in sophistication. Conclave offers a fully policy-based management system, and once we got the hang of it, we could do powerful things very simply. SonicWall Pro doesn't have nearly as much power as Conclave, but its GUI is simple. You can't do certain things, but what you can do, you don't make mistakes doing.

Conversely, with eNetwork and Raptor we found it possible to make an insecure configuration or one with holes in it without noticing anything.

Reporting and alerting

After securing your network, you'll want to know how well your firewall is functioning. We found that Gauntlet provided the best reports with the most useful information, followed by Sidewinder.

Conclave and SonicWall Pro both prepare HTML reports, although neither impressed us with its completeness or flexibility. IBM's eNetwork Firewall and Axent's Raptor have neglected reporting; both expect you to dump your logs into a database to generate reports. We found IBM's logging system particularly obscure when we were trying to debug problems in our configuration: The error message "ICA3015" is unclear unless you've got the manual lying around.

The flip side of reporting is alerting: a firewall's ability to alert you when something is amiss. The strongest in this area are Axent's Raptor and, to a lesser degree, IBM's eNetwork Firewall. With Raptor, we had much more flexibility than we even needed. After working with these firewalls in production, we found that the ability to selectively silence a particular alert is often invaluable, and we could do that easily with Raptor.

Overall, we were less impressed with reporting and alerting features than any other aspect of the products we looked at. While firewall vendors have concentrated on increasing security features, most have not paid as much attention to logging, reporting and alerting. Admittedly, better reports aren't as critical as a better SMTP proxy, but there's little excuse for firewall veterans to have done such a poor job at reporting after so many years of experience with building firewalls.

Final analysis

Overall, we found that Axent's Raptor beat IBM's eNetwork Firewall and Network Associates' Gauntlet for large networks. Secure Computing's Sidewinder stands apart from the others because of its blazing performance. If you have a faster link than a T-1 to protect, you may want to keep an eye on Unix-based firewalls such as Sidewinder. (Raptor, eNetwork Firewall and Gauntlet are also available on Unix, though we tested them on NT.) Secure Computing's hardened kernel sets it apart, particularly if you need to run services on your firewall system - something we don't recommend.

For branch offices, the SonicWall Pro has a lot going for it, including a huge price advantage, a strong HTTP proxy and an easy-to-use GUI. SonicWall Pro's simple configuration interface made it easy to make secure configurations quickly, and - more importantly - hard to make insecure configurations. This distinguishes it from the others we looked at.

Internet Dynamics' Conclave is a product to watch closely. Conclave has some strong advantages - no other firewall in this review lets you manage multiple systems as a single entity. However, the product still has a good distance to go before it matches the basic firewall features available from other vendors.

RELATED LINKS

Scorecard and NetResults
Key findings, pricing and vendor contact info.

Performance tests turn up big differences
Some specific numbers from our tests.

The final roster
How we decided which firewalls to review.

Snyder is a senior partner with Opus One, a consulting firm in Tucson, Ariz., specializing in e-mail, security and networking. He can be reached at joel.snyder @opus1.com.

Gwinn is associate director of computing services at the Edwin L. Cox School of Business at Southern Methodist University in Dallas where he works on wireless networking and security projects. He can be reached at allen@radio.net.

ICSA firewall resources

and ICSA certification criteria

Review: Firewwall/VPN combos
Network World, 4/19/99.

1998 review and buyer's guide: Firewalls
See what we recommended last year. Network World, 6/1/98.

Review and buyer's guide: VPNs
Network World, 5/10/99.

Firewall RFP
See what the vendors are responding to. Includes links to all the RFP responses.

Issues and trends
Where the firewall market is headed and what to look for. Network World, 7/19/99.

Interactive buyer's guide
Detailed specs on 52 models. Find the one that meets your criteria or compare two or more models on different specs.

Forum: Firewalls