VPNs: Fast, not friendly

Want to use the Internet as your corporate WAN, but you're worried about security? Don't be. We tested 15 virtual private network (VPN) products and found that all provide solid security for your company's Internet-based transmissions.

But be prepared to make some tough choices, because no one product aced our performance, interoperability and management tests. If you're a performance fiend, VPNet's VPNware VSU-1100 2.51 and 3Com's PathBuilder S580 will knock your socks off. Have to mix and match multiple vendors' gear? Radguard's cIPro System 3.30 is as compatible as it gets in the proprietary world of VPNs. If management is your concern, Internet Dynamics' Conclave 1.52 is far ahead of the pack, though it has a lot of ground to gain in the interoperability race. If price is your top priority, RedCreek's Ravlin 10/5100 3.0.2 may be the place to start.

But wait - there's more. Want a combined VPN and firewall? We tested five. Want a software-only solution? Just pick your platform: NT, NetWare or Unix.

We know it's not easy to narrow a field that's this wide. It took nearly six weeks of testing for us to determine our top picks. In the end, Radguard's cIPro earned our Blue Ribbon Award. An all-around contender, cIPro set the bar for interoperability, which was a major obstacle for many vendors. While its management features are not outstanding, cIPro turned in near-top speeds and proved more flexible than most of the VPN products we tested.

We found three devices that deserve honorable mentions. Placing a close second to cIPro in our Score Card was VPNet's VPNware. Among the top three in our performance tests, VPNware delivers solid management features, better-than-average interoperability and good flexibility.

Also praiseworthy are Check Point Software's VPN-1 Gateway Solution with VPN-1 Accelerator Card and RedCreek's Ravlin, an affordable, interoperable product held back only by its average performance.

Our testing focused on site-to-site VPN products used to connect private LANs over an insecure network, such as the Internet. In a future test, we'll look at client-to-site VPN products that help secure the connection from a single low-speed system, such as a PC dialing into the Internet, to a corporate LAN.

Most of the products we tested are VPN devices with minimal firewall features. You'll find more complex firewall technology with VPN features in some products, such as Intel's LanRover VPN Gateway 6.6 and Lucent's VPN Gateway 2.0. A few products, including Internet Dynamics' Conclave, Check Point's VPN-1 Gateway and Axent's Raptor Firewall/VPN Server, are firewalls at the core with VPN features added. 3Com's new PathBuilder is really a router with built-in VPN hardware assist. And Microsoft and Novell have added VPN features to their existing operating systems.

A tough lesson in interoperability

We take interoperability for granted with most products, but VPNs are another story. As enterprises build huge VPNs, they'll quickly discover that no one vendor has a good solution for all environments and that multivendor interoperability is key to a successful deployment. Vendors who come to this realization as well will prosper; those who don't will wither.

Our interoperability tests were no small undertaking. As the number of boxes in the lab increased, interoperability testing took longer and longer. It took nearly two full days for us to test the final product against all the other vendors' products.

While results could vary wildly depending on the profiles and products tested, one thing our tests showed clearly was that universal interoperability is by no means a given. In part, poor interoperability exists because vendors have not had many opportunities to test their products with each other - and because the product marketing managers would rather you bought everything from one source.

Even the best product in our interoperability tests, Radguard's cIPro, worked with only a small subset of the other vendors' gear. The reason it scored well is because it works better than any other product we tested. We scored the products relative to each other, not against a perfect-world benchmark.

Admittedly, our interoperability tests were particularly harsh. We required vendors to support Data Encryption Standard (DES), Triple-DES encryption algorithms and IP Security (IPSec), with its Internet Key Exchange (IKE) protocol. Our goal was to verify that different vendors' products could talk to each other using a common profile, which was a real impediment for some. For example, Intel's LanRover product wouldn't talk to any other products using our selected profile, but it did reasonably well in a simpler fallback test.

Radguard's cIPro, VPNet's VPNware, and Data Fellows' F-Secure VPN+ 4.0 all scored well for interoperability because of their extensive logging and analysis features. This isn't interoperability per se; however, because these devices provided such good logging facilities, we were able to get them up and running with more products. Rather than return a cryptic error message - "invalid payload" was about all we got out of the TimeStep Permit Gateway 4520 no matter what we tried - these products reported a great deal of detail about what they saw and what they were expecting. If every vendor offered the quality of logging that Data Fellows does, our testing would have been a lot easier.

Flexibility helped other products talk to more systems. RedCreek's and Nortel's gear interoperated with a surprising number of products with a surprisingly small amount of tweaking.

For some products, no amount of tweaking helped. Internet Dynamics, Microsoft and Novell failed our interoperability tests because they currently don't support the security protocols we required. (Lucent is not part of our interoperability matrix. It took nearly two days for us to get Lucent's VPN Gateway 2.0 up and running, despite having a Lucent technician on site. Because of this delay, we were unable to perform interoperability testing with the Lucent VPN Gateway.)

Private tests, such as those run by the International Computer Security Association, generally show more encouraging results than we found in our lab. But those tests usually have product developers from both sides in attendance and allow a full week for the experts to talk to each other. Our tests present a more typical customer picture because we had one vendor representative - usually a field engineer, not a developer - available to configure products in just a few hours. Having a vendor's representative on site was invaluable because we often had to tweak hidden variables or debug parameters on each device.

Our tests were also more generous because we only required one-to-one interoperability - we required each product to talk to each other product but only one at a time. Based on what we saw, true multivendor interoperability won't be as successful. For example, Radguard was able to get high marks by turning on and off the sending of vendor ID, a fairly new feature in the IPSec negotiations. But this feature is per system, not per tunnel. This means that Radguard cIPro might not have been able to interoperate with as many vendors simultaneously as it could on a one-by-one basis.

Speed for sale

We asked our vendors to perform Triple-DES encryption in our performance tests for two reasons. First, Triple-DES offers military-grade security. No one has successfully cracked a data stream encrypted with Triple-DES. Second, there are readily available hardware accelerators for Triple-DES encryption. Other algorithms commonly supported in VPN systems, including Blowfish, CAST and RC5, may be as secure as Triple-DES, but can't be accelerated with commonly available hardware tools.

Encryption is compute-intensive. We were amazed that Axent managed to pump out 8.3M bit/sec on a standard 350 MHz Pentium II system - and that was slow compared with Lucent's 30.2M bit/sec on a hardware-accelerated 300 MHz Pentium II.

Products tested with and without acceleration showed dramatic differences: Intel's Pentium-based LanRover VPN Gateway jumped from 2.7M bit/sec to 8.9M bit/sec, and Check Point's VPN-1 Gateway went from 6M bit/sec on a standard Sun Sparc 20 Workstation to 23.1M bit/sec when we enabled the vendor's VPN-1 Accelerator Card.

In general, if you need speed, bring money. That's the lesson we learned from the superfast products we tested. VPNet's VPNware and 3Com's PathBuilder blew us out of the water. We don't really know how fast they can go, except that they pegged the meter on our test bed. Both turned in a blistering 60M+ bit/sec of multistream Triple-DES encryption on a 100M bit/sec LAN with real world traffic. But both vendors want more than $16,000 per LAN.

Unfortunately, VPNs put IP at a disadvantage. The additional overhead of a VPN increases the size of IP packets. This causes packet fragmentation, which means that network resources are used inefficiently. Encryption also increases latency, which reduces overall throughput. Our performance numbers in multistream tests represent the best throughput available in a low-latency, zero-loss Internet with real-world software at both ends.

Even on this almost ideal Internet, adding encryption cost us almost 30M bit/sec compared with our no-encryption baseline. If you put a VPN into your network, be prepared to see a lot of bandwidth disappear.

If you want to add VPN capabilities to an existing firewall or server, performance is a major concern. Encrypting a T-1 link will eat up most of the processing power of an existing Pentium-based firewall, so go for hardware encryption or a dedicated server if you anticipate a significant amount of traffic. Radguard delivers the bandwidth bargain - you get hardware encryption at the lowest price-per-megabit we found.

We tested IP compression with Nortel's Contivity Extranet Switch 1500 2.0 and VPNet's VPNware. IP compression helps to combat the ill effects of VPN packet explosion. Because of the scrambling effect of encryption on data streams, products must apply compression before the data is encrypted, either at the application level or in the VPN itself. Link-based encryption yields no performance increase.

When we tested Nortel's Contivity switch with compression turned on, multistream performance jumped from 11.9M bit/sec to 26.7M bit/sec on our easily compressible data stream. However, VPNet displayed different results: Compression dropped performance from a chart-topping 60M bit/sec down to 48.5M bit/sec. We suspect that this performance loss is because of the overhead involved in shipping packets between specialized hardware encryptors and the general purpose CPU.

Management: The weakest link

We found the biggest differences in the products we tested to be in their management systems. With a couple of exceptions, the vendors have a long way to go in this category.

Most vendors treat each VPN box as a separate, and separately managed, entity. Because VPNs are almost always used in pairs, this makes no sense to us.

The worst offenders were 3Com and Compatible Systems. Both use a per-device command-line interface that was often overtly hostile. (Compatible Systems also has a graphical user interface, but the current release of the VPN code was not in sync with the current GUI. For example, the GUI we had couldn't be used to build VPN tunnels between devices.)

Axent and Nortel also failed to impress us with the management of their respective Raptor and Contivity products. Their devices require a completely separate GUI, and there's no way to share configuration information between two different systems. Data Fellows crafted a uniquely confusing management system, requiring no less than three different GUIs and one out-of-band secure channel to keep its boxes running.

RedCreek, Radguard, and TimeStep do a better job of making the network manager's life easier. Although they treat each box as a separately managed entity, you can look at multiple boxes simultaneously to synchronize configurations and share common information. We could put up with their management GUIs for a small number of tunnels.

Check Point, VPNet and Internet Dynamics are the only vendors that help network managers work with a VPN as a tunnel, not as two disjoint end points. Among these, Internet Dynamics' Conclave was the real winner. Conclave, a combined firewall and VPN, lets the network manager think in terms of policies: How secure do you want the traffic going from here to there? Once you explain the policy you want, Conclave determines how to build a VPN between your systems to yield the desired security. And if it can't give you the security your policy requires, Conclave doesn't let the traffic move. Internet Dynamics' management interface stands head and shoulders above the others.

Because of the complexity of encryption and VPN configuration, good documentation is crucial to making informed decisions. Simple misconceptions about key terms could leave your data entirely exposed.

We have the usual complaints about product documentation. Vendors are using the online documentation trend as an excuse to do little more than provide online help for their GUI. However, some stood out with good hard copy and online documentation. TimeStep, for example, provides a white paper on VPN security that contains one of the most lucid explanations of IPSec we've ever read. It's worth looking at TimeStep's product just to read the tutorial. At the other end of the spectrum were Novell and Compatible Systems, whose documentation stopped just short of being worse than nothing.

There are many factors to consider before settling on a VPN strategy. Among them are compatibility with existing equipment, management workloads and extra features (see "How to choose the right VPN product"). We suggest that you take the time to test prospective products before you buy. Security is just the beginning.

RELATED LINKS Scorecard and NetResults
How we ranked them in several categories, key findings, vendor contact info, pricing and performance metrics.

How we did it
A look at our testing methodology.

Public keys
A look at PKI interoperability. Network World Fusion, 5/10/99.

Interactive buyer's guide
Find the VPN that best matches your criteria. We have detailed specs for 57 models. You can search on specific criteria or compare two or more products against each other.

Forum: VPN tips
Discuss VPNs with Joel Snyder and other Fusion users.

VPN RFPs
A user RFP results in proposals from 11 VPN vendors. See what systems they would build.

How to choose the right VPN product

A market divided
The state of the VPN market today. Network World, 5/10/99.

VPN Net Resources
Additional info, from primers to more advanced topics.

Snyder is a senior partner with Opus One, a consulting firm in Tucson, Ariz., specializing in e-mail, security and networking. He travels the world helping people build bigger, faster, safer and more reliable networks. He can be reached at joel.snyder@opus1.com.