Open-source security tools fit the enterprise bill

Our product tester, Joel Snyder, shares his picks for antispam, vulnerability testing, intrusion detection, VPNs and more

Open source technologies already permeate most data centers, and their influence is spreading. However, data center managers who wouldn't think twice about dropping a new Linux server into a rack feel very differently about building an open source firewall as the main barrier between their own network and the great unwashed. Security remains outside the open-source comfort zone.

See related story on How you're already using open source security

Still, there are four primary arguments in favor of open-source security tools: agility in the face of changing threats, control of one's own destiny with full source code, customization to one's own requirements, and lower cost (see "Agility, control, customization, affordability"). With that in mind, good examples of freely available security products abound.

Greater agility in mail security

The e-mail security gateway is a perfect example of how open source products can answer the need for agility. The function of this gateway has changed from interoperability between disparate mail systems to security, with protection against spam and viruses - and now phishing protection and compliance requirements - at the top of the list. The gateway landscape continues to change quickly, with commercial products entering or leaving the market rapidly, and requirements changing just as fast. If companies opt for an open source solution - in which they build their own gateway from multiple components - they gain a high degree of agility, even though they also take on a substantial integration effort.

Antispam tool SpamAssassin, probably the poster child for open source security, is powerful enough to be at the core of several commercial products, including the popular Barracuda mail gateway. SpamAssassin is far from data center-ready, however. Companies using it probably will have to create (or adapt existing open source) Web front-end applications and find a framework for scaling across multiple systems. There also is the need for user quarantines for suspect mail, tools to deliver mail, periodic quarantine management, reporting and alerting, and system management. Companies also will have to wrap a message transport agent, such as Postfix, around SpamAssassin to send, queue and receive e-mail. While some open source projects, such as the MailWasher server and Maia Mailguard, have integrated an antispam engine with management tools and quarantine, none has the active and lively development and huge user community that SpamAssassin does.

SpamAssassin by itself is no longer the state of the art in spam identification. Reputation-based filtering has been demonstrated to be very effective when combined with a good content filter; and new protocols, such as Sender ID and DomainKeys, help fight phishing attacks. Integrating freely available reputation-based services, such as SpamHaus or SpamCop, with other antispam tools isn't impossible, but requires expertise in mail-gateway design and the open source applications. Antivirus capabilities also belong in any mail security gateway. The only credible open source option is ClamAV, although a company choosing a Linux base for its e-mail gateway also has the option of several commercial engines that run on Unix.

Other antispam engines, such as CRM114 (see "The antispam man"), DSPAM and Bogofilter, are not as popular in large-scale environments because they rely on user training to achieve very high spam catch-rates. However, those building their own custom gateways can experiment with any filtering tool to see if it fits into the enterprise.

In control with intrusion-detection systems

An intrusion-detection system (IDS) doesn't just detect attacks. It also is useful for forensics, detecting network misuse and misconfiguration, and even network performance profiling.

To meet such varied needs, the IDS requires a way to collect and store events from sensors deployed throughout a network, as well as to search, collate and analyze events as they come in, archive and retrieve IDS events, generate instant alerts from some sets of events, manage all these components, and report on long-term trends. In more advanced deployments, IDS data uses a correlation engine to look for trends across events.

The Snort team (most of whom work for Sourcefire, selling a commercial IDS and intrusion-prevention system [IPS] based on the open source Snort engine) has taken care of the first half of this picture with its powerful IDS detection engine. As with SpamAssassin, Snort alone is almost completely useless. Yet it is easily layered on top of operating systems such as Linux or BSD (Berkeley Software Distribution or Berkeley Unix) to build an IDS sensor that detects traffic and generates events. Still, without an infrastructure to manage Snort and the events, companies might as well not bother.

Data center managers who want to build a 100% open source IDS they fully control might consider starting with Snort-based IDS sensors that typically run on Linux and then using a dozen or more other open source components to manage the sensors.

Managing the sensors can require home-grown scripts or applications, although there are specific tools, such as Oinkmaster and IDS Policy Manager, for keeping Snort rule sets updated properly. To log events, the common approach is to use Barnyard, a Snort add-on, along with the MySQL database. Once events are logged, tools such as Analysis Console for Intrusion Databases or the newer Basic Analysis and Security Engine - combined with a Web server and various scripting and graphics tools - can be used for trending and forensics.

But because the most difficult part of creating an enterprise IDS is turning the sensor's data into useful information, rather than building from scratch, a better solution may be to use open-source IDS sensors and a commercial IDS "super console" to handle events, alerts, archiving and forensics. This approach still minimizes the risk of being stuck with a network of IDS sensors from a commercial vendor that goes out of business, a significant concern considering that 40% of the IDS and IPS vendors in Network World's 2003 test (and 50% in our 2002 test) have gone under or left the IDS/IPS market.

Most of the security information management products from such vendors as ArcSight, NetIQ, Network Intelligence and Tenable Network Security, for example, will work perfectly well with Snort-based sensors. For an additional license fee, Sourcefire's 3D Defense Center will accept events from open source Snort as easily as from Sourcefire's packaged offerings.

Custom code for vulnerability analysis

Knowing what's on the network and what services are in use is an important part of security. Unfortunately, application programmers and system operators don't always keep the security team in the loop as systems are brought online, updated, patched and reconfigured. A referee, in the form of a vulnerability-analysis tool, can be a valuable adjunct in keeping abreast of services and servers automatically.

Tenable's Nessus, a popular tool for service discovery and vulnerability management, is pushing the limits of what open source means in the data center. Originally it was a fully open source tool; last year, the primary developers of Nessus made it free, but also proprietary, when they released Version 3 of the scanning engine. Changing the license was unpopular with the always-volatile open source community, but the number of enthusiastic users doesn't seem to have diminished. Nessus Version 2 is maintained as an open source project (see related story, "Is it free, or isn't it?").

With a client/server architecture and several GUI interfaces available, Nessus needs less additional software to make a fully functional package than does SpamAssassin or Snort, depending on how the information Nessus provides will be used. Other vulnerability-analysis scanners and network discovery tool vendors offer more tools for managing scan results, linking to patch-management systems and handling the vulnerability life cycle, but the Nessus team has focused more on making a highly configurable engine.

To gain closer parity with commercial products, Nessus users can buy Tenable's Security Center. This is a centralized management tool for Nessus scan data that contributes reporting functions, asset and vulnerability management, and a correlation engine that links IDS engine events with detected vulnerabilities to give security managers a better idea of what's important. In addition, most commercial security information management products can digest and correlate Nessus scan data.

Nessus is an active vulnerability scanner, which means it probes systems to discover services, operating systems and vulnerabilities. At many organizations, however, active scanning is unacceptable. The bad reputation they have for crashing or slowing systems, along with other political issues, has spawned a market for passive scanners.

Some limited types of passive scanning (such as operating system fingerprinting) are available from the open source community, but network managers interested in a more comprehensive approach should stick to commercial scanners available from Sourcefire (Realtime Network Awareness) and Tenable (Passive Vulnerability Scanner). (See our Clear Choice Test on these products.)

Containing costs in VPNs

As inexpensive as centrally managed firewall-VPN devices are, the cost to build a large-scale, site-to-site VPN can be high. Sometimes the problem is the vendor mix, because vendor-supplied firewall management tools can't handle multiple vendors. Other times, it's a question of a company being stuck with perfectly good branch-office firewalls that don't do VPN very well.

Great open source options in four security areas
Mail security gatewaySpamAssassinDSPAM


MailWasher Server

Maia Mailguard

Clam AntiVirus

Intrusion detection


IDS Policy Manager
Vulnerability scanning



Site-to-site SSL VPN



The best open source alternative is OpenVPN, an SSL-based VPN tool that easily and quickly links broadband-connected remote sites to a central data center. As a technology, OpenVPN has advantages, even over some commercial VPN products based on the more efficient and better-behaved IPSec protocols. Because most broadband ISPs use network address translation (NAT), businesses using low-cost connections without statically assigned IP addresses have found that the complex IPSec protocols don't always work reliably through NAT devices. Encapsulating traffic inside a single TCP connection is a good way to get around the problem, and SSL-based VPNs such as OpenVPN have always done this.

From the point of view of cost, OpenVPN is even more attractive. If there's a remote server running Windows, almost any flavor of Unix, or Mac OS X, the server can be used as a VPN gateway to connect the remote site securely using OpenVPN. Because the software works with the servers already deployed at remote sites, it can be retrofitted easily into existing networks without requiring new hardware.

OpenVPN isn't the answer to all ills. High-speed connections work better over IPSec, and the idea of running VPN traffic through a server won't go over well in many environments. In addition, OpenVPN is not suitable for large, meshed VPNs because it lacks a large-scale management system.

Like many open source tools, the OpenVPN management interface is a command line. Because of the popularity of the product and the well-documented API in OpenVPN, a number of open source, GUI-based tools are available to help in configuration and system monitoring.

In hub-and-spoke VPN configurations of small branch offices - broadband connections coming into a central data center - OpenVPN works well and has high availability and scalability features that wouldn't normally be expected in open source products.

Snyder, a Network World Test Alliance member, is a senior partner at Opus One. He can be reached at

< Previous story: Protection from the inside out | Next story: CISO inner circle >

Learn more about this topic

Agility, control, customization, affordability


Is it free, or isn't it?


Where you’ll find open-source security code


Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2007 IDG Communications, Inc.

IT Salary Survey: The results are in