Test: Spam in the wild

Mail content management tools are obvious places to perform spam filtering, along with other tasks. Virus scanning is the most popular, and all but five products included virus scanners or made them available as an integrated option. (Cloudmark, Computer Mail Services, Corvigo, MailFrontier and Trend Micro had no integrated virus scanning capability.)

In a couple of cases, though, adding spam protection to an existing content management tool wasn't well executed. Trend Micro's SPS is a perfect example: It's a stand-alone, simple application that is completely un-integrated into its other products. We also had a bad experience with Clearswift's add-on to Mailsweeper, its venerable policy management system. The integration in that case was exceptionally poor. GFI's MailEssentials had similar issues - a badly thought-out anti-spam product bolted onto an older and more stable e-mail management tool.

When diving into the full-fledged content management arena, we found some products where integration of anti-spam functionality gave the network manager an incredibly rich feature set. Choosing among these products can be difficult. The first thing a network manager has to decide is whether he needs anti-spam and anti-virus capabilities (because these are so common to every company), or whether the additional tools of content management and policy enforcement would be valuable. If so, products such as Mailsweeper, MMS, Praetor and PureMessage had the greatest power and flexibility in both managing e-mail content and offering a wide variety of possible actions. For example, you could use MMS to identify incoming messages from an airline's "Internet-only specials" mailing lists and defer delivery until after business hours - unless they were being sent to your travel department.

Managing configurations

Defining your e-mail policy, whether it's anti-spam, anti-virus or content/policy means driving some sort of GUI. Most products picked either a local management application, such as a Win32 or Microsoft Management Console tool, or offered a Web GUI; SurfControl gave us both. For some products, including Cloudmark's Authority and Trend Micro's SPS on Unix, you also could simply edit the local configuration files. We were also impressed with Postini's batch command language, which is useful if you needed to update settings on hundreds or thousands of users or define many policies. Of course, the simpler the product, the simpler the GUI. This makes comparing products with many functions, such as SurfControl, to spam-filter-only products, such as Authority, a little unfair.

The highest level of flexibility came in two Sieve-based products (Vircom's modusGate and ActiveState's PureMessage) and in Computer Mail Services' Praetor, which uses Visual Basic as its scripting language. All invited the network manager to get down-and-dirty, writing in Basic or Sieve language, defining their own policies and rules for searching, classifying and identifying messages of interest. This can be important, because we found that not every vendor has a good idea of how to abstract out the idea of e-mail policy configuration in its GUI. One of the best was Tumbleweed's MMS, which we found easy and intuitive, letting us construct rules using any of the dozens of criteria, actions and notifications MMS supported - without having to dive into a programming language. MMS had some rough edges, though: If you wanted to add a string to the end of a subject line (such as "[SPAM]"), you used one kind of rule, and if you wanted to put the same string at the beginning, you used another. Whoever thought that up wasn't thinking clearly.

Other systems brought a lot of power, but with a serious lack of flexibility. Praetor is a good example. The authors of Praetor came up with a bunch of very interesting and typical scenarios that could be used, wizard-style, to build a mail policy. But like many wizards, they also are completely inflexible. Go down the path given to you, and it's easy; but decide you want to whitelist traffic for different domains in different ways, and you're kicking and screaming all the way. It's not impossible in Praetor,but a lot harder than it needs to be.

Getting updates is another management problem worth solving. Not every product needed spam "signature" updates because of the algorithms being used, but some that did weren't designed all that well. For example, Computer Mail Services' Praetor and GFI's MailEssentials both depend on word lists to help identify spam, and neither one has automated updates. Possibly this is because the word lists shipped are such poor spam filters that any network manager would have customized them so much that an update wouldn't do any good.

Corvigo's MailGate also requires manual updates, and while Clearswift, Cloudmark and ActiveState all support automatic updates, they don't build them into the product. That's dangerous because without simple things such as cryptographic hashes applied to a virus or anti-spam update, it's easy to propagate bad or corrupted updates and start bouncing or filtering mail inappropriately. Such a lackluster and amateur design to such a critical part of the system can be a costly problem to fix.

Speed can be a problem

We ran some simple performance tests on the anti-spam gateways to see if speed was going to be a problem. In our tests, we used a traffic generator to throw a stream of 10,000 e-mail messages (at approximately 20 messages per second) at each product. We were happy to see that some products could keep up well.

In the lab, Computer Mail Services' Praetor, which slowed down to one message every two seconds, turned in the worst performance. This might be caused by the internal architecture of Praetor, which converts your anti-spam rules into Visual Basic for actual filtering. That number gave us pause because that's only about 40,000 messages per day, well below what even a midsize company would see. (For more on our performance tests, see "How we did it".)

We also saw fairly low numbers out of ActiveState, modusGate, MailEssentials and SurfControl. In the case of SurfControl, we tested the software on slightly different hardware than everyone else: The CPU speed was faster, but the I/O subsystem was Advanced Technology Attachment-based rather than SCSI. Because a typical mail relay will see peak loads of five to 10 times the average load, it wouldn't take too much of a burst to make any of those products fall behind in the middle of the day.

On the service-based spam filters, we couldn't do the same kind of testing, but we did some testing with some statistics showing an alarming performance problem in Singlefin's spam gateway. Although Singlefin received and accepted messages at a good clip, it only could return them to us at 39% of the speed it accepted them, indicating that Singlefin has some performance bottlenecks to work around. Compare this to EasyLink, MX Logic and Postini, all of which could return messages to us as fast as we could send them.

Our performance problems with Singlefin were not just visible in message-processing speed. When logging on to manage user quarantines, we had delays of 15 to 45 seconds between screens. We also found the administrative interface to be tediously slow.

Services vs. software or appliances

Our experience during the month of testing was ambiguous as to which approach was better. In fact, we had disturbing failures on the appliance, service and software fronts during our months of testing.

The most innocuous problem was on our Corvigo appliance, which didn't recover properly from a power failure. It continued to accept messages but wouldn't deliver them. An e-mail to technical support provided an easy solution - reboot the system using the front panel. Sure enough, after a clean reboot, our messages started to appear.

Software-based gateways also had their fair share of failures. Late in the test, the Clearswift Mailsweeper software suddenly started refusing incoming messages, bouncing them all over the Internet, probably because of a time change. Rebooting didn't fix the problem, and Clearswift's 40-hour-per-week technical support staff didn't get back to us for several days, by which time the problem mysteriously went away (perhaps because the clock had caught up with itself).

On the service front, MX Logic turned up a problem during our performance testing that we hadn't noticed before: It was refusing certain kinds of messages. MX Logic fixed the problem within minutes once we convinced the help desk that the problem was on the company's side and had excellent response time for a Saturday morning. Our experience is that all of the approaches have some chance of failure.

Reporting features

Like all enterprise applications, reporting is an important check mark for an anti-spam gateway. Unfortunately, what reports are needed is something on which no two vendors agreed. Across the board, it's hard to say whether one set of reports was strong or weak. Our favorite, though, was clear: Corvigo's MailGate can provide a combination of reporting and log data through its Web interface so you easily can track any message through the system. While other products had this hidden in their mostly undocumented log files, Corvigo was the only one to make it easy to do. Clearswift's Mailsweeper came in a close second.

We also were interested in products that provided a good dashboard function: something that could instantly display system load, queue lengths, message counts and a current status of what's happening on the gateway. The best dashboard was SurfControl's GUI, which gave us incoming mail load, queue lengths, local statistics and status reporting. Corvigo's MailGate, Clearswift Mailsweeper, Cloudmark's Authority and Tumbleweed's MMS all did a good job at giving us instantaneous snapshots of how their servers were operating.

Vircom tried something unusual in its modusGate: It exported its performance and status information via Windows "perfmon" counters. This has a distinct advantage in a Windows shop: You simply can add the statistics you care about to your existing perfmon screens or run a separate copy.

What's best for me?

Although it's easy to rank products based on their ability to identify and filter out spam, your own requirements will determine which product is best for you. Your first decision has to be whether you consider individual user quarantine control and settings important. This feature will reduce mail server load and give users control over their own spam settings and whitelists. Top-rated products that include this feature include Postini's Perimeter Manager managed service offering, Corvigo's MailGate appliance and MailFrontier's Anti-Spam Gateway Windows-based software.

If per-user controls and quarantines are not as important as other policy-based and content-based mail filtering, Tumbleweed's MMS appliance did an excellent job of filtering spam and gave very flexible control over mail flows.

The breakdown	Filtering performance 40%	Per-user facilities 25%	Advanced functions 20%	Spam filtering control 10%	Speed 5%	TOTAL SCORE
Postini Perimeter Manager v3.3	5	5	3	3	5	4.4
Tumbleweed Communications Messaging Management System	4.5	3	5	4	4	4.15
MailFrontier Anti-Spam Gateway	5	4	2	4	5	4.05
MX Logic Email Threat Management Service	4	4	3	3	5	3.75
Corvigo MailGate v1.5-10	4	5	2	3	3	3.7
ActiveState PureMessage v4.0	3.5	3	5	4	2	3.65
Vircom modusGate v2.15	3	4	5	3	2	3.6
Cloudmark Authority v2.0	4.5	2	3	4	5	3.55
SurfControl E-Mail Filter v4.6	3.5	2	5	4	2	3.4
Singlefin E-mail Protection Service	3.5	4	3	2	3	3.35
Computer Mail Services Praetor v1.5	2.5	2	4	4	1	2.75
Trend Micro Spam Prevention Services v1.0	3.5	1	2	3	5	2.6
EasyLink MailWatch	1	2	5	3	5	2.45
Clearswift CS Mailsweeper 4.3	2	1	4	2	3	2.2
GFI Software MailEssentials v8.0	1	1	4	4	2	1.95