Test: Spam in the wild

Practically every vendor on the planet claims to be able to solve your spam woes. So we tested 16 products on a live production network to see who could back those claims. For the entire month of June, we threw a live mail stream, spam and all, at the products to see who could survive the spam onslaught, and who would choke.

Estimates of the amount of unwanted e-mail range from 40% to 75%, but we can give you an exact percentage - 69%. That's how much spam we saw during the month of June. And things are getting worse, not better - in a similar Network World test we ran in February (see review), only 50% of the mail stream was spam.

Fortunately, good products prevailed, and can help you significantly reduce your spam problem. With a very broad field, including service-based mail filters, appliances and traditional software on Unix and Windows, network managers should be able to solve their spam problem with a minimum of disruption - to the accolades of their users.

How well do they work?

We tested mail-filtering gateways by feeding them an e-mail stream in real time, as it came into our labs (see "How we did it"). Each product received two scores. The first score, sensitivity, measures how well the filter identified spam. A perfect score would be 100%. The second score is the false-positive rate, the ability of the filter to make sure that non-spam messages do not get tagged as spam. A perfect false-positive rate would be 0%. (For more about the different ways to measure a spam filter, see "Spam and statistics".)

Spam filtering is such that a high sensitivity naturally also would have a high false-positive rate. Similarly, a low false-positive rate might let a lot of spam through. We feel that enterprise network managers would be more concerned with false positives, so we asked the vendors to tune their products for a false-positive rate of about 1%.

Products from seven companies - Cloudmark, Corvigo, MailFrontier, MX Logic, Postini, Trend Micro and Tumbleweed Communications  - met our 1% requirement.

To identify the top products in filtering spam, we looked for a sensitivity rate of at least 80%. Products from seven companies also met that level - ActiveState, Cloudmark, Computer Mail Services, MailFrontier, Postini, Singlefin  and Tumbleweed. (For complete results, see graphic.)

Combining these lists gives us the top overall performers: Cloudmark's Authority, MailFrontier's Anti-Spam Gateway (ASG), Postini's Perimeter Manager and Tumbleweed's Messaging Management System (MMS).

Click to see: Accuracy and False Positives

Of course, your results will vary, depending on your own message-stream characteristics and how well you tune the products. For example, Postini's spam-detection engine is at the heart of Trend Micro's recently released Spam Prevention Service (SPS). However, we got very different results with the two products, largely because Postini officials told us to tune their product using one set of numbers, while the Trend Micro team gave us a different set. This resulted in both a higher false-positive rate and lower spam sensitivity for Trend Micro.

Many vendors predicted that their false-positive rate would be much lower than 1%. Corvigo's CTO said its customers report a false-positive rate between 10 and 100 times better than our tests showed. That's easy to understand, because most of the false positives we saw fell into the category of "mail that wouldn't be missed by users," such as news stories forwarded by friends, e-mail from online merchants and postings to mailing lists. For example, Postini, which had the lowest false-positive rate, missed 28 messages it marked as spam. Of those, only five were messages we wanted to see. If we hadn't been combing our mail carefully, we wouldn't have noticed those messages as missing.

Some false positives were understandable, but regrettable. A message with the subject line "IOS fw guru" looked like spam to many of the filters, but turned out to be a job offer for our test lab.

Tuning to improve performance

Most of the products can be tuned to increase sensitivity and decrease false positives, but how this tuning is accomplished and who is responsible for it makes all the difference. There are two main tools used in tuning mail filters. First is the threshold that determines whether a message is spam. The best products offered a series of levels, often expressed as percentages, based on its guess as to whether a message is spam.