Test: Spam in the wild
We throw real traffic at 16 anti-spam products.
By
Joel Snyder
,
Network World
, 09/15/2003
- Share/Email
- Tweet This
- Print
Practically every vendor on the planet claims to be able to solve your spam woes. So we tested 16 products on a live production
network to see who could back those claims. For the entire month of June, we threw a live mail stream, spam and all, at the
products to see who could survive the spam onslaught, and who would choke.
Estimates of the amount of unwanted e-mail range from 40% to 75%, but we can give you an exact percentage - 69%. That's how
much spam we saw during the month of June. And things are getting worse, not better - in a similar Network World test we ran
in February (see review), only 50% of the mail stream was spam.
Fortunately, good products prevailed, and can help you significantly reduce your spam problem. With a very broad field, including
service-based mail filters, appliances and traditional software on Unix and Windows, network managers should be able to solve
their spam problem with a minimum of disruption - to the accolades of their users.
How well do they work?
We tested mail-filtering gateways by feeding them an e-mail stream in real time, as it came into our labs (see "How we did it"). Each product received two scores. The first score, sensitivity, measures how well the filter identified spam. A perfect
score would be 100%. The second score is the false-positive rate, the ability of the filter to make sure that non-spam messages
do not get tagged as spam. A perfect false-positive rate would be 0%. (For more about the different ways to measure a spam
filter, see "Spam and statistics".)
Spam filtering is such that a high sensitivity naturally also would have a high false-positive rate. Similarly, a low false-positive
rate might let a lot of spam through. We feel that enterprise network managers would be more concerned with false positives,
so we asked the vendors to tune their products for a false-positive rate of about 1%.
Products from seven companies - Cloudmark, Corvigo, MailFrontier, MX Logic, Postini, Trend Micro and Tumbleweed Communications - met our 1% requirement.
To identify the top products in filtering spam, we looked for a sensitivity rate of at least 80%. Products from seven companies
also met that level - ActiveState, Cloudmark, Computer Mail Services, MailFrontier, Postini, Singlefin and Tumbleweed. (For complete results, see graphic.)
Combining these lists gives us the top overall performers: Cloudmark's Authority, MailFrontier's Anti-Spam Gateway (ASG),
Postini's Perimeter Manager and Tumbleweed's Messaging Management System (MMS).
Of course, your results will vary, depending on your own message-stream characteristics and how well you tune the products.
For example, Postini's spam-detection engine is at the heart of Trend Micro's recently released Spam Prevention Service (SPS).
However, we got very different results with the two products, largely because Postini officials told us to tune their product
using one set of numbers, while the Trend Micro team gave us a different set. This resulted in both a higher false-positive
rate and lower spam sensitivity for Trend Micro.
Comments (3)
RE: Test: Spam in the wildBy meatpieandtatters on August 27, 2007, 1:33 pmDid you test MIPSpace?
Reply | Read entire comment
gfi mailessentialsBy Anonymous on December 7, 2008, 6:24 amgfi mailessentials is quite possibly the worst antispam product i have ever seen. it's baysian engine blocks real emails, it's keyword lists are useless as they...
Reply | Read entire comment
Alianza Mutual de Seguros de Chile - Instituto de Seguridad del By Anonymous on August 10, 2009, 11:54 amtest
Reply | Read entire comment
View all comments