Anti-spam: The sequel
By Keith Shaw, NetworkWorld.com, 09/07/04
I now know how Hollywood executives feel when they green-light a sequel. When Lab Alliance member Joel Snyder and Network World conducted its first In the Wild test of anti-spam products last year, we were able to round up 16 vendors to submit to a 30-day, 11,234 message test to find out who could filter out spam, who didn't filter out good e-mail, and who would be left standing.
The test was so successful, we thought, hey, let's do this again!
And like a Hollywood blockbuster (and some software, we've noticed), everything in the sequel is bigger, more robust, more exciting, etc.
This year's anti-spam test will have almost 40 products being tested in Joel's Opus One Test Labs, a mix of software, hardware appliances and services. The goal? See what product best eliminates spam in an enterprise environment.
We started this project in May when we asked all of the vendors in our online anti-spam buyer's guide for their input on the methodology for this year's test. Several vendors responded, and with their help we have come up with this year's test, which still looks at SMTP-based systems. To participate in the test, a product or service had to:
• Accept mail using SMTP, either as an application running on an existing server, as a dedicated appliance server, or as a service.
• Filter and/or tag mail according to the likelihood that it is undesirable (aka spam).
• Forward the mail onto the corporate mail system using SMTP.
We recognized that other anti-spam products, such as client-side add-ins, or proprietary server-based products (such as those that only work with Exchange, would not be able to be tested. In addition, we have recognized that several products that use alternative techniques, such as challenge/response, or grey-listing, could not be tested in this article.
If you think your company would benefit from products such as these, go to the online buyer's guide and look at the list of products we have listed there.
With our multitude of products in place, we are just about finished with our pre-testing period, and the real mail flow begins on Sept. 10. If all goes well, the results of our second In the Wild test will be published in mid-December.
Just like a blockbuster Hollywood sequel.
We'd like to hear from readers on their thoughts about anti-spam testing. Do you think that alternate anti-spam techniques, such as grey-listing, challenge response, mail-flow throttling, etc., are worth testing? What is the best way to get the best comparative results for all the anti-spam products out there? Are there things more important in an anti-spam system than the accuracy statistics (how much spam does it catch?) and the false positive rate (how many legitimate e-mails are tagged as spam)?. Send your thoughts by adding comments below this post, or send e-mail to kshaw@nww.com.
TrackBack
Back to Testing Notes
Comments
Post a comment
