Our criteria for inviting vendors into our test and who was not selected or turned us down.
Network World maintains an online Buyer’s Guide, which allows any anti-spam vendor to submit its product information. Based on our award-winning test last year and the 12 months that the Buyer’s Guide has been in operation, we decided that any vendor who wasn’t in the Buyer’s Guide wasn’t very serious about participating in a product test, so we used that as the basis for our initial e-mail invitation list.
Main index: Spam in the Wild, The Sequel
Next, we set criteria for the test to include products that meet the needs of companies looking for large-scale, high-performance anti-spam systems. Specifically, we were looking for products that would:
1: Accept mail using SMTP, either as an application running on an existing server, as a dedicated appliance server or as a service.
2: Filter and/or tag mail according to the likelihood that it is undesirable (aka spam)
3: Forward the mail onto the corporate mail system using SMTP.
Because we were testing the products in parallel, we had to specifically exclude anti-spam products that depend exclusively on alternative technologies, such as challenge/response, or graylisting. The challenge/response system that is ‘famous’ is TMDA (tagged message delivery agent) from TMDA.NET, and the graylisting that is most famous is M-Switch from Isode.
After reviewing the 43 vendors that submitted products for the test, we asked four to drop out. Ipswitch, Kerio, Stalker Softare and Astaro all submitted products that had some SMTP in/out capabilities, but were designed and sold as mail servers with anti-spam capabilities. We also dropped Lightspeed Systems, because its anti-spam product, a transparent Layer 2 firewall, was not compatible with our test bed.
IronPort Systems, a messaging appliance vendor, was asked not to participate in the test because Opus One has an existing consulting contract with this company – including them in the test would have created a conflict of interest. If you are interested in IronPort’s spam catch rates, you can infer them from Borderware or Symantec’s numbers because all three are based on the same anti-spam engine.
We also reached out to the SpamAssassin community (see “What about SpamAssassin?”), but couldn’t find someone who could act as a representative for support and configuration assistance. However, two commercial vendors, Roaring Penguin (on Unix) and NoSpamToday! (on Windows) sent products that exposed their SpamAssassin cores. Although neither met our false-positive threshold for inclusion in the top 12 finalists (probably because of difficulty of tuning Bayesian engines and neural networks in a test lab setting), we were very pleased to have them participate in the project.
Two service vendors, FrontBridge and Message Labs, agreed to participate in the test and were configured into the test bed. However, both vendors pulled out of the test just before formal testing began, citing methodology concerns.
Full disclosure statement – After last year’s test naming the Postini anti-spam service as one of the top products, Opus One signed up as a paying customer of Postini. Some of the companies participating in the test thought that this would present a conflict of interest. We feel that any test lab that hasn’t already implemented an anti-spam system by now could hardly be considered qualified to evaluate how such a product would work. In addition to Postini, Opus One is a customer or purchases products or services from Symantec, Sophos, NetIQ, Ipswitch, and Process Software, all of which participated (or wanted to participate) in this test.
