How we tested the Exchange alternatives

We invited 13 mail server vendors to participate in this test, and announced our test through the Internet Mail Consortium, an industry group of mail vendors and users. Our invitation and the test plan are here.

Nine vendors indicated their interest. Of those nine, Mirapoint and Ipswitch did not deliver product in time. Mirapoint did not provide a reason for not sending its products while Ipswitch missed the deadline because it is between major product releases and was not ready for us to look at its new mobility features. Sun sent us Java Messaging Server, but after a brief review, we determined that it was not a good fit for this test because it was aimed at much larger deployments than the other products.
PostPath, which was recently acquired by Cisco, declined to participate because of an impending change in its product offerings. MailEnable, Openwave, and Open-Xchange did not respond to our invitation.

In each case where the product did show in the lab, we tested the "commercial" version of each product. This is an important distinction, because several of the products also have limited-use, open source or community-based editions that have a smaller feature set than the commercial offering. If you like what we said about Zimbra, for example, don't assume that you will get the same feature set if you use the free version.

After reviewing the basic requirements for each product, we determined that we would need both Linux and Windows operating systems to run the mail servers, so we created base installations in VMware of Windows 2008 (64-bit edition) and RedHat Enterprise Linux 5 (64-bit edition), then applied vendor-recommended patches. These were installed on an eight-core 3GHz Supermicro server with 16GB of memory. Each product was given four CPUs and 2GB of memory. During performance testing, only the product being tested was running on the server. All servers were run on VMware ESX servers linked to a VMware Virtual Center for management and control.

For usability testing, we redirected our tester's mail stream to each product during the testing period of about one week per product, trying out the server with Thunderbird (for IMAP and POP), Microsoft Outlook (for MAPI) and the webmail interface. Where supported, we also linked an Apple Macintosh for native calendar and contact synchronization. We used an iPhone to test ActiveSync compatibility. In some cases, we used other diagnostic tools such as TCPDUMP to check for protocol behavior.

We used a Windows 2008 Active Directory server to check compatibility with Active Directory, and a Microsoft Exchange 2007 server to test migration capabilities from Microsoft Exchange.

To test performance, we used an Avalanche 2700 system from Spirent Communications to generate e-mail traffic at an offered load of 20 messages per second (equivalent to nearly 2 million messages per day) into 20 test users. Our test messages had a combination of normal e-mail with multiple MIME body parts (often called "attachments"), spam e-mail and virus-infected e-mail. We tested both with antispam and antivirus programs provided with each bundle enabled, and with them disabled.