Dynamic clustering provides e-mail

E-mail is becoming more important as a method of communicating. As a result, choosing a complete e-mail server for your business is critical. Organizations need a robust yet easy-to-manage e-mail system and reliable technical support from a system vendor.

Today, ISPs and IT departments face two major problems:

• Mail traffic and the number and complexity of mail access operations grow every day, requiring a robust, scalable e-mail system.
• With e-mail becoming the main business communication media, the system must provide carrier-grade availability, meeting five nines (99.999%) uptime requirement.

None of these needs can be addressed with a single server. Clusters are a must for modern large-scale installations, but building mail server clusters is much more complicated than building Web server clusters because of the high data-modification rate, complicated data processing and variety of access methods that must be supported, such as Post Office Protocol 3, Internet Message Access Protocol 4 and WebMail.

The previous generation of mail servers (developed in the early 1990s) offered scalability by partitioning user accounts - each server in a cluster had its part of the accounts created on its local disks.

A directory server kept the location information for each account, and some sort of mail multiplexer, mail router or front-end server directed user connections to the proper server, based on the directory information.

While this type of system can be scaled effectively (limited mostly by the performance and stability of the directory server), it does not improve site availability, and a failure of any server in the cluster makes all of its accounts unavailable. This type of cluster is often called a static cluster.

Other systems developed five to six years ago tried to address the high-availability issue by using a shared file system (such as Network File System file servers) with several mail servers running in parallel. Such a configuration allowed a site to provide access to all accounts even if some of the servers crashed. The problem with that type of system was data synchronization. The traditional synchronization method was based on file system locks that not only slowed all file operations four or five times, but also were unreliable when used with shared file systems.

The latest systems are based on account-level synchronization implemented with a cluster controller that uses a special inter-server protocol to ensure that any account is used directly only by one cluster member at a time.

If Cluster member A has opened Account X and Cluster member B tries to open the same account, the controller instructs Cluster member B to connect to Server A and use it as a proxy to access the account data. Synchronization within each cluster member is implemented using faster multithreading techniques instead of file system locks.

Unlike unreliable file locks, the account-level locking does not leave a trace in the file system, so no cleanup is needed if any server fails. This type of cluster is often called a dynamic cluster.

This design lets the cluster survive failure of any member, providing access to all accounts as long as at least one server survives. If the cluster controller server fails, some other cluster member assumes the role of the controller. Such a dynamic cluster meets or exceeds 99.999% availability requirements but still provides almost linear scalability because the synchronization overhead is minimal and includes only one fast transaction during account opening. The scalability of a dynamic cluster is limited only by the performance of the shared file system, and the dynamic cluster can employ fast high-end file servers or cluster file systems.

Because of the highly integrated nature of dynamic clusters, they can provide a single-server-image feature, making cluster management as simple as single-server management.

For installations of more than five million ISP-type accounts, a supercluster can be deployed. The supercluster is a static cluster of dynamic clusters - it provides virtually unlimited scalability of static clusters (implemented using cluster front-end servers) and high availability of dynamic clusters (used as back-end systems).

Butenko is president of Stalker Software. He can be reached at Vladimir_Butenko@stalker.com.