New methodologies for 'bottomless' e-mail storage

With e-mail the dominant enterprise communication vehicle — used for everything from simple notes to purchase orders, contracts, invoices and other critical business documents — managing swelling message stores has become a primary concern.

Add to that the range of new compliance and records-retention issues, and it is no surprise that a recent survey by Osterman Research shows e-mail use growing at 20% per year and message stores growing at more than 35% per year.

But today’s most commonly deployed enterprise e-mail servers store data using database architectures that perform large numbers of separate I/O operations to complete a single transaction. To meet the demands, organizations typically add dedicated, costly storage and strictly limit individual storage capacity.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

With e-mail the dominant enterprise communication vehicle — used for everything from simple notes to purchase orders, contracts, invoices and other critical business documents — managing swelling message stores has become a primary concern.

Add to that the range of new compliance and records-retention issues, and it is no surprise that a recent survey by Osterman Research shows e-mail use growing at 20% per year and message stores growing at more than 35% per year.

But today’s most commonly deployed enterprise e-mail servers store data using database architectures that perform large numbers of separate I/O operations to complete a single transaction. To meet the demands, organizations typically add dedicated, costly storage and strictly limit individual storage capacity.

Click to see: Simple file-based e-mail store

New, open e-mail servers, however, enable a new, open messaging-storage process based on less expensive, modern filing systems that overcome the limitations of database architectures and improve overall performance. File-based storage lets these e-mail systems scale cost-effectively and decreases system-management complexity and administration overhead. This flexible approach simplifies the storage model and lets mailboxes (potentially bottomless) grow to sizes that are more conducive to the way employees use e-mail.

Performance issues

Modern Linux filing systems, such as XFS and Ext3, are fast, flexible, reliable and efficient. These systems, for example, support such features as journaling, which is used in playback of operations following a power cut, and semi-offline storage, which allows low-cost storage for rarely accessed files. Linux filing systems also support clustering, letting enterprises build file-system clusters to support any level of file-system reliability.

Leveraging a file-based e-mail store offers significant performance improvements and potential cost savings because the file system does not require multiple read/write commands between the e-mail and the storage subsystem. Performance is improved. Cost savings come from using less expensive commodity-storage systems that let IT provide much larger mailboxes economically. Open e-mail servers may also support virtualized file systems, and a single server might even run multiple filing systems, if that’s desired.

If the goal of the IT department is to reduce costs, increase performance and give each user a significantly larger mail store, a file system solves the problem at the source, making e-mail server storage easy to manage and maintain. A file-system approach also addresses:

* Storage for large data objects. Open e-mail server systems can employ single-instance storage at the file-system level for large data objects attached to messages — or even for large e-mail bodies. Each large object can be put into a separate file that can be linked to from multiple places. Single-instance storage not only saves on storage space but also provides much higher performance, because the file is written only once.

* Backup operations. Using a file-based storage system for backup operations is simple, live (no freeze or snapshot step is required), incremental, and detailed down to the message (file) level. This makes backing up the mail store as simple as backing up a file server. Additionally, file-server backup allows incremental backups (backing up just messages that have changed since the previous day) with industry-standard backup tools. And administrators can make mailboxes significantly larger as a result of backup time being eliminated.