* The two general forms of data de-duplication
Want to save money on your storage budget? Stop saving so much information. Want to save money and still stay in business? Find a way to keep and manage all that information without storing so much data.
These days, anyone looking at the marketing collateral accompanying storage and many networking products (particularly those offering Wide Area File Services – WAFS) is sure to come across two terms, “de-duplication” and “compression.” Judging by some of my e-mail, many readers apparently don’t make a distinction between these two terms. They should, for while both de-duplication and compression should be classified as data reduction, they go about their chores in quite different ways. Fortunately, if you are clear on what they do and how they do it, you can take maximum advantage of both.
Data de-duplication comes in two general forms. In its most basic implementation it makes sure that only a single instance of a file gets stored, eliminating file redundancy and thereby also eliminating the need to store multiple instances of the same file. Obviously, when it comes to large files this can be quite useful as there is rarely a reason to save several copies of any document. Just let the software save one copy, distribute pointers appropriately throughout your file systems so that the data can be accessed as needed, and suddenly … you find yourself running a sleek new storage environment. Those pointers only take up bytes, not gigabytes, so overall it’s a pretty good alternative to saving multiple copies of a 4.25 GB movie.
Some files never change (some kinds of reference data, for example, or any data kept on WORM devices), but many do. Obviously, for an operating system if a file changes it is no longer the same file it once was, and the file system can be excused from saving what to us is an additional file if it indeed differs from the old one. But what if only a single word of text has changed? Do we really need to save to separate files then? Fortunately, the answer is no.
Data reduction can be improved still further when the management software understands changes at sub-file levels. (The term “sub-file” is now used by a number of vendors to reference their ability to look at changed blocks – or even, in some cases, changed bytes – and thus to point to individual blocks or bytes rather than to whole files.) Operating at sub-file levels, we only need to identify and then point to the changed data block (the delta block), applying the same principle as before but now doing so at a much more granular level. The capability to identify such small changes is crucial to most continuous data protection (CDP) software, but it also plays a key role in reducing the amount of data that must be local and remote devices, that must be sent out over LANs and WANs during replication, and so forth.
Many vendors, including Diligent, IBM, Network Appliance, Packeteer, and Permabit, now offer data de-duplication at the sub-file level in a variety of applications spanning WAFS for data transfer, CDP and virtual tape libraries. When installed, some products can go out and begin applying data reduction to data already on your SAN or NAS devices. When that happens, such investments often pay for themselves almost immediately.
Next time: a look at data compression, at how it differs from de-duplication, and at how, by properly combining the two, you will save on your storage budget, clear up acne and get your kids into the colleges of their choice.
