* How data compression differs from de-duplication
Last time we talked about data de-duplication, also called “single instancing.” Today we’ll take a look at data compression, an alterative (but still entirely synergistic) approach to data reduction.
Data compression in one form or another has been with us at least since the 1980’s. Probably the most familiar example today is the ZIP file, but Microsoft has had compression built into Windows for quite a while as well.
We use compression to reduce the amount of space a file takes up. All approaches go about this in the same way: during compression, they reduce the “unnecessary” space within a file, called “white space,” leaving just the useful data. Then, when the file is reassembled for use, they return the white space to the document, relying on some proven algorithm to distribute it accurately. The fly in the ointment is that what works for one type of file does not necessarily work particularly well for another, so a technique that understands the white space in a CAD model does not necessarily provide good service to an Excel spreadsheet, and can’t reduce the size of a speech or graphics file optimally.
The trick of course is to remove unnecessary data without losing necessary information. When it’s done incorrectly, we lose the high and low notes in an MP3 file and some of the color values in a digital photograph. You can well imagine the comparable problems in an incorrectly compressed spreadsheet or Word document.
Because all data compresses differently, compression applications need to understand the documents they compress. This is important with all aspects of compression, but particularly so with Wide Area File Services (WAFS), which reduce data that must be sent in order reduce the bandwidth that must be purchased.
Such “application-versance” is also important when data is written from a virtual tape device to physical tape. The VTL that understands the data and tape type it has can intelligently size each virtual tape prior to tape drive compression, and by doing so can provide a one-to-one relationship between each virtual tape and its physical counterpart (and thus maintain full compatibility with the backup software). The most noticeable aspect of this would be a reduction in the number of physical tapes required in going between the virtual and the physical device – full utilization of the physical tape media would offer significant savings in large environments.
The key point here: a product that works well for one file type probably won’t work optimally for another unless it has been tuned to understand that one as well. If you are really interested in the nitty-gritty of this stuff, one place to look is here.
Using both single-instancing and compression to eliminate redundant data certainly optimizes storage capacity, but it does more. If the vendors’ promises are correct, these two technologies will save you big bucks by reducing the storage you have to buy and the bandwidth you have to use (of particular value to remote offices where bandwidth is always at a premium, for example). The stuff really does deliver the goods.
Somewhere along the way, some clever vendor may even make a reasonable case for the fact that data reduction also helps clear up acne and will get your children into the colleges of their choice. The other day I came across this: “There is a growing incidence of adult acne, caused by stress, hormonal imbalance and poor elimination”. Certainly reducing the amount of data you need to store will reduce stress and help with one aspect of elimination.
The jury is still out on the college thing though.
