Data storage gains relief from deduplication technology

With storage space so tight, a new data-pruning technique has become all the rage.

By Jennifer McAdams, Network World
November 11, 2006 12:01 AM ET

. What's this?

Battling armies of cloned files that bog down enterprise storage operations, new data-deduplication techniques rid systems of extraneous versions of the same information - a powerful promise that is causing a stir among enterprise IT buyers.

Deployed by backup clients or software agents onto servers, desktops or laptops, data-deduplication features make use of algorithms or object-oriented processes to home in on redundant data segments.

Data deduplication and you Before you decide if this storage management technology is right for your enterprise …

ASK YOURSELF

•

Is it important to deduplicate data immediately or could the process be completed at a later time?

•

Does my organization want to pursue a disk-to-disk strategy or other means of augmenting tape?

•

Do I have back-end capacity constraints or am I lacking enough disk to house all of my data?

ASK YOUR VENDORS

•

Where does deduplication take place: at the client before sending the data, at the disk device after the data is sent or on the virtual tape library as a process?

•

Can your solution handle all of my backup streams?

•

Does your system support standards such as the IETF’s Session Initiation Protocol?

•

If deduplication will be performed at the client side, can your product ingest data fast enough to keep up with high-transaction systems?

•

What methods are used to identify duplicate information and how does the product guarantee it will not falsely label the data?

Source: David Russell, Gartner vice president

Repeated data copies that bloat total storage volume 10 to 20 times more than necessary are stomped out, freeing up gobs of extra storage space.

Data deduplication's staying power seems virtually guaranteed, because storage space is at an all-time premium. Lawyers, government regulators and corporate leaders are breathing down the necks of IT managers, who are loath to scrap any information for fear the move will haunt them in a future lawsuit or audit.

Shrinking the size of the stored data volumes seems to be one of the few options left.

"There really is a lot of buzz around deduplication. At the same time, it is a technology that is here to stay, because the benefits are so powerful," says David Russell, Gartner vice president, storage and strategies.

Data deduping for SOX

Data deduplication proved more than a buzzword for Vaalco Energy, a Houston company that harvests and processes crude oil and natural gas. "The technology satisfied a real-world need for us," says Dereck Stubbs, Vaalco IT specialist.

Vaalco's very real need for data deduplication centered on a Sarbanes-Oxley Act (SOX) financial audit that came barreling at the company last year. Vaalco had to prove quickly that its backup and recovery procedures met the statute's stringent requirements. It turned to Asigra, which packs data-deduplication functionality into its Televaulting software.

"We needed a solution in days, since the audit was going to come up in a couple of weeks," recalls Robert Walston, Vaalco IT and purchasing supervisor. "The e-mail requirements were especially tricky, since we had retention requirements to hold e-mail 'X' amount of years. There is some duplication when you are going back that far," he says.

Although Vaalco initially hit on deduplication in its scramble to satisfy e-mail retention mandates tied to SOX, the company quickly found greater benefits. It reduced data volumes to the point where formal off-site storage became unnecessary, and that gave the company peace of mind, Stubbs says.

Deduplication's role in enterprise efforts to avoid tape backup and off-site storage has many companies interested in the technology, says Heidi Biggar, an analyst at Enterprise Strategy Group. "If you free up more storage capacity, you could choose to keep data in-line. It is a powerful technology," she says.