CIA-backed Cleversafe announces 10-exabyte storage system

Massive repositories can be used for big data analytics

Object-based storage vendor Cleversafe today announced the availability of a storage system that can house up to 10 exabytes (that's 1 billion gigabytes) of data in a single pool of capacity.

To put a storage system of that size in perspective, 1,000 gigabytes is a terabyte, and a terabyte of storage can hold about 300 hours of video. Cleversafe's new storage system could hold 1 million times as much data as that.

Building a 10-exabyte storage system would require 4.5 million 3TB hard drives. Today's 3TB hard drives can cost as little as $150, but a storage system the size of Cleversafe's would still cost $705 million for the spinning disks alone.

Russ Kennedy, vice president of Cleversafe's product strategy, said the entire system -- with racks, networking equipment and Cleversafe software -- would run in the "single-digits" billions of dollars.

To continue reading, register here to become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

Object-based storage vendor Cleversafe today announced the availability of a storage system that can house up to 10 exabytes (that's 1 billion gigabytes) of data in a single pool of capacity.

To put a storage system of that size in perspective, 1,000 gigabytes is a terabyte, and a terabyte of storage can hold about 300 hours of video. Cleversafe's new storage system could hold 1 million times as much data as that.

Building a 10-exabyte storage system would require 4.5 million 3TB hard drives. Today's 3TB hard drives can cost as little as $150, but a storage system the size of Cleversafe's would still cost $705 million for the spinning disks alone.

Russ Kennedy, vice president of Cleversafe's product strategy, said the entire system -- with racks, networking equipment and Cleversafe software -- would run in the "single-digits" billions of dollars.

Cleversafe said it created the design for a 10-exabyte data storage system to address customers' need to capitalize on the intelligence gained through big data analytics, which require larger and larger data stores for unstructured data.

Although the company hasn't yet built out the full storage system, it has created a reference configuration that is tens of petabytes in size and dispersed in data centers in eight states, including New Jersey, California, Florida, Texas and Illinois.

"This configuration was built to prove it would work," Kennedy said. "We'll build it when [customers] want it. We have some very interested ones to date."

With worldwide Internet traffic volume increasing at a rate of 32% each year, companies looking to mine that data would "effectively analyze 80 exabytes of data per month by 2015," he said.

Cleversafe, a privately-held company founded in 2004, is well funded; it has received more than $31 million in venture money, including money from In-Q-Tel , a branch of the CIA that invests in startups.

"To any company, data is a priceless component. However, it's only valuable if a company can effectively look across that data over time for trends or to analyze behavior and to do it cost-effectively," said Kennedy. "In its true sense, Cleversafe's limitless data storage solution is a critical foundational enabler to Big Data analytics."

Big data tools are being used to analyze everything from IP traffic patterns for fraudulent activity to purchasing patterns for online retailers.

Cleversafe's new massive data storage buildout uses the same technology the company has been selling since its inception. Cleversafe's technology, which it calls Dispersed Storage, works by using a mathematical formula called the Cauchy Reed-Solomon Information Dispersal Algorithm to divide data before storing it.

The divided or "sliced" data, as Cleversafe calls it, is spread across multiple storage nodes (server appliances) using TCP/IP, typically across three or four data centers. Like RAID, the algorithm uses parity information to ensure that if any slices of data are lost or become corrupted, they can be rebuilt from the other slices.