How to manage petabytes of data using open source software

By Alpha Doggs on Mon, 02/11/08 - 3:26pm.

The University of California at San Diego’s supercomputing group has upgraded its time-tested data management software and made it available as an open source offering designed to handle up to petabytes of data.

The Data-Intensive Computing Environments (DICE) group at the San Diego Supercomputer Center has issued iRODS (Integrated Rule-Oriented Data System) 1.0 , which the outfit says improves upon and goes beyond the Storage Resource Broker it has developed over the past 10 years.

“iRODS equips users to handle the full range of distributed data management needs, from extracting descriptive metadata and managing their data to moving it efficiently, sharing data securely with collaborators, publishing it in digital libraries, and finally archiving data for long-term preservation,” said Reagan Moore, director of the DICE group at the San Diego Supercomputer Center, in a statement.

The heart of iRODS is a rules-engine that supports customized data management needs, such as reTtaining records in one place for a set period of time, then automatically destroying them. (Compare Storage Resource Management products)

While iRODS is simple enough to be used by an individual end user, at the San Diego Supercomputer Center it is used to handle a petabyte of data and 200 million files for 5,000 users, according to UC San Diego. The Supercomputer Center , by the way, has won a best-practices award for the design of its new data center that will help to make it a model of energy efficiency.

The National Archives and Records Administration (which funded the project along with the National Science Foundation) and Ocean Observatories Initiative are among other iRODS users.

Version 1.0 of iRODS runs on Linux, Solaris, Macintosh and AIX, with Windows support on the way. It runs on either the open source PostgreSQL database or Oracle databases. It works across multiple physical servers.

The DICE team is offering tutorials and workshops for interested users.