Open source tool emerges for managing petabytes of data

The University of California at San Diego's supercomputing group has upgraded its time-tested data management software and made it available as an open source offering designed to handle up to petabytes of data.

The Data-Intensive Computing Environments (DICE) group at the San Diego Supercomputer Center (SDSC) has issued iRODS (Integrated Rule-Oriented Data System) 1.0, which the outfit says improves on and goes beyond the Storage Resource Broker it has developed over the past 10 years.

"iRODS equips users to handle the full range of distributed-data-management needs, from extracting descriptive metadata and managing their data to moving it efficiently, sharing data securely with collaborators, publishing it in digital libraries, and finally archiving data for long-term preservation," said Reagan Moore, director of the DICE group, in a statement.

The heart of iRODS is a rules engine that supports customized data-management needs, such as retaining records in one place for a set period of time, then automatically destroying them. (Compare Storage Resource Management products.)

While iRODS is simple enough to be used by an individual user, at the SDSC it is used to handle a petabyte of data and 200 million files for 5,000 users, according to the university. The center, by the way, has won a best-practices award for the design of its new data center that will help to make it a model of energy efficiency.

The National Archives and Records Administration (which funded the project along with the National Science Foundation) and Ocean Observatories Initiative are among other iRODS users.

Version 1.0 of iRODS runs on Linux, Solaris, Macintosh and AIX, with Windows support on the way. It runs on the open source PostgreSQL database and Oracle databases, and works across multiple physical servers.

The DICE team is offering tutorials and workshops for interested users.

For more on network research, visit our Alpha Doggs blog.