SOA driving interest in virtual data warehouses

When the Minnesota Department of Agriculture needed to marry data from the state’s financial systems with its own laboratory management system for billing purposes, Kurt Wood considered using a data warehouse but balked at the manpower required to create and maintain multiple copies of the data sets. He found an alternative in data federation.

Data federation, or enterprise information integration, is the ability to access multiple data sources with a single query. It works like a virtual data warehouse: The data being queried remains in its place, rather than copied to a central repository, so companies don’t have to keep duplicate versions in sync. Under the covers, the software takes on tasks such as reconciling data formats, maintaining data integrity and building an aggregate view of information.

“We’re able to get real-time financial data from the state over the WAN, and we can manipulate it here," says Wood, data and application manager at the Department of Agriculture. The agency doesn’t have to worry about incompatible formats, because IBM’s data federation software handles the translations required to link data from DB2 and Oracle databases, nor does it have to fuss with extraction, transformation and load (ETL) tools.

“We could have gone with the ETL method, or passing text files, but we’ve got just a small staff of programmers here," Wood says. “When you do something like that, the data being transferred over can be old, or someone forgets to do it, or the transfer doesn’t go right and needs to be redone. We figured, why not just tap directly into the system?"

While it’s not a new concept, data federation is gaining attention as enterprises look to deploy service-oriented architecture (SOA) environments.

The demands of SOA are a good fit for data federation, says Ted Friedman, a research vice president at Gartner. “Just as you have application services and business services in an SOA, we believe you’ll also have data services that provide for common and consistent access to data, movement of data, transformation of data and so forth," Friedman says.

The desire for more real-time reporting also is driving interest. With data federation technology, companies can tap into the most up-to-date information sources instead of querying a data warehouse fed by batch processes. A law enforcement agency could consolidate intelligence data from federal, state and local sources in a single view, for example, or a financial services firm could query a handful of operational systems to determine a customer’s current account status and pair it with historical information from a data warehouse.

A federated view of data is quicker to build and easier to modify than a data warehouse, advocates say. Plus, data federation allows for the integration of data from non-relational data sources, such as e-mails and text files.

Data overload

A host of vendors offer data federation products, including integration veterans such as BEA Systems, IBM, Oracle and Sybase, as well as smaller specialists such as Attunity, Certive, Composite Software, MetaMatrix and Sypherlink. The products emerged as companies struggled to address a common problem in today’s enterprise environments: a proliferation of independent data sources, including application-specific repositories.