Real-time needs drive data retooling

Business battles are fought in real time, and IS must keep pace. Real-time business intelligence infrastructures promise a never-ending stream of fresh information, insight and decision support to frontline knowledge workers.

Nevertheless, real-time business intelligence has not graduated to enterprise primetime yet. Most production business-intelligence implementations rely on data warehouses, which consolidate operational data loaded via scheduled batch transmissions rather than real-time updates from source databases. As a result, many organizations have rich stores of historical data in their data warehouses, but few contain information that is refreshed continuously.

A traditional data warehouse operates in store-and-forward mode, introducing latency into data delivery to reports, dashboards and other business intelligence applications. Most of today's data warehouses have been optimized for specific latency-producing operations: extraction, transformation and loading (ETL) of data from operational database management systems (DBMS); retention of that data in persistent repositories; and retrieval of that stored data into reports, graphical dashboards, multidimensional online analytical processing cubes and other business intelligence outputs.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

Business battles are fought in real time, and IS must keep pace. Real-time business intelligence infrastructures promise a never-ending stream of fresh information, insight and decision support to frontline knowledge workers.

Nevertheless, real-time business intelligence has not graduated to enterprise primetime yet. Most production business-intelligence implementations rely on data warehouses, which consolidate operational data loaded via scheduled batch transmissions rather than real-time updates from source databases. As a result, many organizations have rich stores of historical data in their data warehouses, but few contain information that is refreshed continuously.

A traditional data warehouse operates in store-and-forward mode, introducing latency into data delivery to reports, dashboards and other business intelligence applications. Most of today's data warehouses have been optimized for specific latency-producing operations: extraction, transformation and loading (ETL) of data from operational database management systems (DBMS); retention of that data in persistent repositories; and retrieval of that stored data into reports, graphical dashboards, multidimensional online analytical processing cubes and other business intelligence outputs.

It is possible to retool data warehouses to support real-time business intelligence. Some data warehousing vendors have begun to address these requirements in their products. Doing so requires that data warehouses - as enterprises' master data management hubs - be redesigned to serve also as real-time, application-layer data routers (in the broad sense of that term). For example, NCR Teradata's active data warehousing adds support for near-real-time ETL and data delivery. Just as important, the vendor has added the policy-driven event detection, processing and notification features needed to manage the flow of real-time events between data sources and consumers, as brokered through the data warehouse.

Though organizations are beginning to use active data warehouses for real-time business intelligence, no one is seriously considering deploying them as general-purpose, application-layer routers, because data warehouses usually are deployed in hub-and-spoke configurations and thus can become significant bottlenecks. Some in the industry have proposed data warehouse federation to alleviate the potential bottleneck, but most federation scenarios are fundamentally hub-and-spoke, relying on common ETL tools, metadata repositories and data staging areas.

Fortunately, other architectural approaches for real-time business intelligence are being explored. Some firms deploy an operational data store, which is similar to a data warehouse but contains only the most current consolidated data fed in through ETL tools. Another popular approach is enterprise information integration (EII), which supports real-time, federated query and update across distributed source DBMSs.

Unfortunately, there are no industry best practices for real-time business-intelligence requirements. Companies must sort through diverse approaches and try to implement them to leverage and extend their traditional, data-warehouse-based business intelligence environments.