Applications that process real-time datastreams are pushing the limits of traditional data processing technologies. These applications are characterized by the need for sub-second response times - whether they involve automating trades, monitoring networks for intrusions, or tracking credit card transactions for fraud. Applications that depend on the traditional store-and-query model cannot handle the volume and velocity of streaming data, whose value might exist only in the moment.

A stream-processing engine (SPE) is data management software that enables the execution of queries and computations - and ultimately, actions - on streaming data in real time. Previously, queries and computations could only be executed with stored data using standard database management systems. An SPE accepts SQL-like, stream-oriented, continuous queries and executes them over live event streams, outputting results in real time.

An SPE achieves real-time operation by integrating several mechanisms. First, it supports inbound processing, in which incoming event streams immediately start to flow through the continuous queries as they enter the system. The queries transform the events as they move, continuously producing results, all in main memory. Read or write operations to storage are optional and can be executed asynchronously in many cases.

Inbound processing overcomes a limitation of the traditional outbound processing model conventional database management systems employ, in which data must be inserted into the database and indexed before any processing can take place. By removing storage from the critical path of processing, an SPE achieves significant performance gains compared with traditional processing approaches.

Second, an SPE adopts a single-process model, in which all time-critical operations (including event processing, storage and execution of custom application logic) are run as part of one multi-threaded process. This integrated approach eliminates high-overhead process switches present in solutions that use multiple software systems to provide the same capabilities.

Third, an SPE provides a flexible, in-process storage model and standards-based access to external databases. In-memory hash tables are used for very fast insert and look-up operations. Embedded databases are used to ensure persistence of data and can be accessed and manipulated using SQL-style declarative queries. External, remote-process databases are accessible through standard Open Database Connectivity calls and are convenient to use when supporting legacy databases or facilitating database sharing with external applications.

Whitepapers

• Selecting Effective Virtual Directories
• Best Practices for HP Servers and HP Enterprise Virtual Array in a Microsoft Exchange
• Compliance, Protection, Recovery: A Layered Approach to Laptop Security
• Endpoint Security: Data Protection for IT, Freedom for Laptop Users
• IT Departments on Data Security: A Research Concepts Survey
• Laptop Security Tool Helps Allina Hospitals Track 97% of IT Assets

View more SOFTWARE whitepapers

Webcasts

• Managing the End User Experience-A Real World Example of Success
• Lowering the Cost of Email Archives
• Google Webcast: Why archive email? Top challenges and best practices
• High Availability for SQL Server 2005 Using Array-Based Replication and Host-Based Mirroring Technologies
• Technical Overview of Exchange Server 2007 with HP Servers and Storage
• Bringing Order and Security to your Mobile Workforce: Corporate Mobility Policy and Device Management On-Demand

View more SOFTWARE webcasts

Special Reports

• Application Performance Management Executive Guide
• Executive Guide: Building a Service Oriented Architecture
• IT Buyers Guide To: Collaboration

View more SOFTWARE special reports