NYSE accelerates BI with switch to data-warehousing

NYSE Euronext Inc., the operator of the New York Stock Exchange, Euronext and several other stock exchanges, has adopted data warehousing appliances from Greenplum Inc. and Netezza Corp. to accelerate its analysis of internal business and performance data.

NYSE Euronext has been using the Greenplum 4500 appliance for the past nine months to store and analyze data to determine where there are slowdowns in the process of taking, executing and acknowledging trades for stocks, bonds and other securities, according to chief data officer Steve Hirsch.

The time for an order to arrive at the NYSE's network, be executed, and then have an acknowledgement, or 'ack,' sent back to the trading party, is already fast: 1 to 2 milliseconds, Hirsch said. But using Greenplum to analyze bottlenecks in this system will help the NYSE "get closer to zero," he said in an interview Thursday.

The Greenplum software includes both data warehousing and BI analytical capabilities. This allows the appliance, which includes a Thumper server from Sun Microsystems Inc. and fiber-attached storage totaling 100 TB in capacity, to analyze the data itself, rather than sending large amounts of data to a separate server for analysis.

"This makes us hundreds to thousands of times faster. It's incredible," Hirsch said. He cited one task that formerly took 12 hours to run and now takes "just minutes," and another report that formerly took 50 to 60 hours to generate and now "runs in a single hour."

Accelerating trade times is an important but new emphasis for the tradition-bound, 216-year-old NYSE. It only began providing electronic trading two years ago while maintaining traditional human auctioneering.

This was partly spurred by the NYSE's merger with electronic-trading pioneer, Archipelago Holdings, in 2005 and Euronext in 2006.

The NYSE has been using Netezza's appliances slightly longer than GreenPlum's appliance in order to track and audit trades and orders, Hirsch said.

Before selecting these products, Hirsch said he "evaluated most of the products out there," including DATAllegro Inc., now part of Microsoft, ParAccel,Aster Data Systems Inc., and a Teradata Inc. appliance that includes SAS analytics onboard.

Both GreenPlum and Netezza are far faster than its prior infrastructure, which primarily consisted of Oracle data warehouses paired with BI servers running SAS, Hirsch said. Having separate servers for data warehousing and BI "doesn't work when you are processing terabytes of data per day ... it doesn't scale to our current levels."

Hirsch said that despite the NYSE doing "pretty sophisticated types of analytics," it hasn't run into problems where the Greenplum or Netezza appliances could not perform desired analytical runs.

"We look at Greenplum as the leader in terms of software-based solutions, and Netezza as the leader in hardware-based solutions," he said.

Hirsch declined to say how much he paid for the Greenplum and Netezza appliances. Greenplum has said it charges about $20,000 per TB of user data, while Netezza is estimated to charge about $29,000 per TB for its 10100 data appliance.

For more enterprise computing news, visit Computerworld. Story copyright Computerworld, Inc.