When I worked for Sybase, much of the advertising we published was based on the TPC performance benchmark figures of Sybase SQL Server compared to the competition. The Transaction Processing Performance Council (http://www.tpc.org/) according to its web site, "is a non-profit corporation founded to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry". Very noble. This attracts both hardware and software vendors to prove that their products are the best on the market. Of course, the reality is that you are only as good as your last benchmark, as the sports cliché goes...
As leaders in the Client/Server database market, Sybase and Oracle in the mid-1990s were competing head-to-head on database performance and price per transaction. Every month new benchmarks were issued to prove that one was faster and cheaper than the other. Of course, the companies regularly "leap-frogged" each other. Unfortunately, Sybase System 10 was released before its time and Oracle 7 grabbed significant traction in the market with superior performance figures and the rest is history. Sybase System 11 was better but by then the proverbial horse had bolted. IBM's DB2 at that time was perceived as a mainframe database so was viewed as a legacy product, but now, since its acquisition of Informix, IBM has a compelling multi-platform story and has re-established itself in the VLDB market along with Oracle. Microsoft SQL Server has been gaining steady market share based on Total Cost of Ownership (I wrote about this in a previous blog entry) but was always deemed to be wanting in the enterprise server market for very large databases. With emphasis on data partitioning and filestream data with other enterprise level features for multi-terabyte databases, Microsoft is hoping that that perception changes with SQL Server 2008.
The TPC-C benchmark application (currently V5.9) is designed to be representative of an OLTP yardstick for hardware and software configurations. Results are published by Performance and by Price/Performance. As of the end of September, top of the list for performance is IBM's DB2 V9.5 running on an IBM Power 595 Server (64-way) under IBM AIX 5L V5.3 with an astonishing 6 million transactions per minute. Who said the single vendor solution was dead? Oracle 10g is second running on an HP Integrity Superdome with 128-way server using Itanium2 processors under HP-UX 11i v3. Results: Four million tpm. So where is Microsoft SQL Server? Well, it's 10th with Microsoft SQL Server 2005 running on the HP Integrity Superdome 64-way server under Windows Server 2003 SP1 64-bit. Just over one million tpm. Not bad. We used to be impressed with a thousand transactions per minute. But hang on, that Microsoft result was published a couple weeks after SQL Server 2005 was released. That's a long time ago. Even in the Price/Performance category, SQL Server is only 4th behind Oracle 11g. Sybase SQL Anywhere 11 is making a notable 5th. Has Microsoft been sitting on its hands?
Enter the new TPC-E, which apparently is a benchmark aimed at modeling a multi-tier brokerage application. It is currently at V1.6. In this benchmark, Microsoft is doing quite well with SQL Server 2008 x64 leading the way running on an IBM System x3950 M2 64-way server under Windows Server 2008. In fact, Microsoft has the top 10 results. A clean sweep! Not surprisingly, Microsoft has gone to press about this just last week: "Microsoft SQL Server 2008 Is the Value Leader According to Latest Benchmarks". Good for Microsoft.
But wait, Microsoft is the only software vendor listed under "All results". Looks like no other database vendor is taking this benchmark seriously. Different vendors are now using different yardsticks which the TPC set out to correct in the first place. This is a shame.
Sounds a bit political though, doesn't it? Sign of the times I am afraid...
Cheers
Brian
Recent posts:
Resource Governor 2008 – First look
Brian D. Egler, MCITP/MCSE/MCT 2009, is currently an instructor with Global Knowledge, teaching various Microsoft training courses. He is a SQL specialist with a focus on SQL Server, Windows, .Net and XML. Egler has been a technical instructor for over 20 years and has more than 10 years experience with SQL Server, data modeling, database design, application development including IMS, DB2, Sybase. Every year he runs the Boston Marathon for cancer research.
The Leader in Network Knowledge
