Update on the workings of the Storage Performance Council
By James E. Bagley, senior analyst, Storage Strategies NOW
[Editor’s Note: I asked Jim Bagley, a new analyst with Storage Strategies now to update us on the workings of the Storage Performance Council. Jim is an expert in solid state drive and memory technologies, as well as in healthcare IT. You can reach him at jbagley@ssg-now.com.]
According to its mission statement, the Storage Performance Council (SPC) is a non-profit corporation founded to define, standardize, and promote storage subsystem benchmarks as well as to disseminate objective, verifiable performance data to the computer industry and its customers. The member-supported organization maintains specifications for benchmarks and publishes the results of the testing performed by its members after a rigorous audit and peer review process.
We recently spoke with Walter Baker, SPC administrator, to gain insight into the organization and its methodology. Walter indicated that memberships are available to manufacturers and end users of computer storage products, and include Full, Associate and Academic membership levels.
When asked about the recent SPC news of the first standard benchmark to include energy use measurements, adding a new and critical dimension to performance reporting, Baker indicated that this was just the beginning, and the organization was forging ahead with additional energy measurement instrumentation. The performance benchmarks come in several versions with measurements and workloads designed to provide a real world, vendor-neutral picture of performance at a component or system level. The increasing use of solid state devices, and their accompanying high speed and low energy use have focused much attention on standard benchmarks, and the SPC is undergoing significant growth as new technology companies, wishing to tout their extreme performance, enter the storage field.
After joining SPC, members have access to the benchmark toolkits that provide the software and instrumentation means to perform the testing in their facilities. In order to be published by the SPC, a rigorous audit of the facility and testing means is performed by the SPC administrator, followed by a disclosure of the results to the membership and a 60-day period of peer review during which any compliance issues can be raised by the SPC membership. Once the review is completed and any compliance questions cleared, the results are published on the SPC Web site and may be publicly reported to the media. As an example, IBM recently announced impressive results of its new solid state drives using the new energy extension to the SPC component benchmark, called SPC-1C/E.
“Membership starts with an online application and an analysis of the applicant’s business interest and membership level,” Walter indicated. The SPC Steering Committee must approve all applications by a vote. Full membership fee is $11,000. Non-voting Associate memberships carry a $3,500 fee, and Academic memberships are available at no fee. The costs associated with the audit of a benchmark are charged on a time and materials basis, and may include travel expenses to the member facility.
“I always want to visit the member’s lab for their first audit, and, depending on the results, subsequent visits may be necessary,” said Baker, who personally performs the audit and administration function. The benchmark instrumentation has plenty of safeguards in the form of audit trail, making it impossible to fake the results, no matter how tempting it may be to try, in the highly competitive storage market environment. “I have plenty of binary fingerprints to follow to ensure proper testing.”
The development of standards and specifications for the benchmarks can spark lively debate among members serving on committees, so the meetings are closed and held under non-disclosure provisos. “Making standards is like making sausage,” Walter bemused, “and so the presence of media in the deliberations is not a good idea.”
But since post-publication industry analyst commentary and guidance are welcome, Walter and the steering committee are exploring methods that analysts and media can participate in the formulation process.
The SPC Web site lists approximately 40member organizations, among them 3PAR, Data Direct Networks, Fujitsu, HP, Hitachi, IBM, NetApp and Seagate.
Information about the SPC, including benchmark specifications, results, and a membership application can be obtained at www.storageperformance.org.
