Just like with the SPEC.ORG and TPC.ORG server benchmarks, a number of tests for virtualization performance are emerging.
Most server benchmarks are designed to measure the performance of a single operating system and application on a single physical computer. The virtualization benchmarks from VMware, Intel and SPEC.org are designed with virtualization in mind ndash; several different applications, several different operating systems on several virtual machines.
VMware’s VMmark is perhaps the most popular of these benchmarks and accounts for the most participation by server OEMs such as Dell, IBM, Sun and HP. The benchmark software gives OEMs, systems integrators and others a measurement of how applications running on x86-base computers perform in virtual environments. VMmark lets testers differentiate the performance of different hardware and virtualization platforms and make decisions regarding the servers they use in their virtual infrastructures.
Users can download VMmark . The benchmark consists of a tile-based scheme and a variety of sub-tests derived from the SPEC.ORG. Each tile consists of a unit of work that may employ a collection of virtual machines that execute a diverse workload. Systems are benchmarked according to the number of tiles they can accommodate.
Although the software developed by VMware is agnostic toward hardware and virtualization software, most if not all benchmarks were performed using VMware ESX Server.
To run VMmark, users must purchase Microsoft Windows Server 2003 Release 2 Enterprise Edition, Microsoft Exchange Server 2003 Enterprise Edition, SPECjbb2005 Benchmark and SPECweb2005 Benchmark.
Results of VMmark benchmarking can be seen here
An alternative to VMmark is Intel’s vConsolidate, launched in 2007. VConsolidate works with x86-based servers and tests virtualization performance by simulating physical server performance.
VConsolidate was designed to test four workloads – Java-type workloads, database, mail servers and web. It consists of what Intel calls consolidate stack units (CSU) – four virtual machines running concurrently. With vConsolidate, users stack multiple CSUs on the system until processing approaches 100% utilization. VConsolidate lets users review the results. Among the vendors who have used vConsolidate for virtualization performance benchmarking are Virtual Iron, IBM and HP.
Finally, the SPEC.ORG is also working on a standard virtualization benchmark. Supporters of this benchmark are AMD, Dell, Fujitsu Siemens, HP, IBM, Intel, Microsoft, Parallels, Red Hat, Sun, Trigence, Unisys and VMware. It is not known when this benchmark will debut.
