Xen-based hypervisors push performance limits

We asked a Citrix spokesperson to comment on the variance in this single test, and Bill Carovano, director of technical product management for XenServer, says the variations were likely caused by the cron jobs that the guests can trigger. Without tweaking, Carovano says these can occur somewhat randomly and may lead to performance variances. In internal testing, Citrix tries to suppress cron jobs to remove fluctuations in its results.

Virtual Iron's performance put it in the overall bottom slot, but it's important to note that the results didn't lag far behind others in all cases. And, Virtual Iron did place second when hosting a single Windows 2008 Server guest across four vCPUs test, a test that gave a single virtual machine a playground of four CPU cores and 2GB of memory -- and all disk resources. That's a pretty wide-open field to run in.

I/O results favor Novell

We tested I/O performance using Intel's IOMeter to assess the number of I/Os per second that each virtual machine could deliver in both under- and over-subscribed conditions.

In the first of our two I/O test scenarios, we used six guest VMs that were assigned one vCPU each, emulating a typical non-oversubscribed server consolidation scenario. The second test made use of six virtual machines with four vCPU, SMP kernels.

Across every IOMeter test, Novell's SLES Xen blew away the competition. The results were so startling (in some cases there was a 10fold advantage in performance for VMs running on Novell's Xen hypervisor), that we retested Novell's SLES Xen across all scenarios. During these retests we carefully watched the disk I/O channel. Our tests include 70% write to 30% read ratio in order to provide large amounts of pressure on the disk channel to emulate virtualization in stressful, high-I/O environments. Servers don't typically see this ratio in many applications, but certain applications such as data warehousing, business analysis, database maintenance and batch processing typical in research applications favor writes over reads, so we test heavily.

In Novell's case we saw that the read/write transactions to disk seemed to come in large cycles, rather than the steady waves that normally typified disk activity while we were testing other hypervisors. From this evidence, we suspected the Novell system was using write caching.

When we asked Novell to comment on this situation, Santanu Bagchi, Novell's senior product manager for virtualization, confirmed our suspicions and told us that write caching is Novell's default when the virtual disk is configured as a file-backed disk as was the case in our test bed.

Write caching prevents bottlenecks when the channel is busy. But it can, in some cases, cause transactional integrity issues. But you can also argue that in many server configurations, write caching can be battery-backed. Being battery-backed staves off the transactional integrity issues by temporarily housing data to be written to disk for the life of the battery or until the transaction is written to media and verified.

In modern data centers, servers are often highly protected with availability features that prevent power outages and other conditions that can corrupt cache and render server data into garbage. It is for these reasons we let the Novell SLES Xen scores stand, realizing that systems purists will likely object to this default installation method and its potential for systems failures.

Citrix XenServer pulled down numbers low enough across most tests for us to query Citrix as to why that was the case. We were told to change the scheduler setting to use the NOOP scheduler, which should have been selected by default but because of a bug in the installer, didn't set correctly on our hardware. This change actually resulted in slightly worse numbers for Windows VMs but resulted in significant improvement with the SLES VMs. Our reported numbers, reflect the NOOP scheduler being in place.

In terms of performance, the Brothers Xen provide some stiff competition. The question is, which is more important to your VM scheme: transactional performance (XenServer is tops there) or I/O performance (Novell’s SUSE Xen screams if you can stand the caching component)? The answer could sway your decision as to which Xen hypervisor might be more suitable for your environment.

Henderson and Allen are researchers for ExtremeLabs, of Indianapolis. Contact them at kitchen-sink@extremelabs.com.