Optimizing virtualized server I/O with SR-IOV

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter's approach.

Server virtualization is being deployed on an almost universal basis to reduce costs and fully utilize data center resources. With the progression to powerful multi-core servers, greater memory capacities and higher bandwidth network pipes, it has become necessary to rethink I/O optimization.

The result is several network technologies, including Single Root I/O Virtualization (SR-IOV). SR-IOV is a PCI-SIG standard that allows a PCI Express (PCIe) I/O device to appear as multiple physical and virtual devices.

IN THE NEWS: PCI building thinner interconnect for handheld devices

Conventional virtual server hypervisors provide virtual machines (VMs) with a set of resources that emulate the functionality of a physical server, allowing each VM to run independently. For I/O, that means each VM uses a virtual I/O device that is presented by the hypervisor. Although this works well for virtualization, it's usually not efficient. This is especially true for received I/O.

With multi-core servers the hypervisor performs the following steps to process received I/O:

• One of the CPU cores is interrupted to inspect the packet and determine which VM should receive it.

• The core that's servicing the VM is interrupted and processes the I/O.

• The core that initially received the packet returns to its normal workload.

Each of these steps slows down I/O and uses significant CPU resources. SR-IOV streamlines this process by introducing the concept of physical and virtual functions:

• Physical function (PF) -- There is at least one PF for each physical port on an adapter. In some cases, adapters can be partitioned into as many as four ports per physical port. In this example, there could be four PFs per port, or a total of 8 PFs for an adapter with two physical ports. The key differentiation is PFs have full configuration capabilities. They are associated with the hypervisor and can be managed like physical devices.

• Virtual function (VF) -- VFs are associated with VMs and are limited to processing I/O streams, basically moving data. They don't support management of the physical device. The number of supported VFs will vary and will likely center around 64 VFs per physical adapter.

Although the SR-IOV standard applies to network and storage I/O, the current and expected implementations are for networking only. Performance concerns for virtual servers are focused on Ethernet traffic that typically creates the largest I/O demand and uses the largest amount of server resources. Storage I/O uses far less overhead and usually achieves full line speed.

Advanced management solutions

One of the conventional solutions to improve VM performance is direct I/O (or pass-through), which requires an assignment of a unique physical port to each VM, bypassing the hypervisor. This method improves performance, but is limited by the number of physical ports that can be attached to the server.

It also results in a complex and costly collection of adapter ports, switch ports and cables. The direct I/O option is also not compatible with migration of a running VM. Migration requires a manual process to shut down, move and restart the VM. Clearly this is less than optimal.

With SR-IOV, VFs are assigned to VMs. This allows one adapter port, switch port and cable to support direct I/O for many VMs. If appropriate, multiple VFs can be assigned to one VM. For example, a VM could be assigned VFs from each physical port of a two-port adapter for high availability.

For received I/O, the server core that is assigned to the VM and its associated VFs executes all of the processing for a packet. There's no need to interrupt cores that are assigned to other VMs. To further enhance performance, I/O's between VFs on the same PF can be processed by the adapter using an internal Layer 2 switch, eliminating routing through a physical switch.

Finally, SR-IOV is compatible with migration of running VMs.

The ecosystem for SR-IOV is in the process of bootstrapping itself. It requires support by suppliers of adapters, switches and hypervisors. It also requires support by server vendors that will be adding management tools to fully enable robust solutions. SR-IOV is currently supported with Kernel Virtual Machine (KVM) in Red Hat Enterprise Linux 6 and SUSE Enterprise Linux 11 (and later).

Other hypervisor providers are expected to announce SR-IOV support.

SR-IOV will be a key technology to optimize I/O for virtualized servers, enabling higher virtualization ratios for maximum cost savings. SR-IOV will provide a much more cost-effective solution than multiple physical ports and will be fully compatible with VM migration.

Emulex is the leader in converged networking solutions, providing enterprise-class connectivity for servers, networks and storage devices within the data center. Emulex's Fibre Channel HBAs, NICs, CNAs, controllers, embedded bridges and switches, and connectivity management software are trusted by the world's largest IT environments.

Learn more about this topic

The dark side of server virtualization

Microsoft: Next level of virtualization unlocks server OS, applications

Survey finds many disappointed in virtualization, cloud computing

Editors' Picks
Join the discussion
Be the first to comment on this article. Our Commenting Policies