How we tested the iSCSI SAN servers

We tested iSCSI servers by installing and configuring them, putting them through a series of typical tasks, and running performance tests.

Our test bed was based on a dedicated data network provided by an Enterasys C2G124-48 Gigabit Ethernet switch, along with a dedicated control network using another Enterasys switch C2G124-48 switch. The two networks were connected to each other and to our production lab network using a Nokia firewall.

Each iSCSI SAN server was connected directly to the Enterasys network using as many Ethernet ports provided by the vendor with each server. If it was possible to separate control and data traffic, we did so by connecting the control ports to our control network. In some cases, it wasn't possible to separate out the control and data networks, so we ran both control and data over the same network.

Many storage servers support "jumbo" datagrams, a non-standard extension to Ethernet that allows for larger Ethernet frames (up to 9000 octets is common). Depending on the device, this can increase total system throughput by reducing overhead used for packet headers and per-packet processing. The Enterasys switch supported "jumbograms," so we turned on jumbograms for every server that supported it.

Another performance enhancement often used is link aggregation (sometimes called "bonding" or "teaming"), which combines multiple Ethernet ports into a single virtual port that can accommodate the throughput of the component ports and provide high availability. Where link aggregation was supported, we made use of it.

In some cases, we had to configure link aggregation manually on the Enterasys switch; in others, the storage server supported the link-aggregation control protocol (LACP) which automatically brought the ports together. In that case, we simply verified that the Enterasys switch and device agreed on the link aggregation configuration.

We attached four identical servers to the data network, each with a single dedicated 1Gbps connection. Each server also had a 100Mbps connection to the control network. We used an Avocent AMX5100 KVM to control the servers. The servers were all dual-Xeon systems with 3.0GHz CPUs and at least 2GB of memory. Each booted from a locally attached SCSI drive.

Two of the servers used dedicated QLOGIC QLA4050C iSCSI adapters. These adapters connect to the data network with TCP/IP, and handle all of the iSCSI protocol on the adapter directly. This reduces the load on the host and makes certain operations, such as booting directly from an iSCSI virtual disk, much simpler. The other two servers used Intel Pro/1000MT adapters provided by Intel to connect to the data network. Both the QLA4050C and the Pro/1000MT adapter are 64-bit-wide PCI-X 133 MHz adapters with a single gigabit Ethernet connection.

Originally, we had configured two servers with Windows 2008 (one each with QLogic and Intel adapters) and two servers with Centos 5 Linux. We used the Intel adapters with the native iSCSI initiator, Microsoft's own in the case of Windows 2008 and Open-iSCSI on Centos 5. However, the Linux servers proved to be extremely problematic when we came to performance testing. Although we had heroic assistance from the technical support team at Reldata, we were unable to complete testing with Linux and Iometer. We found that the results with our benchmark tool, Iometer, were not credible on the Centos 5 Linux platform using either Qlogic or Intel adapter. This pointed to some interaction between Iometer and the operating system. Two of the iSCSI server vendors confirmed known problems between Centos 5 Linux and Iometer, so we dropped Centos 5 from the performance test and replaced it with more Windows 2008 Servers. However, we did continue to use Centos 5 Linux and the Open-iSCSI initiator as part of our interoperability and usability tests.