We know, you’re thinking that this is a trick question. Everyone knows that multiple PCIe lanes of 5Gbps each trumps a single 6Gbps SAS link (Serial Attached SCSI). Even we can do that math. And what about latency, everyone knows that bus-attached devices have less latency than going through an adapter to a SAS-linked SSD. The SCSI Trade Association says ‘not so fast.’
True, a four-lane PCIe can deliver upwards of 20Gbps in total throughput, but filling that bandwidth is not easy when the access latency of flash memory is on the order of 25 usecs. The real advantage of PCIe is that it supports 25 watts of power as opposed to 9 watts on the SAS link. This power is necessary to access multiple flash devices: the more power, the more simultaneous access to flash chips. But to gain this bandwidth potential, PCIe faces some real trade-offs:
• No multi-host protocol — This means that you are dedicating a very expensive device to a single server. I/O virtualization devices like the NextIO vSTOR can mitigate this, but limitations still remain in terms of sharing resources over a pool of servers and single points of failure.
• Availability constraints — Compared to a server with dual SAS host bus adapters, availability is severely limited in the case of a single point of failure.
• No hot-swapping — A server needs to be physically disassembled in order to service a single PCIe drive, as compared with a JBOD array that supports hot swappable SAS (or SATA) SSDs.
• No ubiquitous protocol — PCIe users are forced to use proprietary drivers provided by the manufacturer. The only exception is the LSI PCIe SSD which uses – guess what – a SAS driver. SAS has advantages in security, end-to-end protection, zoning and longer distances. It can also support both SAS and SATA devices on the same interface.
• No compatibility with existing management software — SAS can be managed through any supplier’s management suite, providing continuity for storage administration.
• Bus channel blocking — Not only is the server doing basic management of flash chips (wear-leveling, error correction coding, page management et. al.), it is also getting its Bus (half of the bandwidth in the four-lane example) dedicated to the relatively slow (when compared to DRAM) flash memory.
The ability of the SAS interface to aggregate multiple drives and share them between an array of host devices provides significant ability to fill the 6Gbps pipe. Concurrency of access to multiple full-duplex drives provides overlap of flash latency on multiple devices, and can provide IOPs and transfer speeds that actually exceed many PCIe implementations, as shown by benchmarks provided by the SCSI Trade Association. As the SAS road map proceeds to 12Gbps and mechanical changes specifically designed for SSD cabling to address the higher wattage requirements of the flash interface come online, the disadvantages of SAS at the raw speed level will become increasingly mitigated.
PCIe is also undergoing significant development in standards, speeds and lower costs at the board level due to further integration of functionality in the next generation of Intel and AMD processors. PCIe SSDs will remain a major corner of the SSD market envelope as we discussed in the last newsletter. New products from vendors such as OCZ and Virident are examples of how rapidly the performance of these devices are improving and corresponding decreases in cost per gigabyte. However, we do not expect these products to crowd out drive format and memory array appliance entries in the SSD market in the future, as some have predicted.
