InfiniBand is a standard that defines several key technologies that help it deliver high I/O performance and reliability with low latency in the data center: credit-based flow control, channelized I/O with hardware-based QoS, and a transport optimized for moving massive amounts of traffic with minimal load on the server.
InfiniBand uses a credit-based, flow-control mechanism to ensure the integrity of the connection, so packets rarely are dropped. With InfiniBand, packets will not be transmitted until there is verified space in the receiving buffer. The destination issues credits to signal available buffer space, after which the packets are transmitted. This eliminates congestion as a source of packet loss, greatly improving efficiency and overall performance.
InfiniBand also uses strict QoS controls implemented in hardware. When multiple servers share network resources, it's important to prevent a flood of low-priority traffic from blocking time-sensitive traffic, and the problem is compounded when multiple virtual servers are implemented on a group of servers.
InfiniBand's credit-based flow control is provided separately across many channels, providing a simple yet robust QoS mechanism for protecting traffic. Traffic protection is critical for converged wire strategies because it lets a single interconnect replace multiple, parallel networks required for clustering, storage, communications and management traffic. It also is critical for virtualized environments.
With high bandwidth and end-to-end latency in the range of 1 microsecond, 20Gbps InfiniBand (commonly called DDR, or double data rate) is helping to address data center I/O challenges and is rapidly gaining acceptance in high-performance computing centers.
Because InfiniBand was designed to connect servers and storage efficiently in close proximity, the InfiniBand transport protocol was optimized for this environment. TCP, on the other hand, is the most ubiquitous transport protocol - implemented on devices ranging from refrigerators to supercomputers - but generality comes at a price: It is complex, the code is large and full of special cases, and it's difficult to offload. InfiniBand transport was defined later, during the era of multigigabit networking and high-performance servers, and is more streamlined, making it suitable for offloading to purpose-built, efficient hardware adapters. Offloading the InfiniBand transport enables very high performance with minimal load on the host CPU, so the CPU can focus on useful application processing.
However, most users of InfiniBand also run TCP/IP for application compatibility. Some traffic between servers requires the performance and offloading of the InfiniBand transport, while other traffic requires the protocol and application compatibility provided by TCP/IP. Many InfiniBand-powered data centers use both.
InfiniBand is defined by the InfiniBand Trade Association and supported by many vendors and the OpenFabrics Alliance, an open source software community.
From: InfiniBand improves efficiencies, Network World Tech Update, 04/12/07.
Additional resources
InfiniBand Trade Association
Vendor group.
