Ethernet fabric switching for next-generation data centers

Traditional approaches to data center networking cannot satisfy the scale, bandwidth, latency and cost points required for evolving data center software architectures.

Fibre Channel and InfiniBand remain costly due to the specialized knowledge needed to implement, tune and administer, while traditional Ethernet switches fail to meet the stringent new demands. The best alternative is Ethernet fabric solutions designed specifically for the modern data center.

An Ethernet fabric provides a low-cost, high-speed, ultra-low-latency, loss-less and scalable network infrastructure. It can offer a full cross-sectional interconnect for 1 Gigabit Ethernet (GE) or 10 GE attached servers, and allows the compute and storage servers to maximize aggregate compute power in a cost effective way. Key attributes are:

• Scalability to hundreds of 10 GE ports and thousands of 1 GE ports.

• Nonblocking (wirespeed) full cross-sectional bandwidth.

• Loss-less packet delivery.

• Ultra-low latency (sub-six-microsecond latency across a multidevice core).

Conventional Ethernet networks consist of a multitier hierarchy, with bandwidth oversubscription in every tier, as shown in Figure 1. These networks are typically designed for Fast Ethernet and 1GB servers and clients, where 10GB is often used for interconnecting the Ethernet switches.

Click to see: Diagram showing overcoming Ethernet limitations in the data center

Spanning tree protocols are used for loop avoidance and active-standby resiliency. While spanning tree accommodates loop-free Layer-2 topology, it reduces the overall bandwidth efficiency of the network because of the lack of readily available alternate paths. The reason is that the time needed to reconstruct the tree, even with the so-called rapid spanning tree protocol, takes many seconds rather than the few milliseconds required. Therefore, conventional Ethernet switches are not optimized for high-density 10GB interfaces.

Ethernet fabrics, on the other hand, are based on a Clos architecture, also known as a fat-tree topology. Unlike traditional multitier Ethernet networks that oversubscribe 1GB or 10GB links to interconnect switches, Ethernet fabrics require every switching tier to be connected to the next tier closer to the root with higher aggregate bandwidth and without any over-subscription, guaranteeing a nonblocking switch fabric.

While this can be easily implemented for 1GB links, it becomes challenging for 10GB links. For 10GB server links the aggregate capacity facing the root needs to be accommodated by a group of 10GB links. In order to assure nonblocking bandwidth, these groups of 10GB links cannot be blocked as in conventional enterprise Ethernet topologies.

So Ethernet fabrics use high-capacity nonblocking 10GB switching nodes and employ Layer-2 multipath technology where all link capacity is available simultaneously to construct the fat-tree topology. Significantly, these switches utilize cut-through technology to maintain ultra-low end-to-end latency. As a result, this architecture enables full cross-sectional bandwidth through the Ethernet fabric, and the ability to scale nonblocking bandwidth with low latency is well beyond what is possible using today’s traditional approach.