Ethernet Adapts for Data Center Applications – Part 1

Many networks need to marry their Fibre Channel SAN protocols to Ethernet.  But Ethernet is an easy-going protocol (let's call it West Coast) and Fibre Channel is a structured protocol (East Coast). "Data Center Bridging" will be to these two what the central country is to the Coasts - the means by which the two connect.

Let's start with a quick review of Ethernet operations today. In general we build hierarchical networks with clearly defined roles and boundaries between L2 and L3 traffic. This provides many benefits ranging from scale and fault tolerance,  to change domains and has a great track record of success. A lot of thought and lessons learned from the school of hard knocks is behind those designs and when we peel the onion on the design considerations and configurations we see a lot of time, energy and effort is placed on controlling STP.

STP, like it or hate it, is inherent in most Ethernet topologies to provide the loop-free topology we need while having redundancy to save our bacon if there is a failure. We've implemented dozens of nerd knobs like LoopGuard, RootGuard, Bridge Assurance, PVST+, MST and more to lock down the topology and make it behave in a desirable and predictable fashion.  We also have spent a lot of time understanding a multitude of technologies to simplify and minimize our STP domains as well because we recognize STP isn't easy for most customers and any network engineer will be more than happy to share a "This one time..." story related to it. It is the life-blood of a redundant Ethernet network.

If we compare and contrast Ethernet and Fibre Channel (East Coast/West Coast derived protocols), we see that  Ethernet is the laid back West Coast  surfer  that will try to deliver your frames on time and in order, but if they don't you get a "Sorry dude, couldn't make it happen" response but you'll be OK because TCP will retransmit or for UDP, it was probably real-time traffic and hopefully didn't notice the clipping.

Fibre Channel on the other hand is a very structured and regimented East Coast protocol that won't tolerate delays and drops. Significant efforts are made to ensure this on-time delivery including a hop-by-hop buffering system and classes of service that can guarantee in-order delivery. If Fibre Channel frames hit the deck, bad things happen. Most applications and operating systems don't like it when their storage is pulled out from under them while the network converges - recent personal experience was a great reinforcement of this principal. Wonder why your SAN admins get nervous when you mention FCoE? The laze-faire approach of Ethernet is the reason.

So how do we solve this challenge and merge the East Coast rigidity of Fibre Channel onto the West Coast laid back Ethernet - Data Center Bridging is the answer. Data Center Bridging (DCB) is a collection of enhancements to Ethernet that make it capable of providing lossless transport for protocols like FCoE. The IEEE Data Center Bridging Task Force has been working on these enhancements and includes the following IEEE 802.1 specifications:

• IEEE 802.1 Qbb - Priority-based Flow Control (PFC) . PFC enables us to leverage the existing  Class of Service (CoS) model with 8 delineations and provide both lossless and lossy priorities at the same time on the same wire. It allows us to pause the traffic that isn't categorized or marked as lossless to ensure the lossless traffic goes through.
• IEEE 802.1 Qaz - This standard has two main sub-components that are important in the evolution of Ethernet. Enhanced Transmission Services (ETS) allows bandwidth guarantees and priority groups to enable deterministic services in the network. Data Center Bridging Exchange (DCBX) enables dynamic learning of the network as switches are interconnected and will leverage Link Level Discovery Protocol (LLDP). DCBX should help ensure consistent network characteristics and guarantee that a non-lossless switch doesn't have a frame that requires lossless capabilities switched onto it.
• IEEE 801.2Qau - Quantized Congestion Notification (QCN). IEEE Qau provides the ability for the network to notify senders of congestion in the network to slow their transmission rate rather than drop frames. QCN enabled switches will have  the ability to sample traffic as it traverses  and if congestion is experienced, reverse the MAC addresses and notify the sender of the congestion. Some in the industry will argue that QCN is a pre-requisite for multi-hop FCoE but as they operate at different layers there is not a dependency.

To summarize, DCB are IEEE standards that improve Ethernet and provide the ability to carry lossless traffic, such as Fibre Channel.

So, if you have a conversation with your SAN team and calm their fears over the network losing their Fibre Channel frames, the next hurdle will be multi-pathing.  Stay tuned: this will be the subject of my next post.