Achieving true 10 Gig performance

Authors of the 802.3ae standard detail options to address performance issues.

From its inception, 10G Ethernet was intended to retain backward compatibility and full interoperability with 10/100/1000M bit/sec Ethernet while adding a tenfold increase in performance.

In the 802.3ae standard, attaining tenfold performance is not as simple as it might appear. A critical difference between the 10/100/1000M bit/sec and 10G Ethernet media access control (MAC) layer affects the 10G Ethernet interface’s ability to deliver 10G bit/sec of line-rate performance. The authors of the 802.3ae standard detailed three options to address this issue.

Inside the 10G Ethernet MAC – a critical difference

In 10/100 and Gigabit Ethernet, the MAC layer works in a linear manner – data moves serially in and out of the MAC layer with all the starting and ending control messages (including clocking and synchronization) embedded inside the datastream. With 10G Ethernet, it is much more complex.

To attain a 10G bit/sec bandwidth rate, the IEEE altered the way that MAC layer interprets signaling. Rather than producing a serial stream, the 10G Ethernet layer operates in parallel to interpret data. The transmit and receive paths each comprise four data lanes, and the datastream broken down into bytes is handled in round-robin fashion across the four lanes, numbered 0 to 3. On the transmit path, for example, the first byte aligns to Lane 0, the second byte to Lane 1, the third byte to Lane 2, the fourth byte to Lane 3, the fifth byte back to Lane 0, and so on.

Ethernet frames have clearly defined beginning and ending boundaries, or delimiters. These are marked by special characters and a 12-byte interpacket gap (IPG) that dictates the minimum amount of space or idle time between packets.

Because of the parallel nature of the 10G Ethernet MAC layer, it is impossible to predict the lane in which the ending byte of the previous datastream will fall. This makes finding the starting bit – a requirement for maintaining timing and synchronization – more difficult. The 802.3ae standard mandated an elegant solution: the “start control character,” or very first byte of a new data frame, must always align on Lane 0. However, this solution complicates the way the MAC handles the IPG, directly affecting performance. Nevertheless, the IEEE provided three options for the vendors to address this issue: 1) pad (increase), 2) shrink, or 3) average the “minimum” IPG.

Padding increases the IPG

If the ending character of the previous packet falls into 12th slot in the 12-byte IPG, the MAC layer does nothing and maintains the 12-byte minimum IPG, and the starting character of the next packet automatically aligns in Position A of Lane 0. If, however, the ending character of the previous packet falls into other positions, the MAC layer must add bytes respectively into the IPG to ensure that the starting character of the next packet aligns properly in Position A of Lane 0. This results in a minimum IPG that ranges from 12 to 15 bytes (12-byte minimum plus additional pad).

Increasing the size of the IPG beyond the 12-byte minimum decreases the available bandwidth on the 10G Ethernet link by as much as 10%, depending on packet size.

Shrinking the IPG

After padding bytes to the IPG to ensure the starting character aligns to Position A, the MAC eliminates the middle column of idle characters. This results in an IPG that ranges from eight to 11 bytes (12-byte minimum plus padding minus the four idle bytes).

Decreasing the 10G Ethernet IPG by eliminating the middle column of idle bytes provides an additional 5% available bandwidth on the 10G Ethernet link, depending on packet size.

Averaging the IPG

This option uses a combination of options 1 and 3 with the addition of a deficit idle counter, which keeps track of the number of added or deleted idle bytes (ranging from 0 to 3). In some cases, the MAC layer will add and, in others, eliminate bytes. Over the long run, the net result will be an average 12-byte minimum 10G Ethernet IPG.

Averaging the 10G Ethernet IPG ensures that the port delivers 100% of the available bandwidth and lets the connection maintain zero-loss line-rate performance.

Eisenberg is director of technical marketing at Force10 Networks. He can be reached at marshall@force10networks.com.