Many networks use 1500-byte MTU size, but the MTU size can be reduced by encapsulation, tunneling or other overlay network protocols. These situations reduce the end-to-end effective MTU size which reduces throughput and network efficiency and sometimes causes application problems. Many network devices now support larger sizes of Ethernet frames and use of Jumbo Frames is becoming more common. This article covers how to determine if your network is capable of using Jumbo Frames and if you should enable this feature.
Compensate by Increasing the MTU Size
A previous article on "MTU Size Issues" discussed the issues of MTU size, how Path MTU Discovery (PMTUD) is performed and the results of fragmentation. Today, the vast majority of data networks use a default 1500-byte MTU size. This is because this is the default Ethernet MTU size for hosts and switches. Frequently, links between enterprise routers and the upstream ISP routers only support 1500-byte MTU. This is also true on the links between MPLS CE routers and PE routers. The primary issue with MTU size occurs when encapsulation is taking place between sites that only support 1500 byte MTU.
It would be highly desirable to be able to increase the MTU size over the WAN. If the MTU size was able to be increased throughout the traffic path across the WAN, then the added encapsulation overhead could be compensated for by the WAN interface of the routers. This would eliminate the need to reduce the MTU size on the tunnel interfaces, adjust MSS, and alleviate the routers from performing any fragmentation.
Jumbo frames are network-layer PDUs that have a size much larger than the typical 1500 byte Ethernet MTU size. These jumbo frames are sometimes also called "Giants". Also, the term "Baby Giants" refer to frames that are just slightly larger than the 1500 byte Ethernet MTU size (e.g. 1998 bytes). In some situations, jumbo frames can be used to allow for much larger frame sizes if the networking hardware is capable of this configuration. Most modern routers and switches are capable of jumbo frames and much of the networking hardware within the data centers are capable of this.
To configure the jumbo frame MTU size on a Cisco IOS device, just enter the MTU command on the interface configuration like this:
Router(config)# interface GigabitEthernet 4/1
Router(config-if)# mtu 9216
The "show interface" command will verify the interfaces new MTU size.
For other manufacturers equipment you just have to look for a configuration command within the physical or virtual interface that allows you to set the MTU size greater than 1500 bytes.
The key concept to keep in mind is that all the network devices along the communication path must support jumbo frames. Jumbo frames needs to be configured to work on the ingress and egress interface of each device along the end-to-end transmission path. Furthermore, all devices in the topology must also agree on the maximum jumbo frame size. If there are devices along the transmission path that have varying frame sizes, then you can also end up with fragmentation problems. Also, if a device along the path does not support jumbo frames and it receives one, it will drop it.
Jumbo Frames should not be confused with jumbograms. When discussing communications protocols, "frames" are the Protocol Data Unit (PDU) used at Layer 2 (Data Link Layer) of the OSI model, and "packets" are the PDU used at Layer 3 (Network Layer) of the OSI model. The term "datagrams" is the PDU used at Layer 4 (Transport layer) of the OSI model. A jumbogram is a larger Layer 3 packet that exceeds the link MTU size. IPv4 is capable of generating payloads up to 65535 bytes, while IPv6 is capable of a 32-bit "Jumbo Payload Length" size within a Hop-by-Hop option header. Therefore, IPv6 could support a ridiculous 4.2GB payload. Clearly, that packet could not be transported on any type of common networking interface. Just imagine the repercussions of a retransmission.
LAN Speed Records
To be able to go fast you need to have larger frame sizes. With larger frame size, and thus larger payload size, you can have less protocol overhead and are able to achieve higher protocol efficiency. In other words, your "Goodput" improves with larger frame sizes.
There are organizations that push the limits of modern computer and communications technology and try to set records for the highest amount of throughput. These are called LAN Speed Records and there are informal competitions and documented record holders. Three years ago Solarflare set a LAN Speed Record by sending 118Gbps of network throughput.
Internet2 has a Land Speed Record (I2-LSR) competition that takes into consideration distance and the measurements are in Terabit-Meters per second. The current record is held by a team of people from University of Tokyo, the WIDE Project, NTT Communications and others.
Besides using jumbo frames, these teams tune their operating systems on fast processor computers, tune TCP parameters, optimize I/O, and try to keep buffer bloat to a minimum.
Products Supporting Jumbo Frames
Most network devices support a jumbo frame size of 9216 bytes. However, you want to check with your particular manufacturer on the largest frame size their devices support and how to configure the changes. Also, even within a single manufacturer's line of network products, the MTU capabilities may vary greatly so it is important to do a thorough investigation of all your devices in the communication paths and validate their settings.
With the Cisco 6500 switches, the various modules supported different maximum jumbo frame sizes based on the ASIC used. Most new modules support the 9216 MTU size, but some older modules may only support 8092 byte frame size. This can complicate the configuration because as you look at all the devices along the transmission path you need to consider each module's jumbo frame size within the 6500 chassis.
Cisco Nexus 7K/5K switches also support jumbo frames in NX-OS. To increase the MTU size use the "system jumbomtu 9216" global command and use the interface-specific configuration command "mtu 9216" to activate this for the interface. Again, use the "show interface" command to verify the setting.
Cisco 3750/3560 switches have a default MTU size of 1500 bytes, but it can be increased. This is done using the "system mtu 1546" command on 10/100 interfaces and using the "system mtu jumbo 9000" command on GigabitEthernet inrerfaces. The "show system mtu" command can show you how the switch is currently set.
Older Cisco workgroup switches may only be capable of 2000 byte MTU sizes.
Here is a Cisco document titled "Jumbo/Giant Frame Support on Catalyst Switches Configuration Example" that runs down all their switch models.
Meraki (now Cisco) Ethernet switches default to 9600 byte MTU size on their interfaces.
Most Wireless LANs do not support jumbo frames because the WAPs are limited to 1500 bytes and the controllers do not support jumbo frames.
Check with your particular manufacturer to determine if jumbo frames are a configurable option.
Problems with MTU size reduction due to tunnels, IPsec encryption, and overlay protocols can be problematic. If you are using encapsulation technologies then you should consider increasing the MTU size. It would be nice if we were able to increase the MTU size in the core of the network or WAN to avoid the fragmentation and PMTUD issues. Ask your service provider if they support larger frame sizes within their network and on the link between their PE and your CE router. Learning about the benefits of jumbo frames may be beneficial to your network's performance. However, it is important to explore if our network devices support jumbo frames before you turn it on. Some of the biggest gains of using jumbo frames can be realized within and between data centers. However, we should be cognizant of the fragmentation that may occur if those large frames try to cross a link that has a smaller MTU size.