The maximum transmission unit (MTU) is the largest number of bytes an individual datagram can have without either being fragmented into smaller datagrams or being dropped along the path between its source and its destination.\nFor Ethernet frames\u2014and many other types of packets\u2014that number is 1500 bytes, and it generally meets the requirements of traffic that can cross the public internet intact.\n\nSo, if 2000-byte Ethernet packets arrive at a router, it will split their payloads in two and repackage them into two packets that are each smaller than 1500 bytes and so meet the MTU.\nAn alternative is that the router drops the packet but sends the source device an internet control-message protocol (ICMP) packet-too-big message. The intent is for the source device resend the payload in smaller packets, but it might not be configured to support this.\nMTU size also comes in to play when, for a frame to get from its source to its destination, it may have to cross a network that use a different protocol than that used by the source and destination networks. For instance, a device on an Ethernet LAN might want to send a payload to a device on an Ethernet LAN in another city and have to cross an MPLS connection on the way.\nIn that case the size of the Ethernet frames must be taken into consideration. If encapsulation of Ethernet in MPLS pushes the size of the MPLS frame past the MTU of the MPLS edge switches, the switches will drop it.\nMTU size\nThe size of an MTU is governed by the physical properties of the communications media. Historically, network media were slower and more prone to error, so MTU sizes were set to be relatively small. For most Ethernet networks this is 1500 bytes, and this size is used almost universally on access networks. Ethernet II networks have a standard\u00a0frame size\u00a0of 1518 bytes, which includes a 14-byte Ethernet II header and a four-byte frame-check sequence (FCS). Other communications media have different MTU sizes.\nEncapsulation overhead\nWhen one protocol's packets or frames are encapsulated within another protocol, it increases the overall frame size. Encapsulation adds a protocol header, so any packets that are created at 1500 bytes and are then encapsulated will exceed MTU the network can handle. The number of bytes encapsulation adds varies by type of protocol:\n\nGRE (IP Protocol 47) (RFC 2784) adds 24 bytes (20 byte IPv4 header, 4 byte GRE header)\n6in4 encapsulation (IP Protocol 41,\u00a0RFC 4213) adds 20 bytes\n4in6 encapsulation (e.g. DS-Lite\u00a0RFC 6333) adds 40 bytes\nAny time you add another outer IPv4 header adds 20 bytes\nIPsec encryption performed by the DMVPN adds 73 bytes for ESP-AES-256 and ESP-SHA-HMAC overhead (overhead depends on transport or tunnel mode and the encryption\/authentication algorithm and HMAC)\nMPLS\u00a0adds 4 bytes for each label in the stack\nIEEE\u00a0802.1Q\u00a0tag adds 4 bytes (Q-in-Q would add 8 bytes)\nVXLAN\u00a0adds 50 bytes\nOTV\u00a0adds 42 bytes\nLISP\u00a0adds 36 bytes for IPv4 and 56 bytes for IPv6 encapsulation\nNVGRE\u00a0adds 42 bytes\nSTT\u00a0adds 54 bytes\n\nThere are many other situations where protocol encapsulation occurs, so you must be aware when this happens and take steps to accommodate it. A packet may originate as a standard IPv4 packet with a designated MTU of 1500 bytes, but depending on its destination it may pass through encapsulation that pushes its size over the MTU.\nPath MTU Discovery (PMTUD)\nRouters can fragment packets to cut them down to fit smaller MTUs, but this is not optimal. A packet incoming to a network device may be smaller than the MTU, but if it gets encapsulated by the device and the new total packet size exceeds the MTU of the outgoing interface, the device may fragment the packet into two smaller packets before forwarding the data.\nFor example, an IPv4 router will fragment and forward packets that exceed the MTU, but also send back an ICMP message-too-big error message to tell the source device that it should use a smaller MTU. On the other hand, IPv6 routers do not fragment oversized packets on behalf of the source; they just drop them and send back an ICMPv6 packet-too-big error message.\nThe main problem with MTU size being reduced across the network is that some applications may not work well in this environment.\nTo complicate matters, some routers ignore packet-too-big messages and keep sending packets that exceed the MTU. They are not following a standardized technique called path MTU discovery that can avoid fragmentation across a network.\nSome nodes that send 1500-byte packets into the DMVPN and subsequently receive an ICMPv4 packet-too-big message from the router may choose to ignore this. These nodes are not performing\u00a0Path MTU Discovery\u00a0(PMTUD) as prescribed by IETF RFC 1191\u00a0or\u00a0RFC 1981,\u00a0and are therefore relying on the IPv4 routers to perform this fragmentation on behalf of the source host.\u00a0RFC 2923\u00a0also covers the topic of \u201cTCP Problems with Path MTU Discovery.\u201d If the application cannot function properly in this environment, there could be end-user impacts. Also, if there is a firewall in the middle of the communication path somewhere that is blocking the ICMP error messages, then that would definitely prevent PMTUD from operating properly.\nOne method to test and detect a reduced MTU size is to use a ping with a large packet size. Here are some examples of how to do this.\nC:UsersScottHogg> ping -l 1500 192.168.10.1\nOn a Windows host you can also set the Do Not Fragment (DF) bit to 1 with the -f ping parameter.\nC:UsersScottHogg> ping 192.168.10.1 -l 1500 \u2013f\nOn Linux the command would be:\nRedHat# ping -s 1500 -M do 192.168.10.1\nOn a Cisco IOS device the command would be:\nRouter1# ping 192.168.10.1 size 1500 df-bit\nOn a Cisco NX-OS device the command would be:\nSwitch7K# ping 192.168.10.1 packet-size 9216 c 10\nOn a Cisco IOS XR device the command would be:\nRP\/0\/RP0\/CPU0:Router1#ping 192.168.10.1 size 1500 donnotfrag\nOn a JUNOS device the command would look like:\nroot@J4350-1# run ping 192.168.10.1 size 1500 do-not-fragment rapid\nFragmentation\nIPv4 routers fragment on behalf of the source node that is sending an oversized packet. Routers can fragment IPv4 packets unless the Do-Not-Fragment (DF) bit is set to 1 in the IPv4 header. If the DF bit is set to 0 (the default), the router splits a packet that is too large to fit into the outgoing interface and sends two packets toward the destination. When the destination receives the two fragments, the destination's protocol stack must reassemble the fragments before processing the protocol data unit (PDU). But there's a danger when an application sends its packets with DF set to 1, does not pay attention to the ICMP \u201cpacket too big\u201d messages, and does not perform PMTUD.\nAll IPv6 networks must support an MTU size of 1,280 bytes or greater (RFC 2460). This is because IPv6 routers do not fragment IPv6 packets on behalf of the source. IPv6 routers drop the packet and send back an ICMPv6 Type 4 packet (size exceeded) to the source indicating the proper MTU size. It then falls on the shoulders of the source to perform the fragmentation itself and cache the new reduced MTU size for that destination so future packets use the correct MTU size.\nWhen routers perform fragmentation on behalf of the source, that adds CPU processing overhead on the router. If IPsec is being used, then the routers on both ends of the tunnel will need to handle the fragmentation and reassembly of the packets. If the routers are performing fragmentation on behalf of the source node, it may be desirable to have the fragmentation performed prior to encryption, so the destination tunnel router doesn't have to reassemble the fragments and then perform the decryption.\nThe following two Cisco IOS global configuration commands can control this behavior.\nRouter(config-if)# crypto ipsec fragmentation before-encryption\nRouter(config-if)# crypto ipsec fragmentation after-encryption\nThere is a good document from Cisco on the 7600 switches and how to resolve these issues, entitled \u201cConfiguring IPSec VPN Fragmentation and MTU\u201d.\nMTU and MSS\nAnother method to handle the increase in MTU size due to encapsulation and the resulting fragmentation is to utilize the\u00a0TCP Maximum Segment Size\u00a0(MSS) parameter. The MSS is the largest number of bytes of payload that can be sent in a single TCP packet. In other words, the MSS is the largest amount of TCP data (in bytes) that can be transported over a computer network. This is negotiated during the TCP 3-way handshake in the SYN packet. The MSS is defined in\u00a0RFC 879 for IPv4 and in\u00a0RFC 2460\u00a0for IPv6. The MSS does not include the TCP header (20 bytes) or the IPv4 header (20 bytes; IPv6 header is 40 bytes).\nWhen IPsec is being used, it is customary to set the MTU size on the tunnel interfaces to 1,400 bytes and to set the TCP-MSS-adjust to 1,360 bytes. This can be configured in a Cisco IOS device using these commands.\nRouter(config)# interface tunnel 4\nRouter(config-if)# ip tcp adjust-mss 1360\nRouter(config-if)# ip mtu 1400\nFor IPv6-enabled interfaces we can use the same type of functions, but the IPv6 header is 40 bytes instead of IPv4\u2019s ~20-byte header. We must also consider the 20-byte TCP header, which is the same size for IPv4 and IPv6.\nRouter(config)# interface tunnel 6\nRouter(config-if)# ipv6 tcp adjust-mss 1340\nRouter(config-if)# ipv6 mtu 1400\nThis MSS option does not work for UDP applications: UDP is a connectionless protocol, so there's no way to negotiate this during the handshake. For UDP applications that do not perform PMTUD and set the DF bit to 1, one option may be to configure a policy that sets the DF bit back to zero.\nFor more on this topic, read \u201cResolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC\u201d from Cisco.\nCompensate by increasing the MTU size\nAs we've seen, the primary issue with MTU size arises when encapsulation takes place while the links between sites only support a 1,500-byte MTU. This is frequently the case for links between enterprise routers and the upstream ISP routers, or between CE routers and PE routers.\nIt would be highly desirable to be able to increase the MTU size over the WAN. If the MTU size could be increased throughout the path across the WAN, then the added encapsulation overhead could be compensated for by the WAN interface of the routers. This would eliminate the need to reduce the MTU size on the tunnel interfaces, adjust MSS, and alleviate the routers from performing any fragmentation. That's where jumbo frames come in\nJumbo frames\nJumbo frames\u00a0are network-layer PDUs that have a size much larger than the typical 1,500 byte Ethernet MTU. In some situations, jumbo frames can be used to allow for much larger frame sizes if the networking hardware is capable of this configuration. Most modern routers and switches, as well as most datacenter networking hardware, can support jumbo frames.\nLarger frames can also boost speed. With larger frame sizes \u2014 and thus larger payload sizes \u2014 you can have less protocol overhead and are able to achieve\u00a0higher protocol efficiency. In other words, your "goodput" improves with larger frame sizes. You can also reduce network bandwidth and CPU cycles on network hardware.\nTo configure the jumbo frame MTU size on a Cisco IOS device, just enter the MTU command on the interface configuration like this:\nRouter(config)# interface GigabitEthernet 4\/1\nRouter(config-if)# mtu 9216\nThe show interface command will verify the interface's new MTU size.\nFor other manufacturers' equipment, you just have to look for a configuration command within the physical or virtual interface that allows you to set the MTU size greater than 1,500 bytes.\nThe key concept to keep in mind is that all the network devices along the communication path must support jumbo frames. Jumbo frames need to be configured to work on the ingress and egress interface of each device along the end-to-end transmission path. Furthermore, all devices in the topology must also agree on the maximum jumbo frame size. If there are devices along the transmission path that have varying frame sizes, then you can end up with fragmentation problems. Also, if a device along the path does not support jumbo frames and it receives one, it will drop it.\nJumbograms\nJumbo frames should not be confused with\u00a0jumbograms. When discussing communications protocols, frames are the PDU used at Layer 2 (the data link layer) of the OSI model, packets are the PDU used at Layer 3 (the network layer). A jumbogram is a larger Layer 3 packet that exceeds the link MTU size. IPv4 is capable of generating payloads up to 65,535 bytes, while IPv6 is capable of a 32-bit "Jumbo Payload Length" size within a hop-by-hop option header. Therefore, IPv6 could support a ridiculous 4.2GB payload. Clearly, that packet could not be transported on any type of common networking interface \u2014 just imagine the repercussions of a retransmission.\nJumbo frame support\nMost network devices support a jumbo frame size of 9,216 bytes. This isn't standardized like Ethernet's 1,500 byte MTU, though, so you want to check with your particular manufacturer on the largest frame size their devices support and how to configure the changes. Even within a single manufacturer's line of network products, the MTU capabilities may vary greatly, so it is important to do a thorough investigation of all your devices in the communication paths and validate their settings. For instance, some Intel Gigabit adapters support jumbo frames but many do not.\nRecommendations\nProblems with MTU size reduction due to tunnels, IPsec encryption, and overlay protocols can degrade network performance. If you are using encapsulation technologies, then you should consider increasing the MTU size, particularly in the core of the network or WAN to avoid fragmentation and PMTUD issues. Ask your service provider if they support larger frame sizes within their network and on the link between their PE and your CE router.\nLearning about the benefits of jumbo frames may be beneficial to your network's performance. However, it is important to explore if and how your network devices support jumbo frames before you turn this feature on. Some of the biggest gains of using jumbo frames can be realized within and between data centers. But you should be cognizant of the fragmentation that may occur if those large frames try to cross a link that has a smaller MTU size.