Troubleshooting an MPLS VPN
The previous section outlined the proper operation of an MPLS VPN network and basic verification of the LFIB and VRF CEF tables. However, ambiguity can exist when traffic is not passing properly when it was previously. A PE might not have installed the BGP label into the CEF FIB. If so, clear the VRF route to resolve the issue and look for a software bug using the Cisco Bug Toolkit. You can enable the debug mpls lfib cef command to get more information about LFIB additions and deletions. This command can be useful in troubleshooting label mismatches in FIB and LFIB but can be intensive on production routers. Therefore, use it with care. Table 7-3 lists some key tips and commands when troubleshooting an MPLS VPN problem.
Table 7-3 Typical Problems Seen When Troubleshooting an MPLS VPN
Symptom | Tip | Command |
---|---|---|
VPNv4 traffic is not getting forwarded end to end | Check the label information in BGP and LFIB at the advertising PE router. | show ip bgp vpn vrf <vrf> label | include <prefix> show mpls forwarding vrf <vrf> | include <prefix> |
Check the label information in BGP and FIB at the receiving PE router. | show ip bgp vpn vrf <vrf> label | include <prefix> show ip cef vrf <vrf> <prefix> | |
Incoming MPLS traffic is dropped at the egress PE | Check the FIB and LFIB entries on the route processor (RP), line cards (LCs), and relevant hardware engines if present. | show mpls forwarding vrf <prefix> |
Table 7-4 lists helpful show commands.
Table 7-4 Helpful show Commands When Troubleshooting an MPLS VPN
show Command | Purpose |
---|---|
show ip route vrf <vrf> | Shows the route entries relevant to the specified VRF only |
show ip arp vrf <vrf> | Shows Address Resolution Protocol (ARP) entries relevant to the specified VRF only |
show ip cef vrf <vrf> <prefix> | Shows the label stack and outgoing interface for the particular prefix |
show mpls forwarding vrf <vrf> <prefix> | Shows the LFIB lookup based on a VPN prefix |
show mpls forwarding | include <prefix> | Shows whether a prefix is in the LFIB |
show mpls forwarding label <label> | Shows the LFIB lookup based on the incoming label |
show ip cef vrf <name> exact-route <source address> <destination address> | Shows the path of traffic from the PE's perspective for a VPN prefix |
Table 7-5 lists helpful debug commands.
Table 7-5 Helpful debug Commands When Troubleshooting MPLS VPNs
debug Command | Purpose |
---|---|
debug ip bgp vpnv4 | Useful when troubleshooting label-related problems in BGP |
debug mpls lfib cef [acl] | Useful when troubleshooting label mismatch among BGP, FIB, and LFIB for VPN prefixes |
debug ip routing vrf <vrf> [acl] | Useful when router does not install VPN prefixes in the VRF routing table |
CEF Considerations When Troubleshooting MPLS VPN Across Various Platforms
When troubleshooting MPLS VPNs on Cisco devices, you need to understand the platform architecture. Different architectures and processors require different commands to troubleshoot. Troubleshooting is different on a software-based platform such as a Cisco 7200 router with an NPE-G2 processor. More troubleshooting steps must be taken on a distributed platform such as the Cisco 7500 to check the line card information. With the Cisco Catalyst 6500, the type of supervisor determines the device usage possible in an MPLS VPN environment and the type of troubleshooting. Hardware-based forwarding on a Cisco 12000 series router varies with line card engine type. With advanced technology, Parallel Express Forwarding (PXF) requires additional troubleshooting on a Cisco 10000 series router. Caveats of these platforms are discussed in the following sections.
Cisco 7200 Router with an NPE-G2
Many customers use the Cisco 7200 with an NPE-G1 or NPE-G2 in MPLS environments. On these platforms serving as a PE device, the same rules previously covered apply. The NPE-G2 has optimized performance but does not do hardware-based switching. So, when troubleshooting an MPLS VPN issue, you have to examine the routing, CEF, and LFIB tables in one place only. This is also true for other software-based platforms such as the Cisco ISR routers.
Cisco 7500 Router
On the other hand, on a distributed platform such as a Cisco 7500 router acting as a PE device, you need to check the routing, CEF, and LFIB tables on the route processor level and on the line card level. On a 7500 with Versatile Interface Processors (VIPs), most of the time forwarding occurs on the VIP. Sometimes inconsistencies exist between the LFIB on the route switch processor (RSP) and on the VIP. This should not be the case if forwarding is happening on the VIP. If the LFIB and other information look fine on the RSP but traffic forwarding issues remain, you might have to look at the LFIB on the VIP. If an inconsistency exists, it most likely is a software bug. After attaching to a VIP, you can run the same commands that are run on an RSP to troubleshoot MPLS issues and check consistency with commands such as show mpls forwarding <prefix> or show ip cef <prefix>. Example 7-6 shows that the labels and next hops are consistent to reach 10.1.1.1/32 on the RSP and VIP.
Example 7-6 Checking Consistency for Prefix 10.1.1.1/32 on RSP and VIP
7513#show mpls forwarding 10.1.1.1 Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 17 Untagged 10.1.1.1/32 0 Et9/1/0 10.1.16.1 7513#if-con 9 Console or Debug [C]: Entering CONSOLE for VIP2 R5K 9 Type "^C^C^C" or "if-quit" to end this session VIP-Slot9#show mpls forwarding 10.1.1.1 Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 17 Untagged 10.1.1.1/32 0 Et9/1/0 10.1.16.1
In one case, an MPLS inconsistency between the VIPs and RSP is not a bug. By default, slow serial interfaces (less than 2 Mbps) use RSP-based weighted fair queuing (WFQ). If an interface has RSP WFQ enabled, the VIP punts a packet destined for the VRF prefixes out these interfaces to the RSP, and then the RSP performs WFQ and switches the packets. A distributed CEF punt happens and the incoming VIP does not store the incoming label/LFIB information because of RSP-based WFQ. It is desirable for the VIP rather than the RSP to forward the traffic. Therefore, you should enable VIP-based fair queuing instead of RSP-based fair queuing. With VIP-based fair queuing enabled, label switching and WFQ will occur on the VIP.
Cisco Catalyst 6500 with a Supervisor 2
On a Cisco Catalyst 6500 with a Supervisor 2 (SUP2), MPLS VPN troubleshooting is different than on routers because the architecture is different. In general, the ingress and egress interfaces must be FlexWAN or Optical Services Module (OSM) PXF-enabled interfaces to support MPLS VPNs. This is because these cards are MPLS aware and required for imposition and disposition. Because the Policy Feature Card 2 (PFC2) switching ASICs cannot handle MPLS packets, MPLS tables are loaded directly to OSM or FlexWAN cards from the route processor to handle switching.
The general troubleshooting at the software level remains the same as that discussed in the section "Troubleshooting an MPLS VPN," earlier in this chapter. However, on a Cisco Catalyst 6500 platform, the main benefit is hardware switching. This factor adds another twist to troubleshooting MPLS VPNs. The switch does not use the Supervisor 2's FIB Ternary Content Addressable Memory (TCAM) for MPLS functionality. The OSM or FlexWAN stores all the MPLS and VPN forwarding tables. The route processor passes the LFIB table to the Toaster Tag FIB (TTFIB) on each card. The TTFIB is a condensed form of the LFIB for optimized use with the PXF processor and FlexWAN. The OSM and FlexWAN line cards perform the actual lookup and rewrite for MPLS packets in hardware using the TTFIB.
For a Cisco Catalyst 6500SUP2 serving as an MPLS PE device, the ingress card determines the VPN that the IP packet belongs to and performs a lookup in the TTFIB to determine the egress interface. The ingress index directs the IP packet to the egress port, bypassing the PFC2. The source MAC address is encoded with a TTFIB index so that the egress is able to determine the proper lookup and labels. The egress adds the two labels—one VPN and one IGP label—and sends the packet out.
In the reverse direction of label disposition, an MPLS packet arrives and the SUP2 uses the label to determine the VPN owner for the packet. The ingress index directs the packet to the egress interface with special information. The egress pops the label, does a TTFIB lookup, and forwards the IP packet.
When a Cisco Catalyst 6500 SUP2 device serves as a P device, you must have a FlexWAN or OSM as the ingress and egress card. The ingress card does a lookup in the TTFIB based on the label to determine the egress interface. The device forwards the packet directly from the ingress to the egress. The packet arrives at the egress module and it does another TTFIB lookup. Then the egress performs the appropriate label swap.
When troubleshooting MPLS VPNs on a SUP2 platform with OSMs or FlexWANs, examination must occur on the hardware tables of these cards. Therefore, in troubleshooting, you must compare the output of the MPLS forwarding table on the RP with the TTFIB on the ingress line card. On the line card, check the FIB, LFIB, and hardware CEF entries.
Catalyst 6500 with a Supervisor 720 3BXL
On a Supervisor 720 3BXL, the EARL 7/PFC3 switching ASICs can handle MPLS packets and there is no longer a requirement to have an OSM or FlexWAN card as the ingress/egress for imposition and disposition. The Layer 3 ASIC on the EARL 7 populates the FIB TCAM with MPLS entries. It also handles label imposition, swapping, and disposition. With a SUP720, any line card is capable of being an MPLS link. The PFC3/DFC3 perform the MPLS VPN forwarding decision for OSM PXF-enabled interfaces and FlexWAN interfaces unlike in a PFC2 system, where the OSM PXF interfaces and FlexWAN perform the forwarding decision. The RP on the SUP720 builds the VRF entries. It sends information to the SP to program the TCAM. The FIB TCAM stores a VPN table ID along with the label and prefix entry. Table 7-6 shows helpful commands to use when troubleshooting a P device that has an SUP720 in an MPLS environment.
Table 7-6 Helpful P Troubleshooting Commands on Native SUP720s
show Command | Function |
---|---|
show mpls forwarding <prefix> detail | Shows software information |
show mls cef <prefix> detail | Shows hardware forwarding information |
show mls cef adj entry <index> detail | Shows hardware adjacency |
Table 7-7 shows helpful commands to use when troubleshooting an MPLS VPN PE device that is a Cisco Catalyst 6500 with a SUP720.
Table 7-7 Helpful PE Troubleshooting Commands on Native SUP720s
show Command | Function |
---|---|
show ip route vrf <name> <prefix> | Verifies routing entry |
show ip cef vrf <name> <prefix> detail | Verifies CEF entry |
show mpls forwarding vrf <name> <prefix> detail | Shows software forwarding information |
show mls cef vrf <name> <prefix> <mask length> detail | Shows hardware forwarding information |
show mls cef adjacency entry <index> detail | Shows hardware adjacency information |
show vlan internal usage | Shows internal VLAN mapping and outgoing interface |
show mls cef mpls labels <label> detail | Shows hardware label information |
Figure 7-4 shows a Cisco Catalyst 6500 with a SUP720 used as an MPLS VPN PE device.
SUP720 Used as an MPLS VPN PE
Example 7-7 includes show commands for checking software data for VRF prefix 10.1.1.0/24 on a SUP720 serving as a PE device in an MPLS VPN network. In this example, the VRF name is "red."
Example 7-7 Checking Software Information for Prefix 10.1.1.0/24 in VRF Red
6500PE#show ip route vrf red 10.1.1.0 Routing entry for 10.1.1.0/24 Known via "bgp 65001", distance 200, metric 0, type internal Last update from 192.168.1.1 00:05:07 ago Routing Descriptor Blocks: * 192.168.1.1 (Default-IP-Routing-Table), from 192.168.1.1, 00:05:07 ago Route metric is 0, traffic share count is 1 AS Hops 0 6500PE#show ip cef vrf red 10.1.1.0 detail 10.1.1.0/24, version 33, epoch 0, cached adjacency 172.16.1.1 0 packets, 0 bytes tag information set, all rewrites owned local tag: VPN-route-head fast tag rewrite with Gi1/2, 172.16.1.1, tags imposed: {17 22} via 192.168.1.1, 0 dependencies, recursive next hop 172.16.1.1, GigabitEthernet1/2 via 192.168.1.1/32 (Default) valid cached adjacency tag rewrite with Gi1/2, 172.16.1.1, tags imposed: {17 22} 6500PE#show mpls forwarding vrf red 10.1.1.0 detail Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface None 17 10.1.1.0/24 0 Gi1/2 172.16.1.1 MAC/Encaps=14/22, MRU=1496, Tag Stack{17 22} 000F35E4541900E0142B63008847 0001100000016000 No output feature configured Per-packet load-sharing
Example 7-8 checks the hardware information on the SUP720 for the prefix 10.1.1.0/24 in VRF red. To view the hardware adjacency entry, you must identify the index from the first command's "A:" value.
Example 7-8 Checking Hardware Information for Prefix 10.1.1.0/24 in VRF Red
6500PE#show mls cef vrf red 10.1.1.0 24 detail Codes: M - mask entry, V - value entry, A - adjacency index, P - priority bit D - full don't switch, m - load balancing modnumber, B - BGP Bucket sel V0 - Vlan 0,C0 - don't comp bit 0,V1 - Vlan 1,C1 - don't comp bit 1 RVTEN - RPF Vlan table enable, RVTSEL - RPF Vlan table select Format: IPV4_DA - (8 | xtag vpn pi cr recirc tos prefix) Format: IPV4_SA - (9 | xtag vpn pi cr recirc prefix) M(3211 ): E | 1 FFF 0 0 0 0 255.255.255.0 V(3211 ): 8 | 1 256 0 0 0 0 10.1.1.0 (A:294912 ,P:1,D:0,m:0 ,B:0 ) 6500PE#show mls cef adjacency entry 294912 detail Index: 294912 smac: 00e0.142b.6300, dmac: 000f.35e4.5419 mtu: 1518, vlan: 1020, dindex: 0x0, l3rw_vld: 1 format: MPLS, flags: 0x8418 label0: 0, exp: 0, ovr: 0 label1: 22, exp: 0, ovr: 0 label2: 17, exp: 0, ovr: 0 op: PUSH_LABEL2_LABEL1 packets: 0, bytes: 0