Chapter 7: Understanding CEF in an MPLS VPN Environment

Troubleshooting an MPLS VPN

The previous section outlined the proper operation of an MPLS VPN network and basic verification of the LFIB and VRF CEF tables. However, ambiguity can exist when traffic is not passing properly when it was previously. A PE might not have installed the BGP label into the CEF FIB. If so, clear the VRF route to resolve the issue and look for a software bug using the Cisco Bug Toolkit. You can enable the debug mpls lfib cef command to get more information about LFIB additions and deletions. This command can be useful in troubleshooting label mismatches in FIB and LFIB but can be intensive on production routers. Therefore, use it with care. Table 7-3 lists some key tips and commands when troubleshooting an MPLS VPN problem.

Table 7-3 Typical Problems Seen When Troubleshooting an MPLS VPN

Table 7-4 lists helpful show commands.

Table 7-4 Helpful show Commands When Troubleshooting an MPLS VPN

Table 7-5 lists helpful debug commands.

Table 7-5 Helpful debug Commands When Troubleshooting MPLS VPNs

CEF Considerations When Troubleshooting MPLS VPN Across Various Platforms

When troubleshooting MPLS VPNs on Cisco devices, you need to understand the platform architecture. Different architectures and processors require different commands to troubleshoot. Troubleshooting is different on a software-based platform such as a Cisco 7200 router with an NPE-G2 processor. More troubleshooting steps must be taken on a distributed platform such as the Cisco 7500 to check the line card information. With the Cisco Catalyst 6500, the type of supervisor determines the device usage possible in an MPLS VPN environment and the type of troubleshooting. Hardware-based forwarding on a Cisco 12000 series router varies with line card engine type. With advanced technology, Parallel Express Forwarding (PXF) requires additional troubleshooting on a Cisco 10000 series router. Caveats of these platforms are discussed in the following sections.

Cisco 7200 Router with an NPE-G2

Many customers use the Cisco 7200 with an NPE-G1 or NPE-G2 in MPLS environments. On these platforms serving as a PE device, the same rules previously covered apply. The NPE-G2 has optimized performance but does not do hardware-based switching. So, when troubleshooting an MPLS VPN issue, you have to examine the routing, CEF, and LFIB tables in one place only. This is also true for other software-based platforms such as the Cisco ISR routers.

Cisco 7500 Router

On the other hand, on a distributed platform such as a Cisco 7500 router acting as a PE device, you need to check the routing, CEF, and LFIB tables on the route processor level and on the line card level. On a 7500 with Versatile Interface Processors (VIPs), most of the time forwarding occurs on the VIP. Sometimes inconsistencies exist between the LFIB on the route switch processor (RSP) and on the VIP. This should not be the case if forwarding is happening on the VIP. If the LFIB and other information look fine on the RSP but traffic forwarding issues remain, you might have to look at the LFIB on the VIP. If an inconsistency exists, it most likely is a software bug. After attaching to a VIP, you can run the same commands that are run on an RSP to troubleshoot MPLS issues and check consistency with commands such as show mpls forwarding <prefix> or show ip cef <prefix>. Example 7-6 shows that the labels and next hops are consistent to reach 10.1.1.1/32 on the RSP and VIP.

Example 7-6 Checking Consistency for Prefix 10.1.1.1/32 on RSP and VIP

7513#show mpls forwarding 10.1.1.1
Local Outgoing  Prefix      Bytes tag Outgoing  Next Hop  
tag  tag or VC  or Tunnel Id   switched  interface       
17   Untagged  10.1.1.1/32    0     Et9/1/0  10.1.16.1  
7513#if-con 9
Console or Debug [C]: 
Entering CONSOLE for VIP2 R5K 9
Type "^C^C^C" or "if-quit" to end this session

VIP-Slot9#show mpls forwarding 10.1.1.1
Local Outgoing  Prefix      Bytes tag Outgoing  Next Hop  
tag  tag or VC  or Tunnel Id   switched  interface       
17   Untagged  10.1.1.1/32    0     Et9/1/0  10.1.16.1

In one case, an MPLS inconsistency between the VIPs and RSP is not a bug. By default, slow serial interfaces (less than 2 Mbps) use RSP-based weighted fair queuing (WFQ). If an interface has RSP WFQ enabled, the VIP punts a packet destined for the VRF prefixes out these interfaces to the RSP, and then the RSP performs WFQ and switches the packets. A distributed CEF punt happens and the incoming VIP does not store the incoming label/LFIB information because of RSP-based WFQ. It is desirable for the VIP rather than the RSP to forward the traffic. Therefore, you should enable VIP-based fair queuing instead of RSP-based fair queuing. With VIP-based fair queuing enabled, label switching and WFQ will occur on the VIP.

Cisco Catalyst 6500 with a Supervisor 2

On a Cisco Catalyst 6500 with a Supervisor 2 (SUP2), MPLS VPN troubleshooting is different than on routers because the architecture is different. In general, the ingress and egress interfaces must be FlexWAN or Optical Services Module (OSM) PXF-enabled interfaces to support MPLS VPNs. This is because these cards are MPLS aware and required for imposition and disposition. Because the Policy Feature Card 2 (PFC2) switching ASICs cannot handle MPLS packets, MPLS tables are loaded directly to OSM or FlexWAN cards from the route processor to handle switching.

The general troubleshooting at the software level remains the same as that discussed in the section "Troubleshooting an MPLS VPN," earlier in this chapter. However, on a Cisco Catalyst 6500 platform, the main benefit is hardware switching. This factor adds another twist to troubleshooting MPLS VPNs. The switch does not use the Supervisor 2's FIB Ternary Content Addressable Memory (TCAM) for MPLS functionality. The OSM or FlexWAN stores all the MPLS and VPN forwarding tables. The route processor passes the LFIB table to the Toaster Tag FIB (TTFIB) on each card. The TTFIB is a condensed form of the LFIB for optimized use with the PXF processor and FlexWAN. The OSM and FlexWAN line cards perform the actual lookup and rewrite for MPLS packets in hardware using the TTFIB.

For a Cisco Catalyst 6500SUP2 serving as an MPLS PE device, the ingress card determines the VPN that the IP packet belongs to and performs a lookup in the TTFIB to determine the egress interface. The ingress index directs the IP packet to the egress port, bypassing the PFC2. The source MAC address is encoded with a TTFIB index so that the egress is able to determine the proper lookup and labels. The egress adds the two labels—one VPN and one IGP label—and sends the packet out.

In the reverse direction of label disposition, an MPLS packet arrives and the SUP2 uses the label to determine the VPN owner for the packet. The ingress index directs the packet to the egress interface with special information. The egress pops the label, does a TTFIB lookup, and forwards the IP packet.

When a Cisco Catalyst 6500 SUP2 device serves as a P device, you must have a FlexWAN or OSM as the ingress and egress card. The ingress card does a lookup in the TTFIB based on the label to determine the egress interface. The device forwards the packet directly from the ingress to the egress. The packet arrives at the egress module and it does another TTFIB lookup. Then the egress performs the appropriate label swap.

When troubleshooting MPLS VPNs on a SUP2 platform with OSMs or FlexWANs, examination must occur on the hardware tables of these cards. Therefore, in troubleshooting, you must compare the output of the MPLS forwarding table on the RP with the TTFIB on the ingress line card. On the line card, check the FIB, LFIB, and hardware CEF entries.

Catalyst 6500 with a Supervisor 720 3BXL

On a Supervisor 720 3BXL, the EARL 7/PFC3 switching ASICs can handle MPLS packets and there is no longer a requirement to have an OSM or FlexWAN card as the ingress/egress for imposition and disposition. The Layer 3 ASIC on the EARL 7 populates the FIB TCAM with MPLS entries. It also handles label imposition, swapping, and disposition. With a SUP720, any line card is capable of being an MPLS link. The PFC3/DFC3 perform the MPLS VPN forwarding decision for OSM PXF-enabled interfaces and FlexWAN interfaces unlike in a PFC2 system, where the OSM PXF interfaces and FlexWAN perform the forwarding decision. The RP on the SUP720 builds the VRF entries. It sends information to the SP to program the TCAM. The FIB TCAM stores a VPN table ID along with the label and prefix entry. Table 7-6 shows helpful commands to use when troubleshooting a P device that has an SUP720 in an MPLS environment.

Table 7-6 Helpful P Troubleshooting Commands on Native SUP720s

Table 7-7 shows helpful commands to use when troubleshooting an MPLS VPN PE device that is a Cisco Catalyst 6500 with a SUP720.

Table 7-7 Helpful PE Troubleshooting Commands on Native SUP720s

Figure 7-4 shows a Cisco Catalyst 6500 with a SUP720 used as an MPLS VPN PE device.

Figure 7-4

SUP720 Used as an MPLS VPN PE

Example 7-7 includes show commands for checking software data for VRF prefix 10.1.1.0/24 on a SUP720 serving as a PE device in an MPLS VPN network. In this example, the VRF name is "red."

Example 7-7 Checking Software Information for Prefix 10.1.1.0/24 in VRF Red

6500PE#show ip route vrf red 10.1.1.0
Routing entry for 10.1.1.0/24
 Known via "bgp 65001", distance 200, metric 0, type internal
 Last update from 192.168.1.1 00:05:07 ago
 Routing Descriptor Blocks:
 * 192.168.1.1 (Default-IP-Routing-Table), from 192.168.1.1, 00:05:07 ago
   Route metric is 0, traffic share count is 1
   AS Hops 0
6500PE#show ip cef vrf red 10.1.1.0 detail
10.1.1.0/24, version 33, epoch 0, cached adjacency 172.16.1.1
0 packets, 0 bytes
 tag information set, all rewrites owned
  local tag: VPN-route-head
  fast tag rewrite with Gi1/2, 172.16.1.1, tags imposed: {17 22}
 via 192.168.1.1, 0 dependencies, recursive
  next hop 172.16.1.1, GigabitEthernet1/2 via 192.168.1.1/32 (Default)
  valid cached adjacency
  tag rewrite with Gi1/2, 172.16.1.1, tags imposed: {17 22}
6500PE#show mpls forwarding vrf red 10.1.1.0 detail
Local Outgoing  Prefix       Bytes tag Outgoing  Next Hop  
tag  tag or VC  or Tunnel Id    switched  interface       
None  17     10.1.1.0/24    0     Gi1/2   172.16.1.1  
    MAC/Encaps=14/22, MRU=1496, Tag Stack{17 22}
    000F35E4541900E0142B63008847 0001100000016000
    No output feature configured
  Per-packet load-sharing

Example 7-8 checks the hardware information on the SUP720 for the prefix 10.1.1.0/24 in VRF red. To view the hardware adjacency entry, you must identify the index from the first command's "A:" value.

Example 7-8 Checking Hardware Information for Prefix 10.1.1.0/24 in VRF Red

6500PE#show mls cef vrf red 10.1.1.0 24 detail

Codes: M - mask entry, V - value entry, A - adjacency index, P - priority bit
    D - full don't switch, m - load balancing modnumber, B - BGP Bucket sel
    V0 - Vlan 0,C0 - don't comp bit 0,V1 - Vlan 1,C1 - don't comp bit 1
    RVTEN - RPF Vlan table enable, RVTSEL - RPF Vlan table select
Format: IPV4_DA - (8 | xtag vpn pi cr recirc tos prefix)
Format: IPV4_SA - (9 | xtag vpn pi cr recirc prefix)
M(3211  ): E | 1 FFF 0 0 0 0  255.255.255.0
V(3211  ): 8 | 1 256 0 0 0 0  10.1.1.0      (A:294912 ,P:1,D:0,m:0 ,B:0 
)
6500PE#show mls cef adjacency entry 294912 detail

Index: 294912 smac: 00e0.142b.6300, dmac: 000f.35e4.5419
        mtu: 1518, vlan: 1020, dindex: 0x0, l3rw_vld: 1
        format: MPLS, flags: 0x8418 
        label0: 0, exp: 0, ovr: 0
        label1: 22, exp: 0, ovr: 0
        label2: 17, exp: 0, ovr: 0
        op: PUSH_LABEL2_LABEL1
        packets: 0, bytes: 0