In the hardware adjacency, label2 has a value of 17 and label1 has a value of 22. Traffic destined for 10.1.1.0 in VRF red will have both labels attached. This matches the data for CEF and the LFIB seen in Example 7-7. Looking at the internal VLAN usage for the device can give the outbound interface mapping. In this case, VLAN 1020 maps to interface GigabitEthernet1/2, as seen in Example 7-9.
Example 7-9 Verifying Internal VLAN Mapping and Outbound Interface
6500PE#show vlan internal usage VLAN Usage ---- -------------------- 1006 online diag vlan0 1007 online diag vlan1 1008 online diag vlan2 1009 online diag vlan3 1010 online diag vlan4 1011 online diag vlan5 1012 PM vlan process (trunk tagging) 1013 Control Plane Protection 1014 L3 multicast partial shortcuts for VPN 0 1015 Egress internal vlan 1016 Multicast VPN 0 QOS vlan 1017 IPv6 Multicast Egress multicast 1018 vrf_0_vlan0 1019 GigabitEthernet2/48 1020 GigabitEthernet1/2
When troubleshooting a Cisco Catalyst 6500 platform, you should understand proper operations, such as the path of the packet and the expected label value. You should find consistency among the routing table, MPLS forwarding table, and the LFIB.
Cisco 12000 Series Router
On a Cisco 12000, forwarding occurs through distributed CEF switching. Some of the line cards are also capable of hardware-based forwarding, unlike a VIP. In this case, not only should you examine the route processor's and line cards' software FIB and LFIB tables, but you must also examine the hardware table on the engine cards because that is where forwarding occurs. The commands used depend on the type of engine cards and the router's function.
Cisco 12000 line cards, which have an Engine 0 or an Engine 1, use software forwarding on the line card for MPLS packets. Engine 2 line cards have a packet-switching ASIC (PSA) that forwards MPLS packets in hardware. Engine 3 line cards use the alpha ASIC for packet forwarding in hardware. Engine 4 line cards use the gen6 ASIC and Engine 4+ cards use the gen7 ASIC. To troubleshoot MPLS or IP forwarding early in product life, you had to execute ASIC-based commands on the respective line cards. Because the ASICs used across various engines differed, it became confusing when troubleshooting. Hence, Cisco implemented standard commands for troubleshooting, as listed in Table 7-8.
Table 7-8 Old Versus Standard Commands
| Engine | Old ASIC-Based or Tag Command | Standardized Command |
|---|---|---|
| 2 | show ip psa-cef show tag psa-tag show tag hardware-tag show ip psa-cef vrf | show ip hardware-cef show mpls hardware-lfib show ip hardware-cef vrf |
| 3 | show ip alpha-cef show tag alpha-tag show tag hardware-tag | show ip hardware-cef show mpls hardware-lfib |
| 4 | show ip gen6-cef | show ip hardware-cef |
| 4+ | show ip gen7-cef show tag hardware-tag | show ip hardware-cef show ip hardware-lfib |
Table 7-9 shows the type of forwarding on each line card engine and some helpful troubleshooting commands on a PE router in an MPLS VPN environment.
Table 7-9 Helpful Commands to Troubleshoot MPLS VPNs on a PE
| Engine Type | Forwarding Type | Helpful Commands on Ingress Line Card |
|---|---|---|
| 0 | Software | show ip cef vrf <name> <prefix> |
| Engine 1 | Software | Not applicable. Engine 1 does not have PE functionality in an MPLS VPN environment. |
| Engine 2 | Hardware | show ip hardware-cef vrf <name> <destination prefix> <mask> detail show ip hardware-cef vrf <name> exact-route <source> <destination> |
| Engine 3 | Hardware | show ip hardware-cef tofab vrf <name> <prefix> detail show ip hardware-cef tofab vrf <name> exact-route <source> <destination> |
| Engine 4 | Hardware | Not applicable. Engine 4 does not have PE functionality in an MPLS VPN environment. Therefore, no VRF commands are available for hardware. The router prints an error message when the user configures ip vrf forwarding on a port on an Engine 4 card. |
| Engine 4+ | Hardware | show ip gen7-cef vrf <name> <prefix> show ip gen7-cef vrf <name> exact-route <source> <destination> Note: At the time of this writing, the show ip hardware-cef command does not include VRF commands. Therefore, you must use the show ip gen7-cef command for troubleshooting. |
Example 7-10 shows the output of troubleshooting VRFs on an Engine 3 line card. A key component is the output interface.
Example 7-10 Engine 3 VRF Verification
gsr#execute-on slot 3 show ip hardware-cef tofab vrf test 10.99.99.99 detail ========= Line Card (Slot 3) ========= Root: 0x2419C000 Location: 0x2419C18C Data: 0x000717E0 Node: 0x2438BF00 Location: 0x2438BF60 Data: 0x00000000 0x000717C0 Node: 0x2438BE00 Location: 0x2438BE30 Data: 0x00000000 0x000717A0 Node: 0x2438BD00 Location: 0x2438BD60 Data: 0x00000000 0x00071780 Node: 0x2438BC00 Location: 0x2438BC30 Data: 0x00000000 0x00071760 Node: 0x2438BB00 Location: 0x2438BB60 Data: 0x00000000 0x00071740 Node: 0x2438BA00 Location: 0x2438BA30 Data: 0xA0000000 0x0400F0FC Node for 10.99.99.99: 0x2438BA30 Leaf FCR 4 0x2438BA30 found 7 deep Fast Adjacency: alpha adjacency: 0x201E1F80 [0-7] ui 16 ai 380 oq 4080 in 2B ab 10 hl 18 gp 7 tl 0 loq 8CC0 3/3 mtu 1500 Output interface is GigabitEthernet3/3.13 current counters 0, 0 last reported 0, 0 Output Queue / Local Output Queue Bundle: [0-7] output queue 0x4080 local output queue 0x8CC0 PLU leaf data: 0xA0000000 0x0400F0FC 0xA1020380 0x24000000 Mask bits: 0 Origin AS: 0 Source lookup drop: no QOS group: 0 Traffic index: 0 Precedence not set Default Route: no PBR enabled: no gsr#execute-on slot 3 show ip hardware-cef tofab vrf test exact-route 10.1.1.1 10.99.99.99 ========= Line Card (Slot 3) ========= Leaf FCR 4 0x2438BA30 found 7 deep Fast Adjacency: alpha adjacency: 0x201E1F80 [0-7] ui 16 ai 380 oq 4080 in 2B ab 10 hl 18 gp 7 tl 0 loq 8CC0 3/3 mtu 1500 packets 0 bytes 0 Output Queue / Local Output Queue Bundle: [0-7] output queue 0x4080 local output queue 0x8CC0 10.1.1.1 -> 10.99.99.99 Interface: GigabitEthernet3/3.13
Table 7-10 lists some helpful troubleshooting commands when an ingress line card is functioning as a P router performing label switching.
Table 7-10 Helpful Commands to Troubleshoot MPLS VPNs on a P Router
| Incoming Line Card Engine Type | Software-Based show Commands | Hardware-Based show Commands |
|---|---|---|
0 1 | show mpls forwarding | N/A |
2 3 4 4+ | show mpls forwarding | show ip hardware-cef <prefix> detail show mpls hardware-lfib labels <low label> detail |
Note - In earlier Cisco IOS Software codes, the show mpls hardware-lfib command might not be available. However, the show tag hardware-tag command gives the same information.
Example 7-11 shows output for an Engine 3 serving as a P interface. The important data is the output interface and the tag information.
Example 7-11 Engine 3 LFIB Hardware Output
prp-12008#execute-on slot 0 show ip hardware-cef 10.0.0.99 detail
========= Line Card (Slot 0) =========
Root: 0x240CE000 Location: 0x240D0800 Data: 0x00898000
Node: 0x284C0000 Location: 0x284C0000 Data: 0x00000000 0x01017C00
Node: 0x2C0BE000 Location: 0x2C0BE630 Data: 0xA0000000 0x06004316
Node for 10.0.0.99: 0x2C0BE630
Leaf FCR 6 0x2C0BE630 found 3 deep
Fast Tag Rewrite:
[0-7]: ui 0 ai 7 oq 4080 in 11 ab 50 hl 20 gp 19 tl 4 loq 8000 0/0 mtu 4466
Output interface is POS0/0:1
1 tag: 21
current counters 0, 0 last reported 0, 0
Output Queue / Local Output Queue Bundle:
[0-7] output queue 0x4080 local output queue 0x8000
Example 7-12 shows hardware output for an Engine 4+ serving as a P interface. The key components are the label values and the operation value. In this case, the operation is a push of label/tag 0 out slot 2 or slot 1. This matches the data in the LFIB.
Example 7-12 Engine 4+ LFIB and Hardware Output Comparison
gsr#show mpls forwarding 10.0.0.99
Local Outgoing Prefix Bytes tag Outgoing Next Hop
tag tag or VC or Tunnel Id switched interface
19 0 10.0.0.99/32 0 PO2/0 point2point
0 10.0.0.99/32 0 PO1/0 point2point
gsr#execute-on slot 1 show ip hardware-cef 10.0.0.99 detail
========= Line Card (Slot 1) =========
HW Node : 70085000 = 9930300000203030
HW Node : 71018180 = E230200000443890
HW Node : 7221C480 = FF30100000506020
HW Node : 72830130 = 4A30000000601D40
HW Leaf :7300EA18, IP 10.0.0.99, Leaf: 6BB1670000602998: RPF 1, BGP 3 :Pkt 0, Byt 0
type=load bal: n=8 (addr=0x7300EA18, val=0x6BB1670000602998)
Hash 0 2 4 6 : TagLB ptrA entry=84B1600000140010
type=load bal: n=1 (addr=0x73014CC0, val=0x84B1600000140010)
0 TagLB ptrA entry=61B16000083000E7
type=load bal: n=1 (addr=0x70A00080, val=0x61B16000083000E7)
: TagLB ptrA entry=0, ptrB entry=75A1000000000000
type=mpls C: label #=1, tag =0, next=0x0 (addr=0x73800738, val=0x75A1000000000000)
type=mpls A: ttl=0, cos=0, op=push, idx=0, adj=0xFFA00 ctr=0x3207B (addr=0x71800738, val=0x9A8040FFA003207B)
Lbl_L1 0x797FD000, Lbl Opcode 0, Is_Fast 1, Lbl_L2 ptr 0x797FD008
Lbl_L2[0] 797FD008, L2 opcode 0, OIF 0, LQF 0, OQF 0, S 0, Adj_L3 0x79001000, Bundle 1 Payload Type 0
AdjL3[0] 0x79001000, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000800 = 0, Byt 0
AdjL3[1] 0x79001010, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000808 = 0, Byt 0
AdjL3[2] 0x79001020, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000810 = 0, Byt 0
AdjL3[3] 0x79001030, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000818 = 0, Byt 0
AdjL3[4] 0x79001040, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000820 = 0, Byt 0
AdjL3[5] 0x79001050, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000828 = 0, Byt 0
AdjL3[6] 0x79001060, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000830 = 0, Byt 0
AdjL3[7] 0x79001070, MTU 0, Slot 2, LQ 0, OQ 4000, BHdr 1, OI 0x90100000, Pkt 7A000838 = 0, Byt 0
Hash 1 3 5 7 : TagLB ptrA entry=7B1600000140011
type=load bal: n=1 (addr=0x73014CC8, val=0x07B1600000140011)
0 TagLB ptrA entry=2CB1600008300172
type=load bal: n=1 (addr=0x70A00088, val=0x2CB1600008300172)
: TagLB ptrA entry=0, ptrB entry=75A1000000000000
type=mpls C: label #=1, tag =0, next=0x0 (addr=0x73800B90, val=0x75A1000000000000)
type=mpls A: ttl=0, cos=0, op=push, idx=0, adj=0xFF9BE ctr=0x32029 (addr=0x71800B90, val=0x238040FF9BE32029)
Lbl_L1 0x797FCDF0, Lbl Opcode 0, Is_Fast 1, Lbl_L2 ptr 0x797FCDF8
Lbl_L2[0] 797FCDF8, L2 opcode 0, OIF 0, LQF 0, OQF 0, S 0, Adj_L3 0x79003D80, Bundle 1 Payload Type 0
AdjL3[0] 0x79003D80, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001EC0 = 0, Byt 0
AdjL3[1] 0x79003D90, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001EC8 = 0, Byt 0
AdjL3[2] 0x79003DA0, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001ED0 = 0, Byt 0
AdjL3[3] 0x79003DB0, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001ED8 = 0, Byt 0
AdjL3[4] 0x79003DC0, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001EE0 = 0, Byt 0
AdjL3[5] 0x79003DD0, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001EE8 = 0, Byt 0
AdjL3[6] 0x79003DE0, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001EF0 = 0, Byt 0
AdjL3[7] 0x79003DF0, MTU 0, Slot 1, LQ 0, OQ 4000, BHdr 5, OI 0x90100000, Pkt 7A001EF8 = 0, Byt 0
Some important caveats exist for the Engine 4 and Engine 4+ line cards. The Engine 4 line card does not support MPLS load sharing in label-to-label or label-to-IP paths. If multiple paths exist for label-switching or label-disposition paths, the Engine 4 line card just chooses one to forward traffic if multiple paths exist. DDTS CSCdy41261 documents this issue and is viewable on the Cisco Bug Toolkit. This caveat does not affect the Engine 4+ line card.
Only Engines 0, 2, 3, and 4+ support MPLS VPN PE functionality totally with VPN imposition and VPN disposition.
Cisco 10000 Series Router
A Cisco 10000 is a PXF-based platform. If you have an MPLS issue, not only must you check the FIB/LFIB but you must also evaluate the PXF table, because that is where forwarding stems from. Some other PXF-based platforms do not support MPLS. In this case, these platforms punt the MPLS packets to the CEF path. Again, the key is to check that the labels and outgoing interfaces programmed in hardware are correct and consistent with the LFIB or FIB information, depending on the router's MPLS function.
Some helpful commands to troubleshoot a Cisco 10000 router include the following:
show hardware pxf cpu mpls labels <local label> show hardware pxf cpu cef <prefix> show hardware pxf cpu cef vrf <name> <prefix>
CEF and MPLS VPN Load-Sharing Considerations
Chapter 6, "Load Sharing with CEF," covers CEF and load sharing in an IP-only environment. The following sections address some special cases and troubleshooting techniques in an MPLS VPN environment.
PE-CE Load Sharing: CE Multihomed to Same PE
In Figure 7-5, Router CE1 is multihomed to Router PE1, and two paths exist between Routers CE1 and PE1. Many customers want to load-share across multiple connections between the same PE and CE routers. In the example shown, the number of paths used in the CE1-to-PE1 direction would depend on the routing table on CE1. The same is true in the PE1-to-CE1 direction, although PE1 uses the LFIB to reach CE1. Sometimes the use of MPLS or MPLS VPNs by the service providers confuses the customer. However, the principles remain the same with and without MPLS in this case. The CE has no knowledge of MPLS. Therefore, the load-sharing principles covered in Chapter 6 all apply. The same is true with the PE, except that it uses the LFIB to pass traffic from the MPLS cloud to the CE.
PE-CE Load Sharing