Chapter 12: Multilayer Switching

Cisco Press

1 2 3 4 Page 2
Page 2 of 4

NetFlow switching has given way to a more efficient form of multilayer switching: Cisco Express Forwarding. Cisco developed CEF for its line of routers, offering high-performance packet forwarding through the use of dynamic lookup tables.

CEF also has been carried over to the Catalyst switching platforms. The following platforms all perform CEF in hardware:

  • Catalyst 6500 Supervisor 720 (with an integrated MSFC3)

  • Catalyst 6500 Supervisor 2/MSFC2 combination

  • Catalyst 4500 Supervisor III, IV, and V

  • Fixed-configuration switches, such as the Catalyst 3750, 3560, 3550, and 2950

CEF runs by default, taking advantage of the specialized hardware.

A CEF-based multilayer switch consists of two basic functional blocks, as shown in Figure 12-3: The Layer 3 Engine is involved in building routing information that the Layer 3 Forwarding Engine can use to switch packets in hardware.

Figure 12-3

Figure 12-3

Packet Flow Through a CEF-Based Multilayer Switch

Forwarding Information Base

The Layer 3 engine (essentially a router) maintains routing information, whether from static routes or dynamic routing protocols. Basically, the routing table is reformatted into an ordered list with the most specific route first, for each IP destination subnet in the table. The new format is called a Forwarding Information Base (FIB) and contains routing or forwarding information that the network prefix can reference.

In other words, a route to might be contained in the FIB along with routes to and, if those exist. Notice that these examples are increasingly more specific subnets, as designated by the longer subnet masks. In the FIB, these would be ordered with the most specific, or longest match, first, followed by less specific subnets. When the switch receives a packet, it easily can examine the destination address and find the longest-match destination route entry in the FIB.

The FIB also contains the next-hop address for each entry. When a longest-match entry is found in the FIB, the Layer 3 next-hop address is found, too.

You might be surprised to know that the FIB also contains host route (subnet mask entries. These normally are not found in the routing table unless they are advertised or manually configured. Host routes are maintained in the FIB for the most efficient routing lookup to directly connected or adjacent hosts.

As with a routing table, the FIB is dynamic in nature. When the Layer 3 engine sees a change in the routing topology, it sends an update to the FIB. Anytime the routing table receives a change to a route prefix or the next-hop address, the FIB receives the same change. Also, if a next-hop address is changed or aged out of the Address Resolution Protocol (ARP) table, the FIB must reflect the same change.

You can display FIB table entries related to a specific interface or VLAN with the following form of the show ip cef command:

Switch# show ip cef [type mod/num | vlan vlan-id] [detail]

The FIB entries corresponding to the VLAN 101 switched virtual interface might be shown as demonstrated in Example 12-1.

Example 12-1 Displaying FIB Table Entries for a Specified VLAN

Switch# show ip cef vlan 101
Prefix       Next Hop       Interface     attached       Vlan101       Vlan101       Vlan101

You also can view FIB entries by specifying an IP prefix address and mask, using the following form of the show ip cef command:

Switch# show ip cef [prefix-ip prefix-mask] [longer-prefixes] [detail]

The output in Example 12-2 displays any subnet within that is known by the switch, regardless of the prefix or mask length. Normally, only an exact match of the IP prefix and mask will be displayed if it exists in the CEF table. To see other longer match entries, you can add the longer-prefixes keyword.

Example 12-2 Displaying FIB Table Entries for a Specified IP Prefix Address/Mask

Switch# show ip cef longer-prefixes
Prefix       Next Hop       Interface     attached       Vlan101       Vlan101       Vlan101     attached       Vlan102     Vlan99
     Vlan99     Vlan99
     Vlan99     Vlan99
[output omitted]

Notice that the first three entries are the same ones listed in Example 12-1. Other subnets also are displayed, along with their next-hop router addresses and switch interfaces.

You can add the detail keyword to see more information about each FIB table entry for CEF, as demonstrated in Example 12-3.

Example 12-3 Displaying Detailed CEF Entry Information

Switch# show ip cef detail, version 270, epoch 0, per-destination sharing
0 packets, 0 bytes
 via, Vlan99, 0 dependencies
  traffic share 1
  next hop, Vlan99
  valid adjacency
 via, Vlan99, 0 dependencies
  traffic share 1
  next hop, Vlan99
  valid adjacency
 0 packets, 0 bytes switched through the prefix
 tmstats: external 0 packets, 0 bytes
      internal 0 packets, 0 bytes

The version number describes the number of times the CEF entry has been updated since the table was generated. The epoch number denotes the number of times the CEF table has been flushed and regenerated as a whole. The subnet has two next-hop router addresses, so the local switch is using per-destination load sharing between the two routers.

After the FIB is built, packets can be forwarded along the bottom dashed path in Figure 12-3. This follows the hardware switching process, in which no "expensive" or time-consuming operations are needed. At times, however, a packet cannot be switched in hardware, according to the FIB. Packets then are marked as "CEF punt" and immediately are sent to the Layer 3 engine for further processing, as shown in the top dashed path in Figure 12-3. Some of the conditions that can cause this are as follows:

  • An entry cannot be located in the FIB

  • The FIB table is full

  • The IP Time To Live (TTL) has expired

  • The maximum transmission unit (MTU) is exceeded, and the packet must be fragmented

  • An Internet Control Message Protocol (ICMP) redirect is involved

  • The encapsulation type is not supported

  • Packets are tunneled, requiring a compression or encryption operation

  • An access list with the log option is triggered

  • A Network Address Translation (NAT) operation must be performed (except on the Catalyst 6500 Supervisor 720, which can handle NAT in hardware)

CEF operations can be handled on a single hardware platform, such as the Catalyst 3560 and 3750 switches. The FIB is generated and contained centrally in the switch. CEF also can be optimized through the use of specialized forwarding hardware, using the following techniques:

  • Accelerated CEF (aCEF)—CEF is distributed across multiple Layer 3 forwarding engines, typically located on Catalyst 6500 line cards. These engines do not have the capability to store and use the entire FIB, so only a portion of the FIB is downloaded to them at any time. This functions as an FIB "cache," containing entries that are likely to be used again. If FIB entries are not found in the cache, requests are sent to the Layer 3 engine for more FIB information. The net result is that CEF is accelerated on the line cards, but not necessarily at a sustained wire-speed rate.

  • Distributed CEF (dCEF)—CEF can be distributed completely among multiple Layer 3 forwarding engines for even greater performance. Because the FIB is self-contained for complete Layer 3 forwarding, it can be replicated across any number of independent Layer 3 forwarding engines. The Catalyst 6500 has line cards that support dCEF, each with its own FIB table and forwarding engine. A central Layer 3 engine (the MSFC3, for example) maintains the routing table and generates the FIB, which is then dynamically downloaded in full to each of the line cards.

Adjacency Table

A router normally maintains a routing table containing Layer 3 network and next-hop information, and an ARP table containing Layer 3 to Layer 2 address mapping. These tables are kept independently.

Recall that the FIB keeps the Layer 3 next-hop address for each entry. To streamline packet forwarding even more, the FIB has corresponding Layer 2 information for every next-hop entry. This portion of the FIB is called the adjacency table, consisting of the MAC addresses of nodes that can be reached in a single Layer 2 hop.

You can display the adjacency table's contents with the following command:

Switch# show adjacency [type mod/num | vlan vlan-id] [summary | detail]

As an example, the total number of adjacencies known on each physical or VLAN interface can be displayed with the show adjacency summary command, as demonstrated in Example 12-4.

Example 12-4 Displaying the Total Number of Known Adjacencies

Switch# show adjacency summary
Adjacency Table has 106 adjacencies
 Table epoch: 0 (106 entries at this epoch)
 Interface         Adjacency Count
 Vlan99          21
 Vlan101          3
 Vlan102          1
 Vlan103          47
 Vlan104          7
 Vlan105          27

Adjacencies are kept for each next-hop router and each host that is connected directly to the local switch. You can see more detailed information about the adjacencies by using the detail keyword, as demonstrated in Example 12-5.

Example 12-5 Displaying Detailed Information About Adjacencies

Switch# show adjacency vlan 99 detail
Protocol Interface         Address
IP    Vlan99
                  0 packets, 0 bytes
                  ARP    01:52:50
                  Epoch: 0
IP    Vlan99
                  1 packets, 104 bytes
                  ARP    04:02:11
                  Epoch: 0

Notice that the adjacency entries include both the IP address (Layer 3) and the MAC address (Layer 2) of the directly attached host. The MAC address could be shown as the first six octets of the long string of hex digits (as shaded in the previous output) or on a line by itself. The remainder of the string of hex digits contains the MAC address of the Layer 3 engine's interface (six octets, corresponding to the Vlan99 interface in the example) and the EtherType value (two octets, where 0800 denotes IP).

The adjacency table information is built from the ARP table. Example 12-5 shows adjacency with the age of its ARP entry. As a next-hop address receives a valid ARP entry, the adjacency table is updated. If an ARP entry does not exist, the FIB entry is marked as "CEF glean." This means that the Layer 3 forwarding engine can't forward the packet in hardware because of the missing Layer 2 next-hop address. The packet is sent to the Layer 3 engine so that it can generate an ARP request and receive an ARP reply. This is known as the CEF glean state, in which the Layer 3 engine must glean the next-hop destination's MAC address.

The glean state can be demonstrated in several ways, as demonstrated in Example 12-6.

Example 12-6 Displaying Adjacencies in the CEF Glean State

Switch# show ip cef adjacency glean
Prefix       Next Hop       Interface     attached       Vlan101     attached       EOBC0/0
[output omitted]
Switch# show ip arp

Switch# show ip cef detail, version 688, epoch 0, attached, connected
0 packets, 0 bytes
 via Vlan101, 0 dependencies
  valid glean adjacency

Notice that the FIB entry for directly connected host is present but listed in the glean state. The show ip arp command shows that there is no valid ARP entry for the IP address.

During the time that an FIB entry is in the CEF glean state waiting for the ARP resolution, subsequent packets to that host are immediately dropped so that the input queues do not fill and the Layer 3 engine does not become too busy worrying about the need for duplicate ARP requests. This is called ARP throttling or throttling adjacency. If an ARP reply is not received in 2 seconds, the throttling is released so that another ARP request can be triggered. Otherwise, after an ARP reply is received, the throttling is released, the FIB entry can be completed, and packets can be forwarded completely in hardware.

The adjacency table also can contain other types of entries so that packets can be handled efficiently. For example, you might see the following adjacency types listed:

  • Null adjacency—Used to switch packets destined for the null interface. The null interface always is defined on a router or switch; it represents a logical interface that silently absorbs packets without actually forwarding them.

  • Drop adjacency—Used to switch packets that can't be forwarded normally. In effect, these packets are dropped without being forwarded. Packets can be dropped because of an encapsulation failure, an unresolved address, an unsupported protocol, no valid route present, no valid adjacency, or a checksum error. You can gauge drop adjacency activity with the following command:

Switch# show cef drop
CEF Drop Statistics
Slot Encap_fail Unresolved Unsupported  No_route   No_adj ChkSum_Err
RP    8799327      1    45827   5089667     32      0
  • Discard adjacency—Used when packets must be discarded because of an access list or other policy action.

  • Punt adjacency—Used when packets must be sent to the Layer 3 engine for further processing. You can gauge the CEF punt activity by looking at the various punt adjacency reasons listed by the show cef not-cef-switched command:

1 2 3 4 Page 2
Page 2 of 4
SD-WAN buyers guide: Key questions to ask vendors (and yourself)