Excerpt from CCNP TSHOOT 642-832.
By Published by ISBN-10: ISBN-13:

E-mail this to a friend

Newsletters: Sign-Up & Save! Receive Special Offers, Free Chapters, Articles Reference Guide Updates, and plug into the pulse of what’s happening in your corner of the industry by subscribing to InformIT newsletters! FREE coupon after sign-up!

Try Safari Books Online NOW! Access the largest fully searchable e-reference library for programmers and IT professionals!

This chapter covers the following subjects:

Cisco IOS Diagnostic Tools: This section shows how a few readily accessible Cisco IOS Software commands can be used to quickly gather information as part of a structured troubleshooting process.

Specialized Diagnostic Tools: This section introduces a collection of specialized features, such as Switched Port ­Analyzer (SPAN), Remote SPAN (RSPAN), Simple Mail ­Transfer Protocol (SMTP), NetFlow, and Embedded Event Manager (EEM), which can be used to collect information about a problem.

Key to maintaining and troubleshooting a network is the collection of information about that network. Fortunately, *Cisco IOS offers many commands that* can be used for information gathering. Mastery of these basic tools can dramatically reduce the time a troubleshooter spends isolating the specific information needed for a troubleshooting task. (Note: The highlighted words are part of Cisco Subnet’s February 2010 book giveaway. Click here for monthly contest information.)

Beyond basic Cisco IOS commands, many network devices support features targeted toward the collection of information. Perhaps an event occurs on a network device, such as a router’s processor utilization exceeding a defined threshold. The network device could report the occurrence of such an event. Alternatively, network devices might be periodically queried by a network management system for device and traffic statistics.

This chapter covers several basic Cisco IOS commands, in addition to specialized information collection features. These features not only help a troubleshooter collect information about a problem, but they can create a baseline of network performance. This baseline data can then be contrasted with data collected when a problem is occurring. The comparison of these two data sets often provides insight into the underlying cause of a problem.

“Do I Know This Already?” Quiz

The “Do I Know This Already?” quiz helps you determine your level of knowledge of this chapter’s topics before you begin. Table 3-1 details the major topics discussed in this chapter and their corresponding quiz questions.

Table 3-1  “Do I Know This Already?” Section-to-Question Mapping

1. Which of the following commands displays a router’s running configuration, starting where the routing protocol configuration begins?

   1. show running-config | tee router

   2. show running-config | begin router

   3. c. show running-config | redirect router

   4. d. show running-config | append router

2. Which of the following is the ping response to a transmitted ICMP Echo datagram that needed to be fragmented when fragmentation was not permitted?

   1. U

   2. .

   3. M

   4. D

3. Which portion of the show interfaces command output indicates that a router received information faster than the information could be processed by the router?

   1. input queue drops

   2. output queue drops

   3. input errors

   4. output errors

4. The types of information collection used in troubleshooting fall into which three broad categories? (Choose three.)

   1. Troubleshooting information collection

   2. Baseline information collection

   3. QoS information collection

   4. Network event information collection

5. What features available on Cisco Catalyst switches allow you to connect a network monitor to a port on one switch to monitor traffic flowing through a port on a different switch?

   1. RSTP

   2. SPAN

   3. RSPAN

   4. SPRT

6. Which two of the following are characteristics of the NetFlow feature? (Choose the two best answers.)

   1. Collects detailed information about traffic flows

   2. Collects detailed information about device statistics

   3. Uses a pull model

   4. Uses a push model

7. Identify the Cisco IOS feature that allows you to create your own event definition for a network device and specify the action that should be performed in response to that event.

   1. SNMP

   2. EEM

   3. NetFlow

   4. syslog

Foundation Topics

Cisco IOS Diagnostic Tools

After a problem has been clearly defined, the first step in diagnosing that problem is ­collecting information, as described in Chapter 2, “Introduction to Troubleshooting Processes.” Because the collection of information can be one of the most time consuming of the troubleshooting processes, the ability to quickly collect appropriate information becomes a valuable troubleshooting skill. This section introduces a collection of basic Cisco IOS commands useful in gathering information and discusses the filtering of irrelevant information from the output of those commands. Also included in this section are commands helpful in diagnosing connectivity and hardware issues.

Filtering the Output of show Commands

Cisco IOS offers multiple show commands useful for gathering information. However, many of these show commands produce a large quantity of output.

Consider the output shown in Example 3-1. The output from the show processes cpu command generated approximately 180 lines of output, making it challenging to pick out a single process.

Example: show processes cpu Command

Example 3-1  show processes cpu Command Output

R1# show processes cpu
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
 PID    Runtime(ms) Invoked uSecs   5Sec    1Min    5Min    TTY Process
   1    4   3   1333    0.00%   0.00%   0.00%   0   Chunk Manager
   2    7245    1802    4020    0.08%   0.08%   0.08%   0   Load Meter
   3    56  2040    27  0.00%   0.00%   0.00%   0   OSPF Hello 1
   4    4   1   4000    0.00%   0.00%   0.00%   0   EDDRI_MAIN
   5    21998   1524    14434   0.00%   0.32%   0.25%   0   Check heaps
   6    0   1   0   0.00%   0.00%   0.00%   0   Pool Manager
   7    0   2   0   0.00%   0.00%   0.00%   0   Timers
   8    0   1   0   0.00%   0.00%   0.00%   0   Crash writer
   9    0   302 0   0.00%   0.00%   0.00%   0   Environmental mo
  10    731 1880    388 0.00%   0.00%   0.00%   0   ARP Input
...OUTPUT OMITTED...
 171        0   1   0   0.00%   0.00%   0.00%   0   lib_off_app
 172        4   2   2000    0.00%   0.00%   0.00%   0   Voice Player
 173        0   1   0   0.00%   0.00%   0.00%   0   Media Record
 174        0   1   0   0.00%   0.00%   0.00%   0   Resource Measure
 175        12  6   2000    0.00%   0.00%   0.00%   0   Session Applicat
 176        12  151 79  0.00%   0.00%   0.00%   0   RTPSPI
 177        4   17599   0   0.00%   0.00%   0.00%   0   IP NAT Ager
 178        0   1   0   0.00%   0.00%   0.00%   0   IP NAT WLAN
 179        8   314 25  0.00%   0.00%   0.00%   0   CEF Scanner

Perhaps you were only looking for CPU utilization statistics for the Check heaps process. Because you know that the content of the one line you are looking for contains the text Check heaps, you could take the output of the show processes cpu command and pipe that output (that is, use the | character) to the include Check heaps statement. The piping of the output causes the output to be filtered to only include lines that include the text Check heaps, as demonstrated in Example 3-2. This type of filtering can help troubleshooters more quickly find the data they are looking for.

Example 3-2  Filtering the show processes cpu Command Output

R1# show processes cpu | include Check heaps
   5       24710      1708      14467  1.14%  0.26%  0.24%   0 Check heaps

Example: show ip interfaces brief Command

Similar to piping output to the include option, you could alternatively pipe output to the exclude option. The exclude option can display all lines of the output except lines containing the string you specify. For example, the show ip interfaces brief command can display IP address and status information for all interfaces on a router, as shown in Example 3-3.

Example 3-3  show ip interface brief Command Output

R1# show ip interface brief
Interface       IP-Address      OK? Method  Status  Protocol
FastEthernet0/0 192.168.1.11    YES NVRAM   up      up
Serial0/0   unassigned  YES NVRAM   administratively down   down

FastEthernet0/1 192.168.0.11    YES NVRAM   up  up

Serial0/1   unassigned  YES NVRAM   administratively down   down

NVI0    unassigned  YES unset   up  up

Loopback0   10.1.1.1    YES NVRAM   up  up

Notice in Example 3-3 that some of the interfaces have an IP address of unassigned. If you want to only view information pertaining to interfaces with assigned IP addresses, you can pipe the output of the show ip interface brief command to exclude unassigned, as illustrated in Example 3-4.

Example 3-4  Filtering Output from the show ip interface brief Command

R1# show ip interface brief | exclude unassigned
Interface   IP-Address  OK? Method  Status  Protocol
FastEthernet0/0 192.168.1.11    YES NVRAM   up  up

FastEthernet0/1 192.168.0.11    YES NVRAM   up  up

Loopback0   10.1.1.1    YES NVRAM   up  up

Example: Jumping to the First Occurrence of a String in show Command Output

As another example, you might be troubleshooting a routing protocol issue and want to see the section of your running configuration where the routing protocol configuration begins. Piping the output of the show running-config command to begin router, as shown in Example 3-5, skips the initial portion of the show running-config output and begins displaying the output where the routing protocol configuration begins.

Example 3-5  Filtering the Output from the show running-config Command

R1# show running-config | begin router
router ospf 1
 log-adjacency-changes
 network 0.0.0.0 255.255.255.255 area 0
...OUTPUT OMITTED...

Example: The show ip route Command

Another command that often generates a lengthy output, especially in larger environments, is the show ip route command. As an example, consider the show ip route output presented in Example 3-6.

Example 3-6  Sample show ip route Command Output

R1# show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     172.16.0.0/30 is subnetted, 2 subnets
O       172.16.1.0 [110/65] via 192.168.0.22, 00:50:57, FastEthernet0/1
O       172.16.2.0 [110/65] via 192.168.0.22, 00:50:57, FastEthernet0/1
     10.0.0.0/8 is variably subnetted, 6 subnets, 3 masks
O       10.2.2.2/32 [110/2] via 192.168.0.22, 00:50:57, FastEthernet0/1
O       10.1.3.0/30 [110/129] via 192.168.0.22, 00:50:57, FastEthernet0/1
O       10.3.3.3/32 [110/66] via 192.168.0.22, 00:50:57, FastEthernet0/1
O       10.1.2.0/24 [110/75] via 192.168.0.22, 00:50:58, FastEthernet0/1
C       10.1.1.1/32 is directly connected, Loopback0
O       10.4.4.4/32 [110/66] via 192.168.0.22, 00:50:58, FastEthernet0/1
C    192.168.0.0/24 is directly connected, FastEthernet0/1
C    192.168.1.0/24 is directly connected, FastEthernet0/0

Although the output shown in Example 3-6 is relatively small, some IP routing tables contain hundreds or even thousands of entries. If, for example, you wanted to determine if a route for network 172.16.1.0 were present in a routing table, you could issue the command show ip route 172.16.1.0, as depicted in Example 3-7.

Perhaps you were looking for all subnets of the 172.16.0.0/16 address space. In that event, you could specify the subnet mask and the longer-prefixes argument as part of your command. Such a command, as demonstrated in Example 3-8, shows all subnets of network 172.16.0.0/16, including the major classful network of 172.16.0.0/16.

Example 3-7  Specifying a Specific Route with the show ip route Command

R1# show ip route 172.16.1.0
Routing entry for 172.16.1.0/30
  Known via “ospf 1”, distance 110, metric 65, type intra area
  Last update from 192.168.0.22 on FastEthernet0/1, 00:52:08 ago
  Routing Descriptor Blocks:
  * 192.168.0.22, from 10.2.2.2, 00:52:08 ago, via FastEthernet0/1
      Route metric is 65, traffic share count is 1

Example 3-8  Filtering Output from the show ip route Command with the longer-prefixes Option

R1#show ip route 172.16.0.0 255.255.0.0 longer-prefixes
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     172.16.0.0/30 is subnetted, 2 subnets
O       172.16.1.0 [110/65] via 192.168.0.22, 00:51:39, FastEthernet0/1
O       172.16.2.0 [110/65] via 192.168.0.22, 00:51:39, FastEthernet0/1

Redirecting show Command Output to a File

Imagine that you are working with Cisco Technical Assistance Center (TAC) to troubleshoot an issue, and they want a file containing output from the show tech-support command issued on your router. Example 3-9 shows how you can use the | redirect option to send output from the show tech-support command to a file on a TFTP server.

Notice that directing output to a file suppresses the onscreen output. If you wanted both (that is, for the output to be displayed onscreen and stored to a file), you could pipe the output to the tee option, as demonstrated in Example 3-10.

Example 3-9  Redirecting Output to a TFTP Server

R1# show tech-support | redirect tftp://192.168.1.50/tshoot.txt
!
R1#

Example 3-10  Redirecting Output While Also Displaying the Output Onscreen

R1# show tech-support | tee tftp://192.168.1.50/tac.txt
!

---------------------show version---------------------

Cisco IOS Software, C2600 Software (C2600-IPVOICE_IVS-M), Version 12.4(3b), RELE
ASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Thu 08-Dec-05 17:35 by alnguyen
...OUTPUT OMITTED...

If you already have an output file created and you want to append the output of another show command to your existing file, you can pipe the output of your show command to the append option. Example 3-11 shows how to use the append option to append the output of the show ip interface brief command to a file named baseline.txt.

Example 3-11  Appending Output to an Existing File

R1# show ip interface brief | append tftp://192.168.1.50/baseline.txt
!
R1#

Troubleshooting Connectivity

In addition to show command output, you can use many other Cisco IOS commands to troubleshoot network conditions. A common command, which can be used to check network connectivity, is the ping command. A basic ping command sends Internet Control Message Protocol (ICMP) Echo messages to a specified destination; for every ICMP Echo Reply received from that specified destination, an exclamation point appears in the output, as shown in Example 3-12.

Example 3-12  Basic ping Command

R1# ping 10.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
!!!!!                                                                     

The ping command does have several options that can prove useful during troubleshooting. For example:

• size: Specifies the number of bytes per datagram

• repeat: Specifies the number of ICMP Echo messages sent (defaults to 5)

• timeout: Specifies the number of seconds to wait for an ICMP Echo Reply

• source: Specifies the source of the ICMP Echo datagrams

• df-bit: Sets the do not fragment bit in the ICMP Echo datagram

Not only can a ping command indicate that a given IP address is reachable, but the response to a ping command might provide insight into the nature of a problem. For example, if the ping results indicate alternating failures and successes (that is, !.!.!), a troubleshooter might conclude that traffic is being load-balanced between the source and destination IP addresses. Traffic flowing across one path is successful, whereas traffic flowing over the other path is failing.

You can also use the ping command to create a load on the network to troubleshoot the network under heavy use. Specifically, you can specify a datagram size of 1500 bytes, along with a large byte count and a timeout of zero seconds, as shown in Example 3-13.

Notice that all the pings failed. These failures occurred because of the zero-second timeout. Specifically, the router did not wait for any amount of time before considering the ping to have failed and sending another ICMP Echo message.

Example 3-13  Creating a Heavy Load on the Network

R1# ping 10.4.4.4 size 1500 repeat 9999 timeout 0

Type escape sequence to abort.
Sending 9999, 1500-byte ICMP Echos to 10.4.4.4, timeout is 0 seconds:
......................................................................
......................................................................
......................................................................
...OUTPUT OMITTED...

Perhaps you suspect that an interface has a nondefault maximum transmission unit (MTU) size. You could send ICMP Echo messages across that interface using the df-bit and size options of the ping command to specify the size of the datagram to be sent. The df-bit option instructs a router to drop this datagram rather than fragmenting it if fragmentation is required.

Example 3-14 shows the sending of pings with the do not fragment bit set. Notice the M in the ping responses. An M indicates that fragmentation was required but could not be performed because the do not fragment bit was set. Therefore, you can conclude that a link between the source and destination is using a nonstandard MTU (that is, an MTU less than 1500 bytes).

Example 3-14  Pinging with the Do Not Fragment Bit Set

R1# ping 10.4.4.4 size 1500 df-bit
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
Packet sent with the DF bit set                                               
M.M.M                                                                     

The challenge is how to determine the nondefault MTU size without multiple manual attempts. An extended ping can help with such a scenario. Consider Example 3-15, which issues the ping command without command-line parameters. This invokes the extended ping feature. The extended ping feature allows you to granularly customize your pings. For example, you could specify a range of datagram sizes to use in your pings to help determine the size of a nondefault MTU. Specifically, in Example 3-15, you could determine that the MTU across at least one of the links from the source to the destination IP address was set to 1450 bytes, because the M ping responses begin after 51 ICMP Echo datagrams were sent (with datagram sizes in the range of 1400 to 1450 bytes).

Example 3-15  Extended Ping Performing a Ping Sweep

R1# ping
Protocol [ip]:
Target IP address: 10.4.4.4
Repeat count [5]: 1
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1400
Sweep max size [18024]: 1500
Sweep interval [1]:
Type escape sequence to abort.
Sending 101, [1400..1500]-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!M.M.M.M.M.M.M.M.M.M
.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.
Success rate is 50 percent (51/101), round-trip min/avg/max = 60/125/232 ms

Although the ping command can be useful for testing Layer 3 (that is, the network layer) connectivity, the telnet command is useful for troubleshooting Layer 4 (that is, the transport layer). Specifically, although Telnet typically uses TCP port 23, you can specify an alternate port number to see if a particular Layer 4 service is running on a destination IP address. Such an approach might be useful if you are using a divide and conquer approach, starting at Layer 3 (which was determined to be operational as a result of a successful ping), or a bottom-up approach (which has also confirmed Layer 3 to be operational).

To illustrate, notice the telnet 192.168.1.50 80 command issued in Example 3-16. This command causes router R1 to attempt a TCP connection with 192.168.1.50 using port 80 (that is, the HTTP port). The response of Open indicates that 192.168.1.50 is indeed running a service on port 80.

Example 3-16  Using Telnet to Connect to a Nondefault Port

R1#telnet 192.168.1.50 80
Trying 192.168.1.50, 80 ... Open

Troubleshooting Hardware

In addition to software configurations, a network’s underlying hardware often becomes a troubleshooting target. As a reference, Table 3-2 offers a collection of Cisco IOS commands used to investigate hardware performance issues.

Table 3-2  Cisco IOS Commands for Hardware Troubleshooting

Specialized Diagnostic Tools

The collection of network information occurs not just as part of a structured troubleshooting process but when gathering baseline data. Additionally, the occurrence of certain network events (for example, processor utilization on a network server exceeding a specified threshold) might be configured to trigger the writing of log information (for example, to a syslog server). Cisco offers network maintenance and troubleshooting tools targeted toward this type of data collection. This section introduces you to a sampling of these specialized tools, which can supplement the Cisco IOS commands used for data collection as discussed in the section “Cisco IOS Diagnostic Tools,” earlier in this chapter.

Using Specialized Tools in the Troubleshooting Process

Chapter 2 introduced you to a series of processes (and subprocesses) that can work together in a structured troubleshooting process. Several of these processes involve the collection of information:

• Difficulty report: By proactively monitoring network devices, you might be alerted to impending performance issues before users are impacted.

• Collect information: The collection of information when troubleshooting a problem can often be made more efficient through the use of specialized maintenance and troubleshooting tools.

• Examine collected information: As troubleshooters investigate the information they collected while troubleshooting a problem, they need to know what normal network behavior looks like. They can then contrast that normal (that is, baseline) behavior against what they are observing in their collected data. Specialized maintenance and troubleshooting tools can be used in a network to collect baseline data on an ongoing basis.

• Verify hypothesis: Specialized maintenance and troubleshooting tools help a troubleshooter roll back an attempted fix, if that fix proves unsuccessful.

Most of the information collection tasks performed in the previous processes fall into one of three categories:

• Troubleshooting information collection: This information collection is conducted as part of troubleshooting a reported problem.

• Baseline information collection: This information collection is conducted when the network is operating normally, to form a frame of reference against which other data can be compared.

• Network event information collection: Some network devices can be configured to automatically generate alerts in response to specific conditions (for example, configured utilization levels on a switch, router, or server being exceeded).

Performing Packet Captures

You can use dedicated appliances or PCs running packet capture software to collect and store packets flowing across a network link. When troubleshooting, analysis of captured packets can provide insight into how a network is treating traffic flow. For example, a packet capture data file can show if packets are being dropped or if sessions are being reset. You can also look inside Layer 2, 3, and 4 headers using a packet capture application. For example, you can view a packet’s Layer 3 header to determine that packet’s Layer 3 quality of service (QoS) priority marking. An example of a popular and free packet capture utility you can download is Wireshark (http://www.wireshark.org), as shown in Figure 3-1.

Figure 3-1

Wireshark Packet Capture Application

Capturing and analyzing packets, however, presents two major obstacles. First, the volume of data collected as part of a packet capture can be so large that finding what you are looking for can be a challenge. Therefore, you should understand how to use your packet capture application’s filtering features.

A second challenge is that if you want to monitor, for example, traffic flow between two network devices connected to a switch, the packets traveling between those two devices will not appear on your packet capture device’s switch port. Fortunately, Cisco IOS supports a feature known as SPAN. SPAN instructs a switch to send copies of packets seen on one port (or one VLAN) to another port. You can connect your packet capture device to this other port, as shown in Figure 3-2.

Figure 3-2

Cisco Catalyst Switch Configured for SPAN

Notice that Figure 3-2 depicts a client (connected to Gigabit Ethernet 0/2) communicating with a server (connected to Gigabit Ethernet 0/1). A troubleshooter inserts a packet capture device into Gigabit Ethernet 0/3. However, a switch’s default behavior does not send packets flowing between the client and server out the port connected to the packet capture device. To cause port Gigabit Ethernet 0/3 to receive a copy of all packets sent or received by the server, SPAN is configured on the switch, as shown in Example 3-17.

Notice that Example 3-17 uses the monitor session id source interface interface_id command to indicate that a SPAN monitoring session with a locally significant identifier of 1 will copy packets crossing (that is, entering and exiting) port Gigabit Ethernet 0/1. Then the monitor session id destination interface interface_id command is used to specify port Gigabit Ethernet 0/3 as the destination port for those copied packets. A laptop running packet capture software connected to port Gigabit Ethernet 0/3 will now receive a copy of all traffic the server is sending or receiving.

Example 3-17  SPAN Configuration

Cat3550# conf term
Enter configuration commands, one per line.  End with CNTL/Z.
Cat3550(config)# monitor session 1 source interface gig 0/1
Cat3550(config)# monitor session 1 destination interface gig 0/3
Cat3550(config)# end
Cat3550# show monitor
Session 1
------------
Type    : Local Session
Source Ports    :
    Both    : Gi0/1
Destination Ports   : Gi0/3
    Encapsulation   : Native
        Ingress : Disabled

In larger environments, a network capture device connected to one switch might need to capture packets flowing through a different switch. Remote SPAN (RSPAN) makes such a scenario possible. Consider Figure 3-3, where a troubleshooter has her laptop running a packet capture application connected to port Fast Ethernet 5/2 on switch SW2. The traffic that needs to be captured is traffic coming from and going to the server connected to port Gigabit Ethernet 0/1 on switch SW1.

Figure 3-3

Cisco Catalyst Switch ConFigured for RSPAN

A VLAN is configured whose purpose is to carry captured traffic between the switches. Therefore, a trunk exists between switches SW1 and SW2 to carry the SPAN VLAN in addition to a VLAN carrying user data. Example 3-18 shows the configuration on switch SW1 used to create the SPAN VLAN (that is, VLAN 20) and to specify that RSPAN should monitor port Gigabit Ethernet 0/1 and send packets sent and received on that port out of Gigabit Ethernet 0/3 on VLAN 20. The show monitor command is then used to verify the RSPAN source and destination. Also, note that by default the monitor session id source command monitors both incoming and outgoing traffic on the monitored port.

Example 3-18  RSPAN Configuration on Switch SW1

SW1# conf term
SW1(config)# vlan 20
SW1(config-vlan)# name SPAN
SW1(config-vlan)# remote-span
SW1(config-vlan)# exit
SW1(config)# monitor session 1 source interface gig 0/1
SW1(config)# monitor session 1 destination remote vlan 20 reflector-port gig 0/3
SW1(config)# end
SW1#show monitor
Session 1
------------
Type    : Remote Source Session

Source Ports    :                                                   
    Both    : Gi0/1                                              

 Reflector Port : Gi0/3
 Dest RSPAN VLAN    : 20                                                 

Example 3-19 shows the configuration on switch SW2 used to create the SPAN VLAN (which is not necessary if the switches belong to the same VTP domain), to specify that RSPAN should receive captured traffic from VLAN 20 and send it out port Fast Ethernet 5/2.

Example 3-19  RSPAN Configuration on Switch SW2

SW2# conf term
SW2(config)# vlan 20
SW2(config-vlan)# name SPAN
SW2(config-vlan)# remote-span
SW2(config-vlan)# exit
SW2(config)# monitor session 2 source remote vlan 20
SW2(config)# monitor session 2 destination interface fa 5/2
SW2(config)# end
SW2# show monitor
Session 2
------------
Type              : Remote Destination Session
Source RSPAN VLAN : 20                                                         
Destination Ports : Fa5/2                                                 

Creating a Baseline with SNMP and NetFlow

Simple Network Management Protocol (SNMP) and NetFlow are two technologies available on some Cisco IOS platforms that can automate the collection statistics. These statistics can be used, for example, to establish a baseline in a troubleshooting scenario. Table 3-3 contrasts these two technologies.

Table 3-3  Comparing SNMP and NetFlow

Although both SNMP and NetFlow are useful for statistical data collection, they target different fundamental functions. For example, SNMP is primarily focused on device statistics, whereas NetFlow is primarily focused on traffic statistics.

SNMP

A device being managed by SNMP runs a process called an SNMP agent. A Network Management System (NMS) can then query the agent for information, using the SNMP protocol. SNMP version 3 (that is, SNMPv3) supports encryption and authentication of SNMP messages; however, the most popular SNMP version deployed today is SNMPv2c. SNMPv2c uses community strings for security. Specifically, for an NMS to be allowed to read data from a device running an SNMP agent, the NMS must be configured with a community string that matches the managed device’s read-only community string. For the NMS to change the information on the managed device, the NMS must be configured with a community string that matches the managed device’s read-write community string. To enhance the security available with SNMPv2c, you can create an access list that determines valid IP addresses or network addresses for NMS servers.

Figure 3-4 shows a topology using SNMP. In the topology, router R1 is running an SNMP agent that the NMS server can query.

Figure 3-4

SNMP Sample Topology

Example 3-20 illustrates the SNMP configuration on router R1. The snmp-server community string [ro | rw] commands specify a read-only (that is, ro) community string of CISCO and a read-write (that is, rw) community string of PRESS. Contact and location information for the device is also specified. Finally, notice the snmp-server ifindex persist command. This command ensures that the SNMP interface index stays consistent during data collection, even if the device is rebooted. This consistency is important when data is being collected for baselining purposes.

Example 3-20  SNMP Sample Configuration

R1# conf term
R1(config)# snmp-server community CISCO ro
R1(config)# snmp-server community PRESS rw
R1(config)# snmp-server contact demo@ciscopress.local
R1(config)# snmp-server location 3rd Floor of Wallace Building
R1(config)# snmp-server ifindex persist

NetFlow

Unlike SNMP, NetFlow can distinguish between different traffic flows. A flow is a series of packets, all of which have shared header information such as source and destination IP addresses, protocols numbers, port numbers, and Type of Service (TOS) field information. NetFlow can keep track of the number of packets and bytes observed in each flow. This information is stored in a flow cache. Flow information is removed from a flow cache if the flow is terminated, times out, or fills to capacity.

You can use the NetFlow feature as a standalone feature on an individual router. Such a standalone configuration might prove useful for troubleshooting because you can observe flows being created as packets enter a router. However, rather than using just a standalone implementation of NetFlow, entries in a router’s flow cache can be exported to a NetFlow collector prior to the entries expiring. After the NetFlow collector has received flow information over a period of time, analysis software running on the NetFlow collector can produce reports detailing traffic statistics.

Figure 3-5 shows a sample topology in which NetFlow is enabled on router R4, and a NetFlow collector is configured on a PC at IP address 192.168.1.50.

Figure 3-5

NetFlow Sample Topology

Example 3-21 illustrates the NetFlow configuration on router R4. Notice that the ip flow ingress command is issued for both the Fast Ethernet 0/0 and Fast Ethernet 0/1 interfaces. This ensures that all flows passing through the router, regardless of direction, can be mon­itored. Although not required, router R4 is configured to report its NetFlow information to a NetFlow collector at IP address 192.168.1.50. The ip flow-export source lo 0/0 command indicates that all communication between router R4 and the NetFlow collector will be via interface Loopback 0. A NetFlow version of 5 was specified. Note that although version 5 is the most widely deployed, you should check the documentation for your ­NetFlow collector software to confirm which version to configure. Finally, the ip flow-­export destination 192.168.1.50 5000 command is issued to specify that the NetFlow collector’s IP address is 192.168.1.50, and communication to the NetFlow collector should be done over UDP port 5000. Because NetFlow does not have a standardized port number, please check your NetFlow collector’s documentation when selecting a port.

Example 3-21  NetFlow Sample Configuration

R4# conf term
R4(config)# int fa 0/0
R4(config-if)# ip flow ingress
R4(config-if)# exit
R4(config)# int fa 0/1
R4(config-if)# ip flow ingress
R4(config-if)# exit
R4(config)# ip flow-export source lo 0
R4(config)# ip flow-export version 5
R4(config)# ip flow-export destination 192.168.1.50 5000
R4(conig)# end

Although an external NetFlow collector is valuable for longer-term flow analysis, you can issue the show ip cache flow command at a router’s CLI prompt to produce a summary of flow information, as shown in Example 3-22. A troubleshooter can look at the output displayed in Example 3-22 and be able to confirm, for example, that traffic is flowing between IP address 10.8.8.6 (a Cisco IP Phone) and 192.168.0.228 (a Cisco Unified Communications Manager server).

Both syslog (discussed in Chapter 1, “Introduction to Network Maintenance”) and SNMP (introduced in this section) are protocols that can report the occurrence of specific events on a network device. Although these protocols by themselves lack a mechanism to alert a network administrator (for example, via e-mail) when a network event is logged, third-party software is available that can selectively alert appropriate personnel when specific events are logged.

Earlier, this section discussed how a network device running an SNMP agent can be queried for information from an NMS. However, a network device running an SNMP agent can also initiate communication with an NMS. If an interface goes down, for example, the SNMP agent on a managed network device can send a message containing information about the interface state change to an NMS. Such notifications that a managed device sends to an NMS are called traps. These traps require an NMS to be interpreted because they are not in a readable format.

Example 3-22  Viewing NetFlow Information

R4# show ip cache flow
...OUTPUT OMITTED...
Protocol    Total   Flows   Packets Bytes   Packets Active(Sec) Idle(Sec)
----------  Flows   /Sec    /Flow   /Pkt    /Sec    /Flow   /Flow
TCP-Telnet  12  0.0 50  40  0.1 15.7    14.2
TCP-WWW 12  0.0 40  785 0.1 7.1 6.2
TCP-other   536 0.1 1   55  0.2 0.3 10.5
UDP-TFTP    225 0.0 4   59  0.1 11.9    15.4
UDP-other   122 0.0 114 284 3.0 15.9    15.4
ICMP    41  0.0 13  91  0.1 49.9    15.6
IP-other    1   0.0 389 60  0.0 1797.1  3.4
Total:  949 0.2 18  255 3.8 9.4 12.5

SrcIf   SrcIPaddress    DstIf   DstIPaddress    Pr  SrcP    DstP    Pkts
Fa0/0   10.3.3.1    Null    224.0.0.10  58  0000    0000    62
Fa0/1   10.8.8.6    Fa0/0   192.168.0.228   06  C2DB    07D0    2
Fa0/0   192.168.0.228   Fa0/1   10.8.8.6    06  07D0    C2DB    1
Fa0/0   192.168.1.50    Fa0/1   10.8.8.6    11  6002    6BD2    9166
Fa0/1   10.8.8.6    Fa0/0   192.168.1.50    11  6BD2    6002    9166
Fa0/0   10.1.1.2    Local   10.3.3.2    06  38F2    0017    438

Providing Notifications for Network Events

Whereas responding to problem reports from users is a reactive form of troubleshooting, monitoring network devices for significant events and responding to those events is a proactive form of troubleshooting. For example, before a user loses connectivity with the Internet, a router that is dual-homed to the Internet might report the event of one of its Internet connections going down. The redundant link can then be ­repaired, in response to the notification, thus resolving the problem without users ­being impacted.

Example 3-23 demonstrates how to enable a router to send SNMP traps to an NMS. The snmp-server host 192.168.1.50 version 2c CISCOPRESS command points router R4 to an SNMP server (that is, an NMS) at IP address 192.168.1.50. The SNMP server is configured for SNMP version 2c and a community string of CISCOPRESS. The snmp-server enable traps command attempts to enable all traps. Notice from the output, however, that a couple of the traps are mutually exclusive, causing one of the traps to not be enabled. You can view the enabled traps by using the show run | include traps command. Rather than enabling all possible traps with the snmp-server enable traps command, you can ­selectively enable traps using the snmp-server enable traps trap-name command.

Example 3-23  Enabling SNMP Traps

R4# conf term
R4(config)# snmp-server host 192.168.1.150 version 2c CISCOPRESS
R4(config)# snmp-server enable traps
% Cannot enable both sham-link state-change interface traps.                    
% New sham link interface trap not enabled.                                     
R4(config)# end
R4# show run | include traps
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vrrp
snmp-server enable traps ds1
snmp-server enable traps gatekeeper
snmp-server enable traps tty
snmp-server enable traps eigrp
snmp-server enable traps xgcp
snmp-server enable traps ds3
...OUTPUT OMITTED...

The messages received via syslog and SNMP are predefined within Cisco IOS. This large collection of predefined messages accommodates most any network management requirement; however, Cisco IOS supports a featured called EEM that allows you to create your own event definitions and specify custom responses to those events. An event can be defined based on syslog messages, SNMP traps, and entering specific Cisco IOS commands, as just a few examples. In response to a defined event, EEM can perform various actions, including sending an SNMP trap to an NMS, writing a log message to a syslog server, ­executing specified Cisco IOS commands, sending an e-mail to an appropriate party, or executing a tool command language (Tcl) script.

To illustrate the basic configuration steps involved in configuring EEM, consider Example 3-24. The purpose of the configuration is to create a syslog message, which will be displayed on the router console, when someone clears the router’s interface counters using the clear counters command. The message should remind the administrator to update the network documentation, listing the rationale for clearing the interface counters.

Example 3-24  EEM Sample Configuration

R4# conf term
R4(config)# event manager applet COUNTER-RESET
R4(config-applet)# event cli pattern “clear counters” sync no skip no occurs 1
R4(config-applet)# action A syslog priority informational msg “Please update
   network documentation to record why the counters were reset.”
R4(config-applet)# end

The event manager applet COUNTER-RESET command creates an EEM applet named COUNTER-RESET and enters applet configuration mode. The event command specifies what you are looking for in your custom-defined event. In this example, you are looking for the CLI command clear counters. Note that the clear counters command would be detected even if a shortcut (for example, cle co) were used. The sync no parameter says that the EEM policy will run asynchronously with the CLI command. Specifically, the EEM policy will not be executed before the CLI command executes. The skip no parameter says that the CLI command will not be skipped (that is, the CLI command will be executed). Finally, the occurs 1 parameter indicates that the EEM event is triggered by a single occurrence of the clear counters command being issued.

The action command is then entered to indicate what should be done in response to the defined event. In Example 3-24, the action is given a locally significant name of A and is assigned a syslog priority level of informational. The specific action to be taken is producing an informational message saying, “Please update network documentation to record why the counters were reset.”

To verify the operation of the EEM configuration presented in Example 3-24, the clear counters command is executed in Example 3-25. Notice that entering the clear counters command triggers the custom-defined event, resulting in generation of a syslog message reminding an administrator to document the reason he cleared the interface counters.

Example 3-25  Testing EEM Configuration

R4# clear counters
Clear “show interface” counters on all interfaces [confirm]
R4#
*Mar  3 08:41:00.582: %HA_EM-6-LOG: COUNTER-RESET: Please update network docu-
  mentation to record why the counters were reset.

Exam Preparation Tasks

Review All the Key Topics

Review the most important topics from inside the chapter, noted with the Key Topics icon in the outer margin of the page. Table 3-4 lists these key topics and the page numbers where each is found.

Table 3-4  Key Topics for Chapter 3

Complete Tables and Lists from Memory

Print a copy of Appendix B, “Memory Tables” (found on the CD), or at least the section for this chapter, and complete the tables and lists from memory. Appendix C, “Memory Tables Answer Key,” also on the CD, includes completed tables and lists to check your work.

Define Key Terms

Define the following key terms from this chapter, and check your answers in the Glossary:

input queue drops, output queue drops, input errors, output errors, SPAN, RSPAN, trap, NMS, NetFlow, EEM

Table 3-5  Chapter 3 Configuration Command Reference

Command Reference to Check Your Memory

This section includes the most important configuration and EXEC commands covered in this chapter. To determine how well you have memorized the commands as a side effect of your other studies, cover the left side of the table with a piece of paper; read the descriptions on the right side; and see whether you remember the command.

Table 3-6  Chapter 3 EXEC Command Reference

© Copyright Pearson Education. All rights reserved.