When In Doubt, Capture Packets

The New Embedded Packet Capture Feature

While NetFlow--the subject of my last three posts--is a tool that I use constantly for getting visibility into network traffic, sometimes you need to look at complete packet contents. I have a saying at work: "When in doubt, capture packets." The first few times (or maybe few dozen times) you troubleshoot a problem by using raw packet captures, it'll be confusing and time consuming. As you learn and better understand network fundamentals, though, it'll eventually become one of your favorite tools, and you'll be able to quickly troubleshoot crazy problems that would've otherwise left you scratching your head and randomly rebooting things.

The first step in using raw packet capture in network troubleshooting is an understanding of basic networking: the OSI model, Ethernet, ARP, IP, TCP/UDP, etc. Cisco's CCENT and CCNA curriculum has some good basic material in this area. I also recommend the two classic books in this field: Internetworking with TCP/IP and TCP/IP Illustrated Volume 1. And although it's dated, I still think Charles Spurgeon's Ethernet: The Definitive Guide is a nice reference. The author maintains a list of Ethernet references online, too.

At the same time as you're studying theory, you need to be getting hands on practice with reading packet captures. The standard tool for that these days is Wireshark, which is free. I'd recommend getting a copy of Laura Chappell's excellent new Wireshark book to go along with it--it'll save you a lot of time and frustration.

Finally, you need to be able to actually get raw packet captures off your network. Most people are familiar with the idea of port mirroring, or "SPAN" in Cisco terminology: traffic on one switch port is copied to another switch port on which a packet capture device resides (such as a PC running Wireshark, IDS, forensic appliance, etc.). There are several variations on the SPAN feature: you can copy not just a source port, but a source VLAN; you can copy traffic to a destination VLAN with a remote capture device (this is called RSPAN), or on some platforms you can even tunnel the mirrored traffic across a layer 3 boundary (ERSPAN). SPAN is an essential tool for any network engineer, but it does have its disadvantages: 1) you can easily oversubscribe the destination port and drop packets, 2) some configurations result in capturing duplicate packets, which can be confusing, and 3) it can be a challenge to get SPAN configuration changes through the change management process quickly. For this reason, many designers prefer to use network taps or specialized matrix switches to facilitate captures. Furthermore, the SPAN feature set is typically available only on switch hardware platforms, or on switch modules in IOS routers.

For this reason, I was excited to see the Embedded Packet Capture feature in IOS 12.4(20)T and later. With this feature, you can take a packet capture directly off the router, and pre-filter it with an ACL to get exactly what you want. While this isn't particularly valuable for full-time capture requirements like we find in the security world, it's extremely valuable for network troubleshooting where we just need a bit of extra visibility into what's happening at the packet level.

The feature is somewhat complex, with many tuning options. We'll take a look at a basic example here. The two major configuration objects that you need to set up are the capture buffer and the capture point. The capture buffer holds the captured packets. You need to configure a name, packet size, buffer size, and typically an access-list to select only the packets that interest you. The capture point specifies the interface on which the packets will be captured, and whether they'll be captured out of the CEF path or the process-switched path. For normal data plane packets, use the CEF path. For packets that are destined to the router itself, use the process-switched path.

Here's an example of a EPC configuration that will give us some routing protocol packets. This example was taken from a router running IOS 15.0(1)M. First we define the buffer:

TEST#monitor capture buffer TEST_BUFFER max-size 1600
TEST#monitor capture buffer TEST_BUFFER size 512
TEST#monitor capture buffer filter access-list MY_SPECIAL_TRAFFIC

The "max-size" parameter defines the number of packet bytes to be captured. For some reason, the default is 68 bytes--so unless you want just the packet headers, you'll want to set this to something at least as big as the MTU of your interface. The "size" parameter defines the size of the capture buffer itself, in kilobytes. The default in this image is 1024K; I set it to 512K here just to show an example. Finally, the "filter access-list" parameter filters the buffer based on an ACL of your choice.

The next command defines the switching path, interface, and direction of the capture as a named capture point. Here's an example of what it would look like for the process-switched path:

TEST#monitor capture point ip process-switched PROC_CP both

and here's an example of a CEF path capture in both directions on Fa0/0:

TEST#monitor capture point ip cef CEF_CP Fa0/0 both

Next, we associate the capture point with the buffer and start the capture:

TEST#monitor capture point associate PROC_CP TEST_BUFFER
TEST#monitor capture point start PROC_CP

After a while we can view statistics about the buffer:

TEST#sh monitor capture buffer all parameters
Capture buffer TEST_BUFFER (linear buffer)
Buffer Size : 524288 bytes, Max Element Size : 1024 bytes, Packets : 218
Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
Associated Capture Points:
Name : PROC_CP, Status : Active
Configuration:
monitor capture buffer TEST_BUFFER size 512 max-size 1024
monitor capture point associate PROC_CP TEST_BUFFER

When we think we have the traffic we need, we stop the capture and export the buffer as a .pcap file, for viewing in Wireshark:

TEST#monitor capture point stop all
TEST#monitor capture buffer TEST_BUFFER export tftp://10.77.18.101/testBuf.pcap
*Apr 22 22:06:37.435: %BUFCAP-6-DISABLE: Capture Point PROC_CP disabled.
TEST#monitor capture buffer TEST_BUFFER export tftp://10.30.18.101/testBuf.pcap
!
TEST#

Unfortunately, there's currently no way to decode the packet contents from the CLI. All you can do is get some rudimentary information with the "show monitor capture buffer" command, or you can get a raw hex dump with the "dump" argument:

R4#sh monitor capture buffer TEST_BUFFER
18:03:04.563 UTC Apr 26 2010 : IPv4 LES CEF    : Fa0/0 None

18:03:05.339 UTC Apr 26 2010 : IPv4 LES CEF    : Fa0/0 None

18:03:09.355 UTC Apr 26 2010 : IPv4 LES CEF    : Fa0/0 None

R4#sh monitor capture buffer TEST_BUFFER dump
18:03:04.563 UTC Apr 26 2010 : IPv4 LES CEF    : Fa0/0 None

494EA8F0:                                            01005E00              ..^.
494EA900: 000A000F 8F08DAD7 080045C0 003C0000  ......ZW..E@.<..
494EA910: 00000158 CD980A01 0107E000 000A0205  ...XM.....`.....
494EA920: EECB0000 00000000 00000000 00000000  nK..............
494EA930: 00010001 000C0100 01000000 000F0004  ................
494EA940: 00080C04 010200                      .......

In theory, I think it should be possible to write a TCL script to parse the contents of a capture buffer into something more human readable, but that's wellbeyond my scripting skill level. Maybe we have some enterprising programmers in the audience?

Despite the requirement to export the capture buffer for offline viewing, I've already found this to be an invaluable tool in the relatively short time it's been available. I'm looking forward to the day when all my production routers are running code that has the EPC feature set.

I'll be attending the Rocky Mountain IPv6 Summit later this week, so on the off chance that any readers will be present, be sure to look me up!

• Cisco