Network Packet Monitoring Matrix Switches

New devices can eliminate limitations of using taps and SPAN ports

Have you ever wanted to configure more than two TX/RX/Both Switched Port Analyzer (SPAN) ports on your switches? Have you ever had to deal with the issues related to optical-bypass switches for inline IPS appliances? Have you ever had to deal with the cabling nightmare that results from using Ethernet taps? Have you ever thought about using packet monitoring switches that are available from a variety of manufacturers to save you headaches?

Limitations of SPAN Ports

I have often wondered why manufacturers of Ethernet switches severely restrict the number of SPAN/port-monitoring sessions they allow to be configured. I can only assume that they are limited because of the extra burden this copied traffic places on the emaciated CPUs in the LAN switch. SPAN ports have been known to add to the overall CPU load of the network device. Switches do not have very powerful network processors that are good at performing network management and other packet processing. The process of "forking" packets and sending them to their destination and a monitoring port takes CPU resources. This may be why manufacturers want to limit the number of SPAN sessions. The volume of traffic also has a lot to do with the potential impact on the CPU.

Another issue with SPAN ports is that the SPAN data is treated with a lower priority than regular traffic on the Ethernet switch. Therefore, during times of congestion, it is likely that SPAN traffic will be arbitrarily dropped and not make it to the monitoring device. This is also true when sending SPAN traffic through an 802.1Q trunk along with regular client/server traffic. The SPAN traffic is also treated with a lower priority on the trunk port.

The key issue related to network monitoring scalability is the limitations on the number of SPAN/port monitor sessions that are configurable on Cisco Ethernet switches. Cisco 6500 switches can have up to four transmit-only (Tx, half-duplex) SPAN sessions configured. The problem with Tx-only SPAN sessions is that the IPS will only observe traffic flowing in one direction. These types of SPAN sessions are typically configured to detect malicious packets flowing from a less-trusted network like the Internet to a more trusted network like a server network. Cisco 6500 switches can have up to two Tx/Rx (full-duplex or both) SPAN sessions configured on them. These sessions allow for the capture and analysis of packets flowing in both directions to and from the attacker. This allows for more precise determination of malicious traffic and attacks. This link provides more information about how to configure SPAN sessions along with the limitations of SPAN sessions on Cisco switches. Cisco has also published information on the smaller workgroup and stackable Cisco switches (Catalyst 2940, 2950, 2955, 2960, 2970, 3550, 3560, 3560-E, 3750 and 3750-E switches) the following limitations apply.

Virtual Switching System (VSS) can also provide some advantages for performing SPAN/port-monitoring for data traffic analysis. A monitoring system can be connected to either VSS 6500 switch yet be able to monitor ports on either 6500 switch. In a Virtual Switch domain, the number of SPAN sessions is still limited by what the Virtual Switch active supervisor can provide.

Many organizations have currently maximized the number of SPAN/port-monitor sessions available on their Ethernet switches. In other words, most companies do not have any additional capacity to monitor any other traffic flows on their core switches or DMZ switches. This is a problem if any new security or monitoring system needs to be deployed that requires a view into traffic paths flowing through the LAN switches.

Companies additional requirements for SPAN ports on the core switches. Their IPS appliances, web application firewalls, and web content filtering systems need to be able to observe traffic to and from the Internet to help prevent against security attacks. Currently, these content filtering and deep-packet-inspection (DPI) systems are deployed on internal LANs to help protect the desktop computers used by end-users. The content filtering functions also need to capture packets and observe traffic patterns and also need to use a SPAN port.

Switch features such as SPAN (RSPAN, ERSPAN, etc.) can be beneficial as well; however these have limitations (such as the number of active sessions at any single point in time) and can place an unnecessary burden on the switch that might best be handled by a dedicated device. On Cisco 6500s with Supervisor Engine 720s two RSPAN source sessions can be configured. However, many RSPAN destinations can be configured. That means that a single source can be spanned and the traffic can be sent to multiple security systems that are interested in observing that traffic from that active network.

Using Hubs

Historically, hubs (multi-port repeaters) were used for performing network monitoring. Hubs forwarded all traffic on all ports to all other ports in a half-duplex method. All the data traffic was repeated to all ports so any port could easily observe any traffic for any device connected to the hub. Hubs create a collision domain that reduces the efficiency of the network. However, most of these hubs operated at either 10Mbps or 100Mbps Ethernet speeds. Ethernet switches reduce the "collision domain" and have assumed more popularity over hubs and therefore, there are few manufacturers of hubs today. Ethernet switches maintain a table of Ethernet MAC addresses connected to each port and only forward traffic destined to those ports directly. This creates efficiency because all nodes connected to the switch don't see all the other traffic on the switch and only get packets that they are intended to receive. Therefore, it can be difficult to find a hub and nearly impossible to find a hub that operates at 1Gbps.

Hardware Taps

An alternative to SPAN ports and hubs are hardware taps. Network taps can be very beneficial when monitoring network traffic. A network tap is a hardware device which provides a way to access the data flowing across a computer network. In many cases, it is desirable for a third party to monitor the network traffic between two points in the network, point A and point B. If the network between points A and B consists of a physical cable, a network tap may be the best way to accomplish this monitoring. The network tap has at least three ports: an A port, a B port, and a monitor port. To place a tap between points A and B, the network cable between point A and point B is replaced with a pair of cables, one going to the tap's A port, one going to the tap's B port. The tap passes through all traffic between A and B, so A and B still think they are connected to each other, but the tap also copies the traffic between A and B to its monitor port, enabling a third party to listen. There are several brands of network taps that are popular. NetOptics is probably the most popular manufacturer of network taps. Gigamon and Finisar are other companies that make optical taps.

An advantage of using network taps are that taps do not alter the time relationships of frames, spacing and response times especially important with VoIP and other application analysis including full-duplex analysis. Taps also do not introduce any additional jitter or distortion which is important in application analysis. Taps do not groom data nor filter out physical layer erred packets, short/long frames, frames with bad CRCs. Taps do not drop packets due to congestion. Taps are not IP-addressable devices so they are not susceptible to security attacks. Taps are easy to setup and do not require a network engineer to configure a SPAN port each time they need to be used. Taps also work for IPv4 and IPv6 traffic; currently Cisco SPAN sessions don't capture IPv6 packets.

There's no silver bullet in terms of where to place network taps. Often times taps are used to provide a copy of traffic to a security device (such as a content inspection device or intrusion prevention system, etc.). These devices are typically required due to regulatory or organization-specific requirements and/or policies. Because of this, organization policies should dictate the placement of network taps throughout the environment. The challenges with using network taps are that they can be problematic due to the cabling issues they create. Multiple cables, if not labeled and documented well, can add to troubleshooting and MTTR times. Companies often times need to monitor both the active and the redundant connection so taps create a difficulty for monitoring all possible traffic paths.

In-line IPSs and IPSs that are embedded in other systems are better because they have reduced cabling which reduces the number of components along a transmission path that can fail. IPS systems that are embedded into other appliances allow for the deep packet inspection but don't require other active components, separate configuration, separate software licensing, and thus reduced TCO and improved overall network reliability.

RMON Probes

An alternative to maximizing the use of SPAN ports is to move network test equipment to using taps or probes. Network troubleshooting exercises happen in an ad-hoc way so they don't need to be set up all the time. However, the security systems need to be actively observing traffic 24X7. Other alternatives involve using RMON probes for network troubleshooting. RMON allows for full packet capture. Another solution would be for organizations to rely on using NetFlow more often for higher-level analysis of traffic patterns and network and application troubleshooting.

Cisco Mini Protocol Analyzer (MPA)

Another option for specific network troubleshooting activities on a Cisco router or 6500 switch is to use the built-in packet capture function. The Mini Protocol Analyzer (MPA) function provides the ability to capture packets on an IOS router (12.4(20T)+) or a Cisco 6500 switch (12.2(33)SXI+) that can come in handy. This may not be a permanent solution for performing IPS functionality, but it can provide a function to perform network troubleshooting when all the Tx/Rx/Both SPAN sessions are being used for IPS functions. The following links provides information on using the Mini Protocol Analyzer.


Another alternative to the way that companies can monitor traffic is to use the VLAN ACL (VACL) Capture Port feature. This method uses an access-list applied to a VLAN interface on the 6500 and the destination for the traffic that matches the ACL is sent to a capture port. In this way, the VACLs can perform network traffic analysis in a more granular manner by only capturing specific traffic of interest rather than all other traffic on the VLAN. This method works, but it is still limited to the total number of SPAN sessions configurable on the 6500 switch. Below is a link to how VACL captures can be configured.

Packet Monitoring Matrix Switches

Another solution is to use a device that is specially designed to allow for many taps to be aggregated and also connect to all the monitoring systems. These switches connect network monitoring devices to the points on the network where you want to be able to observe packets. Below is a diagram that shows how the matrix switch is connected to all the different points in the network environment where packet-level monitoring needs to take place. SPAN ports can be configured on the network switches to send much of their traffic to the matrix switch. The matrix switch also connects all the tools that are used to monitor the network. This would include IPS devices, packet sniffers for ad-hoc troubleshooting, RMON probes, or Web Application Firewalls.

Matrix Switch

The packet monitoring matrix switch then has a configuration interface that allows the network administrator to send traffic from the input ports connected to the network devices to the output ports connected to the various tools. Packets can be duplicated from a single input port to multiple output ports when multiple tools need to see the same types of traffic. Most matrix switches have the ability to create complex filters to send only the desired types of traffic from the input ports to the output ports so the network monitoring tools only see the desired traffic types. Matrix switches can also have role-based configuration management so that only specific staff can modify the monitoring for specific purposes. For example, the security teams would have the ability to modify IPS port monitoring configurations and the network administrators would have the ability to modify the packet sniffers, but they couldn't modify each other's monitoring. Most matrix switches also come with a variety of interface types for 1Gbps and 10Gbps interfaces and can throttle the packets being forwarded across ports.

1 2 Page 1
Must read: 10 new UI features coming to Windows 10