When it comes to troubleshooting and threat detection, NetFlow wins over packet capture

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.

With Internet connections to cloud services growing rapidly and cyber attacks becoming craftier and more sinister, the need for improved traffic visibility is in high demand.  In the past, both layer 7 application awareness and malware detection capabilities have been major separators when choosing between flow capture and packet capture for traffic analysis, but today the decision is most often NetFlow in lieu of packet capture.

Until the release of NetFlow v9, flows were limited to roughly 20 common fields.  The rest of the packet contents were discarded. NetFlow v9 can be used to export any details found within a packet including the entire datagram.  What makes flow technologies attractive over the raw packets is the ability to turn existing routers, switches and servers into distributed collection points.

Shortly after the introduction of NetFlow v9, router and firewall vendors started performing Deep Packet Inspection (DPI) to identify the applications hiding on ports such as TCP port 80. After DPI identifies the correct application, this additional information is included in the NetFlow export.  Prior to DPI and the introduction of NetFlow v9, identifying the correct Layer 7 application (e.g., Skype, Citrix, Facebook) was not possible.  Today several vendors including Cisco, Dell-SonicWALL, Palo Alto and nBox all provide layer 7 visibility in their flow exports.  

Because of the enormous amount of detail available in today’s flow exports, major router and firewall vendors have made packet capture less necessary.  Now vendors are moving away from NetFlow v9 and as of July 11th, 2013 a standard for NetFlow has been accepted by the IETF which is called IPFIX.  Several vendors are now supporting both NetFlow v9 and IPFIX.

In a 2012 analysis the Gartner group concluded that flow analysis should be done 80% of the time and packet capture with probes should be done 20% of the time.  But some vendors also include flow details such as round trip time, packet loss, packet size, retransmits, jitter, HTTP host, URL and much more. Even packet sampling like sFlow is possible with NetFlow and IPFIX.  These details allow network analysts to follow a flow and observe the hop by hop performance of a connection.  Isolating exactly where a problem was introduced in the path (say packet loss) becomes much easier when the quality of the flow can be mapped out end to end. This is all done for NetFlow or IPFIX, not packet analysis.

Given that, the percentage of time flow analysis should be used might approach 90% or more.  This is especially true when you consider the cyber threat detection usefulness of flow technologies.

Threat Detection with NetFlow

Cyber threat detection with flow technologies focuses primarily on behavior monitoring.  Rather than performing Deep Packet Inspection (DPI) like a firewall and triggering events based on a single isolated signature match, behavior monitoring watches for odd behaviors over time.  Odd behaviors trigger small increases in an index which can increase and decrease over time.  If the index for any one host rises too fast and breaches a configurable threshold, action can be taken.  Sophisticated malware is often identified with behavior monitoring - firewalls don’t provide this intelligence.  

Behavior monitoring carefully watches flows to and from every host on the network and  a NetFlow analyzer with threat detection provides security administrators with an additional mechanism (i.e. the index) for identifying hosts infected with malware.

Example threat detection with NetFlow methods include:

• Host reputation lookups

• Observation of TCP flags to uncover various types of network scans

• Comparing current behaviors to baselines

• Calculating flow ratios as well as byte/packet counts to unique destinations

The above algorithms can carry different weights when it comes to severity.  A host found to be violating one or more algorithms will end up with a higher index.

Since NetFlow and IPFIX are readily available on most enterprise networks, visibility into all corners of the network is easily attainable.  Gaining the same visibility and awareness using packet probes is simply cost prohibitive in most environments. This is true both because of the initial cost of the probe and the man hours necessary to maintain the deployed hardware. In comparison, flow technologies provide a tremendous value in threat detection, and provide huge benefits as the “go to” solution when a potential threat needs to be investigated.  

Imagine every flow-exporting device to be a network security camera – much like a department store with security cameras mounted on the ceiling in dozens of locations scattered around the store. If the store’s security team is suspicious of a patron, they will turn to the cameras first to monitor the individual real-time as they move around the aisles.  They may also look at past video footage to observe the individual prior to becoming a suspect.  

So even if the shop lifter was caught by an observing employee rather than someone watching the security cameras, the past footage is almost always part of the investigation.  The same holds true when archiving NetFlow and IPFIX data for future reference.  Flow technology is the number one solution for most organizations that investigate application performance and potential security issues.  Every router, switch, server and firewall that is flow capable is considered a security camera – and evidence.

The need for capturing packets isn’t going away however, with the improved insight provided by flow technologies the demand is certainly shrinking.  Flow technologies not only cover more areas of the network, they are also easier for collectors to aggregate in order to display top reports on hosts, applications, protocols, interfaces, and more.   Capturing packets is largely an engineering practice that is best left to the developers of the application who need to fine-tune specific areas of the code. To be clear, packet capture isn’t going away rather, in most cases the problem can be determined with flow data eliminating the need to be at the physical location to plug in a probe.  

It’s time to consider NetFlow v9 and IPFIX -- most probably your existing hardware supports it.  Cisco routers for example, support Application Visibility and Control (AVC) which provide all of the details discussed here.  Although vendor implementations vary in terms of the metrics they export, failing to collect flow data is probably preventing your IT team from minimizing the Mean Time To Know (MTTK) of application performance issues and cyber threats that get right past the firewall.

Is that a risk worth taking?  

Plixer International (http://www.plixer.com/) is a leading provider of Scrutinizer NetFlow-based network traffic reporting and detection solution, and the author of the newly released book Unleashing the Power of NetFlow and IPFIX.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.