Does anyone out there remember the show, "The Fall Guy" with Lee Majors and one of my big time young punk kid crushes, Heather Thomas. The theme song was as popular as the actual show (especially the later seasons which sucked worse then a "Think Outside the Box" workshop). Looking at Cisco Cat switches and routers, to me the true "unknown stuntman" of networking is Netflow.
Netflow seems like a feature that only large networks and pre sales SE's use. Netflow is a large topic and an amazing traffic mining tool. A lot of that comes down to the massive amount of information that Netflow provides. Netflow will analyze traffic flows all across a network. I have used that data to track down bots, viruses, network intrusions, misconfig'ed QoS, etc. In short, Netflow is one of the single best reasons to purchase a Cisco switch. It is so awesome, that it is being used as the base for IETF standards based IPFIX.
That's not how I always thought. I was a big big big fan of HP's EASE/XRMON, especially with HP Traffic Probe. Now, XRMON is often some what incorrectly referred to as SFlow. Without getting into a real pissin' match here between the two, the issue to me with BOTH is that they are taxing to the CPU. If your interested, We did a video segment of the differences between the two on TechWiseTV. Personally, I think its hard to beat NetFlow v9. It's more accurate, has a bunch of 3rd party collectors and yes it runs in hardware. But you have to be all Cisco (or some Enterasys) not counting the IPFIX stuff.
So there are a ton of differences between the two, however, one major difference is the configuration options I have with Netflow. For example, sometimes Netflow gives me too much info and I just want to grab a sample of data for planning and traffic engineering. I can easily config this with the random netflow feature. Before we console in and start pluggin' away make sure you have a few things done first:
- You gotta have CEF or dCEF enabled
- Know your flow. Netflow versions 5 or 9 work if you need to export the data to look at it off device. Smokin' feature if you are using something like SolarWinds NetFlow Traffic Analyzer. With other versions you can view the data on the device.
- You can not have Netflow enabled on a interface your want to run random Netflow on. A device will always give full Netflow precedence over random Netflow.
- Different then full Netflow, enabling random Netflow on an interface does not enable it on a subinterface.
After all that stuff is all checked out and ready to go, lets go ahead and config this up. This awesome feature is config'ed up in two phases; the map phase and the interface phase.
Phase One: The Map Phase; Tellin' Netflow just how random to be.
This command sets up the custom name of the map in conf t mode:
TWTV2800(config)# flow-sampler-map TWTV
This command sets up the actual sampling rate 1 out of 1-65535:
TWTV2800(config-sampler)# mode random one-out-of 200
Save it
TWTV2800(config-sampler)# do wr me
Now on to the interface!
Phase Two: Put the sampler map on the interface to drive the behavior.
Go to your interface you want to enable this map for:
TWTV2800(config)# int eth 0/1
Apply the map:
TWTV2800(config-int)# flow-sampler TWTV
Save it:
TWTV2800(config-int)#do wr me
Check your handy work to make sure:
TWTV2800> sho flow-sampler TWTV
There is also a debug command to debug this as well: debug flow-sampler in case you run into any problems. I use this command all the time when I am at a customer site just wanting to grab a sampling of frames in a production environment. This is real attractive to folks because random Netflow puts a much smaller tug on the CPU, uses a smaller cache so the memory impact is reduced AND you also cut back on the data exploit per interface big time. Random Netflow is another tool to keep in your toolbox; Hey sing it with me!
'cause I'm the unknown feature that made Cisco such a star!!"
Jimmy Ray Purser
Trivia File Transfer Protocol
Nikola Tesla signed his name G.I. for Great Inventor. He could have signed A.K. for Amazing Kid. By the time he was five years old he invented a waterwheel and read the entire 100 volume set of the Complete Voltaire. Not to be out done, by the time I was five I could eat a half a jar of paste and 7 of 24 colors in a box of Crayons
Jimmy Ray Purser is the technical co-host for Cisco's TechWise and BizWise TV. Jimmy Ray also conducts advanced training for engineers across North America and Europe and regularly speaks at industry conferences such as VON, CeBIT, N+I, and Networkers. As a field engineer, Jimmy Ray experiences networking first hand behind the console or in the rack. He is an active member in the IEEE and the Ethernet Alliance and has designed, installed and tested numerous networks for Fortune 500 companies, the United States military and other institutions worldwide. He holds 3 U.S. patents for Ethernet security algorithms with two others pending and one defensive publication, as well as numerous other vendor certifications in networking and security.
Purser holds a Bachelor of Science degree in electrical engineering from Southern Illinois University is currently pursuing a master of science degree in electrical engineering.
The Leader in Network Knowledge
Solarwinds NetFlow - Competitors
Great stuff Jimmy Ray. Cisco NetFlow is fantastic. Since you are mentioning Solarwinds, other competing products include:
www.ntop.org
www.manageengine.com
www.plixer.com
www.fluke.com
thanks for the CLI details. That helps everyone!
Thanks
Reply
Awesome! Thank you for the additional links!
Jimmy Ray
NetFlow Solution Providers
While we appreciate the mention, I'd like provide the correct URL for Fluke Networks' NetFlow solution:
http://www.flukenetworks.com/fnet/en-us/products/NetFlow+Tracker/Overview.htm
NetFlow/sFlow and CPU load
Jimmy Ray,
Your comment about sFlow CPU load was surprising since the critical path (packet sampling) is implemented in hardware on most switches. The following paper from the Amsterdam Internet Exchange discusses their decision to use sFlow rather than NetFlow to monitor switches handling over 200 Gb/s. The paper includes information on CPU utilization when running at line rate, minimum packet size workloads:
http://events.ccc.de/congress/2006/Fahrplan/attachments/1137-sFlowPaper.pdf
The most recent data I could find on NetFlow CPU loads was:
http://www.cisco.com/en/US/technologies/tk543/tk812/technologies_white_paper0900aecd802a0eb9.pdf
Both papers are fairly old (2006 and 2007 respectively), however, the Amsterdam Internet Exchange continues to use sFlow for monitoring (as do many other Internet exchanges) and are now handling loads of over 780 Gb/s:
http://www.ams-ix.net/technical/stats/
FYI The latest Enterasys switches now support sFlow, along with switches from Brocade, Juniper, HP, Dell, IBM, Force10, Extreme, Hitachi, NEC and many others
http://www.sflow.org/products/network.php
Also, there are also a bunch of 3rd party sFlow collectors as well:
http://www.sflow.org/products/collectors.php
Finally, sFlow implemented by most L2-3 switch vendors and NetFlow by router vendors. The two technologies a quite different and for most networks, it makes sense to use both:
http://blog.sflow.com/2009/09/lan-and-wan.html
It's surprising that Cisco still doesn't support the sFlow standard on it's L2-3 switch platforms. Cisco customers still have to resort to SPAN port monitoring and probes to get visibility at the network edge.
Reply
Very nice detailed reply! Thank you for that! All things considered, SFlow should be a lower CPU load then NetFlow due to the design. My experience has been that not all SFlow hardware implementations are the same. SFlow on custom fabbed ASIC is normally very nice, but SFlow on OEM ASICs is not as deep. Some vendors can support full SFlow on all ports while others only recommend the uplinks.
You hit on the on former golden rule of embedded management; SFlow on the LAN, NetFlow on the WAN. I see that changing now now due to higher speeds and the need for 100% accuracy for billing and accounting. But in some case this rule will still apply in a mixed network.
I just do not see Cisco supporting SFlow (this is a real guess on my part) until customer demand is really huge for this. A commitment to SFlow means giving up precious ASIC space. To me that's like installing Windows 7 and then using LANtastic for networking. Folks are going to use whats there from the vendor they choose to do networking with.
I guess I am unsure about your final comment. I use flexible NetFlow at the edge all the time and it works great. Plus SFlow is not a standard it is classified as a Informational RFC not a Standard Track one.
I like SFlow and was a huge fan of SFlow when used with Inmon's Traffice Management solutions. It is good stuff without a doubt. After installing and messin' round with both and using the data out of both, I like NetFlow better because:
- It gives more more config options
- I have more full featured free collectors
- It has very detailed exportable records that can be mined by other applications off system
- For MPLS, it is hard to beat NetFlow
- NetFlow seems to be evolving more and more. SFlow has had some version upgrades but no multi-config options.
Thank you again for the awesome reply!
Jimmy Ray
1U switches and NetFlow
These commands don't work on 3560's or 3750's.
As an FYI, when the 3750 first was released, we were able to do Netflow v5 on the original code. All subsequent code releases don't have this feature.
Unless you have a work-around, we'll continue to deploy Nortel 5500/5600 series switches with Netflow v9 (Nortel calls it preipfixv9) free with code version 6.1 in our Server Farm. The 3750's will continue for the areas where we need BGP, and will get the NetFlow stats from edge routers.
S-Flow is like sampled Netflow v5, without the Templates v9 brings. Not as robust when trying to track all application flows before deploying a capture tool. It is really good for continuous traffic flows or streams without a lot of CPU overhead.
Dave R.
NetFlow & CPU
Isn't the NetFlow cache implemented in hardware in most Cisco switch platforms, leaving only the NetFlow export process for the main CPU? I thought this was the reason for all the controversy over NetFlow in the 6500/7600 series--namely, that the hardware NetFlow cache is too small for networks with very large numbers of unique flows.
neither are a standard?
I decided to look into Jimmy Rays comment about sFlow not being a standard. He seems to be right. http://www.apps.ietf.org/rfc/rfc3176.html to be fair, neither is NetFlow v9: http://www.ietf.org/rfc/rfc3954.txt they are both categorized as 'Informational'.
Someone please correct me on this. The folks at inmon are often stating the "sFlow standard".
http://blog.sflow.com/2009_10_01_archive.html Where is this document?
I found this searching the web "NetFlow provides visibility into routers while sFlow extends visibility into the increasingly important switching layer" look at the source: http://blog.sflow.com/2009/10/probes.html owned by inmon. Seems like they could be a bit more truthful.
Cisco and Enterasys make switches that support NetFlow. It seems that NetFlow is a highend gear technology. sFlow seems to be for less expensive switches.
sFlow standard
SFlow is developed and maintained by an industry consortium, as are many other widely supported standards:
Bluetooth, http://www.bluetooth.com/
USB, http://www.usb.org/
PCI, http://www.pcisig.com/
The sFlow.org web site publishes the sFlow standard and provides an open forum for equipment vendors, software developers and users to share information and propose enhancements.
There are currently 19 switch vendors listed that support sFlow:
http://www.sflow.org/products/network.php
and numerous free and commercial software applications for analyzing sFlow:
http://www.sflow.org/products/collectors.php
There are many full features, high-end products that support sFlow, including products from Juniper, Brocade, Force10 and Extreme Networks. It is true that sFlow is also available in more affordable products as well. sFlow was developed for end-to-end monitoring and was designed to scale to the highest end chassis devices and be cost effective enough for inexpensive stackable switches.
As you point out, Cisco continues to implement their proprietary NetFlow protocol rather than adopting the IETF standard (IPFIX) - in spite of their active involvement in the development of IPFIX. Other vendors also ship variants of NetFlow v5, 8 and 9, primarily on their routers. Many of the same vendors (including Enterasys) choose to implement sFlow on their L2/3 switches. I would be interested to hear of any commercial switches supporting the IPFIX standard - Nortel claims IPFIX support, but implements NetFlow 9.
It's been my experience that
It's been my experience that some switch vendors that implement sFlow intentionally try to confuse customers about its standardization by citing its RFC number (3176) without also pointing out that it is an informational RFC, rather than standards-track. With that rationale, Netflow v9 is also a "standard" since it too has an informational RFC (3954) and a host of companies that support it through its implementation in devices or processing of its exported data.
There is one dominant, open standards organization for networking layers 3-7: the IETF. It selected Netflow v9 as the basis for its IPFIX standards-track RFC suite. Netflow vendors (including Cisco) are currently working on implementing IPFIX, but even small changes to code take time to work their way into commercial products. It is disingenuous to say that they are not adopting IPFIX.
Post new comment