Is BUN Chewing Up Your LAN?

One of the most interesting, and least known problems that can occur on a LAN is Broadcast-Unicast (BUN) flooding. And, it can cause a lot more problems than many other typical LAN issues (like normal broadcasts, spanning-tree recalcs, etc). To put it simply, BUN occurs when a switch doesn’t know a destination MAC address for a unicast frame so it turns the frame into a broadcast. The switch at least can ensure that the traffic gets to the destination since it’s a broadcast now. But the problem is every other device on the VLAN sees the frame too. That can be a lot of frames. Imagine a user is downloading an ISO images (couple gigabytes) and every frame is being turned into a broadcast on the LAN. Every host on the VLAN would have to look at every frame and discard it. That’s a lot of frames. That’s wasted CPU cycles and wasted port bandwidth. Let’s look at a BUN situation. BUN often occurs with asymmetric routing – or, in a typical LAN, what I like to call load balanced routing. Take the situation below.

In this case you have two access switches connected to a core. Layer-3 routing is extended to the access layer, but the access switches are L2 adjacent. So, the host VLAN, where the two PCs are connected, is the same on both switches; a single subnet. Now, let’s say one of the PCs starts a large download. The traffic flows are as follows:

This flow is completely normal. The PC sends its requests out the local switch, which would be the HSRP active router for the VLAN, but the return traffic comes down the ECMP link to AS01. AS01 then sends the traffic over the L2 link to AS02 and then to the PC. Works fine. But here’s where BUN comes in. For the first frame that AS01 receives, AS01 ARPs for the PC’s MAC address. The PC replies and AS01 updates its ARP table (4 hour timeout) and its CAM table (5 minute timeout) with the appropriate entries. Traffic flows correctly – i.e. no BUN – for five minutes. But after five minutes, the CAM table times out. However, the ARP entry is still fine so AS01 does not re-ARP. When a packet arrives, the L3 engine in AS01 (maybe a MSFC card) formats the frame with the PC’s destination MAC address and passes the frame to the L2 engine for switching. But, the L2 engine no longer has the PC’s destination MAC address in the CAM table and has no way on its own to get the MAC address, so the L2 engine turns the frame into a broadcast and sends it to everyone. The PC gets its frame, but so does every other PC on the VLAN. Bandwidth is being chewed up and CPU cycles wasted. This lasts for another 3 hours, 55 minutes until AS01’s ARP table times out and AS01 re-ARPs. Then it works fine for a whole 5 minutes, and then BUN comes back. Now take this problem and multiple it for every host on the network and divide by two since half the traffic will probably not flow asynchronously. That’s a lot of wasted bandwidth! And a lot of wasted CPU cycles on PCs! A simple way to see if you are suffering from this problem is to buy a cheap Linksys switch with blinky lights.

Plug it into a LAN port on one port and your PC into another. Then do not do anything on the PC (i.e. no network traffic). If the lights on the Linksys switch are blinking constantly, you’ve got BUN (that sounds like a bad disease). In a normal switched LAN, nothing should be coming down your port to the Linksys unless it’s (1) traffic destined to your PC or (2) a broadcast. If there is a lot of #2, it’s probably not regular broadcasts, it’s probably BUN. There are a couple tricks to fix BUN. You can play with L3 routing costs to remove the asynchronous routing, but that is messy. It’s changing a L3 protocol to fix a L2 issue. Plus, you lose half of your uplink bandwidth and force all traffic up and down a single link. A better way is to change some timers. The trick is to prevent the CAM table from expiring. So, either lower the ARP timer to below 5 minutes or raise the CAM timer to above 4 hours. We’ve found that in a static LAN environment – like a data center – it’s better to raise the CAM table above the default ARP timer. Devices don’t change or go down very often in a data center, so raising the CAM timer doesn’t affect convergence. However, in a user LAN – where there are many changes as hosts enter and leave – we lower the ARP table to four minutes, below the CAM timer. Lower timers are appropriate in a dynamic environment so entries can timeout quickly and not get stuck. Now get out there with your Linksys, look for blinky lights, and fight the BUN.

More >From the Field blog entries:

The 2008 NPA Award Winner

It’s One of Those Opinionated Days

So What’s This About the Internet Melting Down?

A Great Week at FutureNet

Clearing the Air on the CCDE Practical Beta Exam

CCDE Practical Beta Test Cancelled

Quick Thoughts on the New Nexus 5000

  Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.