Using BGP to Build a Separate Lab Network

I've been on the BGP bandwagon with my last two blog entries: - Making BGP Our Core Enterprise Routing Protocol - Using BGP to Make Our Internet Access Dynamic So, I thought I'd finish up today with a project we recently finished that relies on a complex, but very cool, BGP design. Being at a high-tech company we have many software labs. Development engineers use these labs, some of which include thousands of servers, storage and network gear, to develop and test new products. These labs are located around the world and the software engineers often like to blast data between these labs. This put a strain on the corporate WAN. QoS protects business applications and VoIP, but there's only so much bandwidth. Someone would be impacted and it was often lab traffic since it was in the bottom queue. This slowed product development, increased help desk tickets, and angered the software engineers (who always want more and more bandwidth). Plus, a couple times internal lab tests were incorrectly configured and traffic routed into the corporate network overloading resources and isolating sites. WAN routers can't QoS traffic from 30 miss-configured blade servers sending 3 Gbps of traffic toward the WAN. So, we embarked on a project to build a separate WAN for the labs. Among the requirements, two stood out: 1. Any traffic, whether sourced or destined to a lab, should route over the new Lab WAN. 2. Failover for the Lab WAN should be provided by the corporate WAN, but not vice-versa. The first requirement was the most interesting. For this requirement, any traffic to or from a lab had to route over the new Lab WAN. So, we could have a user at his desktop, on the IT network, communicating with a remote lab. Traffic from this user's PC, which in on the IT network, needed to route into the Lab Network and over the Lab WAN. But, don't forget requirement #2. If the Lab WAN circuit at the site was down, this traffic needed to route over the corporate WAN as backup. Once you draw this out, it gets tricky. Enter BGP. I used two key features in BGP to route traffic correctly, without using any policy routing. First, let's consider getting our IT user's traffic to the Lab Network. As I mentioned in my previous blog, our core routing protocol is BGP. So, we simply extended our BGP design by putting each lab in its own BGP autonomous system and connecting the lab to the IT network via eBGP. Each lab was assigned an AS number from a specific, private AS range - 64600-64799. By having a specific AS number range for the labs, I could identify the routes that originated from a lab. Using a BGP AS filter list inside a route-map, I assigned a higher BGP local preference to lab routes. Now, the IT core would see lab routes from the IT WAN with a lower local preference than from the eBGP connection to the labs and would route traffic from an IT user into the lab. The lab network then routed this traffic over its new Lab WAN circuit. If the Lab WAN circuit goes down, BGP reconverges and uses the same route, with a lower local preference, from the IT WAN. But, what about traffic from the lab back to this IT user? For this situation we needed to send traffic to local users directly into the local IT network and all other traffic over the Lab WAN. But, we still needed backup for non-local traffic via the local IT network should the Lab WAN circuit go down. To achieve this, I used BGP AS pre-pending on routes advertised from the local IT network to the labs. The IT core would send all routes into the Lab network via eBGP, but would AS prepend all non-local routes with 3 extra AS paths. The lab would learn the same subnets via the Lab WAN circuit, but with the normal AS path length. Thus, the Lab WAN was more preferred. Should the local Lab WAN circuit go down, traffic would still have a backup route via the IT WAN; the routes just have a much longer BGP AS path. This BGP design met both requirements listed above. The new Lab WAN has significantly reduced the load on the IT WAN and provides dedicated bandwidth the software engineers. A great solution made possible by BGP.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2007 IDG Communications, Inc.

SD-WAN buyers guide: Key questions to ask vendors (and yourself)