Using BGP to Make Our Internet Access Dynamic

In my last blog post, I went over how we made BGP our core routing protocol. This change has been a fantastic change for us providing stability and scalability. In that blog, I went over the core BGP architecture we use, but, as I mentioned, that just touched the surface of what we have done. One of the other significant enhancements we have done is dynamic Internet routing using our BGP backbone. This has allowed us to consolidate Internet access to only a few large sites, while providing dynamic failover between those large sites. As I mentioned in the last blog, our core routers know all routes in the network and we inject a default route into OSPF to bring LAN traffic toward the core. This traffic includes Internet traffic that needs to continue following the default route. At our Internet border we have either one or two Internet circuits. If there are two circuits, they are from different ISP. From the ISPs, if there are two circuits, we accept a full Internet routing table plus a default route. By accepting a full Internet routing table we are able to make proper outbound routing decisions to the Internet. If there is only a single circuit we accept only a default from the ISP. This default route is then passed via eBGP multi-hop to the internal core routers. The eBGP multi-hop session flows from the Internet routers to the internal core routers through the firewalls. The firewalls do not run BGP, just an IGP to facilitate the eBGP session between the core routers and Internet routers. To ensure the full Internet routing table does not get mistakenly dump to the internal core routers, filters are placed on both ends of this eBGP session. Now that the internal core routers have the default route, they send that route via iBGP to the WAN routers. The WAN routers then advertise the default route to our MPLS providers via eBGP (just like all other routes are advertised). The MPLS providers receive the default route from several of our hub sites. Since the BGP AS path length on the default route is equal, the MPLS providers simply pick the best default route based on their own backbone IGP costs. Thus, sites on the East Coast get Internet access from our East Coast hub site. Sites on the West Coast get Internet access from our West Coast hub site. Same for EMEA. Same for Asia. All dynamically chosen by the MPLS providers based on BGP. Failover is dynamic also. If we have a site with a single Internet circuit and that circuit goes down the Internet router no longer has a default route to advertise to the internal core routers. So, BGP removes the default. This change propagates through BGP to the MPLS providers. The carrier's MPLS backbone picks a default route from another hub site, reconverges, and Internet traffic flows to the other hub site. Now the local site whose single Internet circuit is down gets Internet access from another hub site. This reconvergence takes about 10 seconds. Users hardly notice the change. Furthermore, any field sites that were using this hub site for Internet access also reconverge to another hub site. This can happen globally also. For example, EMEA sites can get Internet access from the US if needed. None of this could have been done without BGP. Its scalability, openness, and route policy features gave us the tools needed to implement this design. Consolidating and making our Internet access dynamic cut ISP circuit costs by 61%.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2007 IDG Communications, Inc.

SD-WAN buyers guide: Key questions to ask vendors (and yourself)