Making BGP Our Core Enterprise Routing Protocol

Over his last few blog posts, Jeff Doyle has been extolling the virtues of BGP and providing design tips. While I don't portend to have Jeff's knowledge about BGP, I have always been a huge fan of the protocol. My affection for BGP has grown in the last few years as we have used it as the core routing protocol for our internal, enterprise network.

On the global enterprise networks I worked on earlier in my career, BGP was used to interconnect regional IGP autonomous systems. eBGP was used on the few large, expensive, international circuits that connected the regions. Inside the region an IGP was used for in-country routing - either OSPF or EIGRP. Sometimes there was iBGP, but there were often synchronization problems that led to routing loops. Often eBGP was just redistributed into the IGP, but that was incredibly messy. This was the after affects of hastily done network conversions. It was easier to redistribute than build a proper iBGP network inside each region that was supported by the IGP.

Over the last three years, MPLS (RFC 2547bis) has exploded into enterprise networks. While many carriers will support OSPF, RIP (gasp!), or EIGRP on links to customers, none do it enthusiastically. Some require special engineering approval. Since their MPLS backbones are based on BGP, they want enterprise customer to use BGP.

Considering what I had seen earlier in my career and the carriers' requirement to use BGP, our team set out last year to build a highly scalable and proper BGP design to support our forthcoming MPLS conversion. During a week-long engineering conference we toyed with redistribution...then we delved into iBGP with an IGP inside each region. We were held back by the belief that since OSPF, at the time, was our core routing protocol, we had to continue using it in the same fashion. However, as we worked through the design, it became clear the best path was the embrace BGP as our core routing protocol and relegate OSPF to a site routing protocol. When we made that decision, our new routing design began to flourish.

The basic tenant of our design is each site in the network - no matter the size - is its own BGP AS using private BGP AS numbers. That provides 1,024 AS numbers which is more than enough for our network (that may not be enough for very large networks, but MPLS carriers are happy to AS override for you). With each site in its own AS, the WAN links at each site - be they MPLS, private-line, or GRE tunnel - would run eBGP. Now, BGP became our core WAN routing protocol. This met the MPLS carriers' requirements and made our WAN routing much simpler. We now had a protocol with the scalability to handle thousands of routes and with enough protocol features (filter-lists, route attributes, communities, etc.) to implement routing policy (something OSPF lacks).

Next we developed what I feel is the best part of our BGP design. At each site in our network all traffic flows through the core. So, we used this rule to design BGP. The core routers (high-end Cisco 7600s) are the center of BGP at each site. These 7600s are iBGP route-reflector clusters that peer iBGP to the WAN routers. By using a route-reflector cluster we avoid the iBGP full-mesh problem. The cores create all BGP routes (via the BGP "network" command) and advertise those routes to the WAN routers. The WAN routers then advertise those routes to eBGP peers over the WAN (MPLS, other sites, etc). Filtering policy is done at the edge on the WAN routers. The cores learn routes for external sites via iBGP from the WAN routers (who already learned the routes via eBGP). Thus, the core routers know all routes in the entire network. BGP easily scales to handle these global routes, unlike OSPF which does not handle thousands of routes well. This sets up a very elegant and fast BGP design. Failover is within 5 seconds when a WAN link goes down and the design can scale quickly.

OSPF is still used, but only for local LAN routing and to allow iBGP to establish between loopback interfaces. The cores, since all packets flow through them, advertise a default route into OSPF. A packet that enters the LAN that is destined for a remote site follows the default route to the core. Since, as I mentioned, the core routers know all the routes in the network, they make the proper next-hop decision to the appropriate WAN router based on iBGP. By using a default route to bring traffic to the core routers, no routing protocol redistribution is necessary. This makes OSPF and BGP very stable.

This is the base routing design of our network. However, this simply cracks the surface of what we've done with BGP in the last 2 years. Over the next few posts, I'll dive deeper into how we've utilized BGP. If you aren't using BGP, start thinking about it. It's a great protocol and, used correctly, can add significant power to your routing design.