In my last blog post, I went over how we made BGP our core routing protocol. This change has been a fantastic change for us providing stability and scalability. In that blog, I went over the core BGP architecture we use, but, as I mentioned, that just touched the surface of what we have done.
One of the other significant enhancements we have done is dynamic Internet routing using our BGP backbone. This has allowed us to consolidate Internet access to only a few large sites, while providing dynamic failover between those large sites. As I mentioned in the last blog, our core routers know all routes in the network and we inject a default route into OSPF to bring LAN traffic toward the core. This traffic includes Internet traffic that needs to continue following the default route.
At our Internet border we have either one or two Internet circuits. If there are two circuits, they are from different ISP. From the ISPs, if there are two circuits, we accept a full Internet routing table plus a default route. By accepting a full Internet routing table we are able to make proper outbound routing decisions to the Internet. If there is only a single circuit we accept only a default from the ISP. This default route is then passed via eBGP multi-hop to the internal core routers. The eBGP multi-hop session flows from the Internet routers to the internal core routers through the firewalls. The firewalls do not run BGP, just an IGP to facilitate the eBGP session between the core routers and Internet routers. To ensure the full Internet routing table does not get mistakenly dump to the internal core routers, filters are placed on both ends of this eBGP session.
Now that the internal core routers have the default route, they send that route via iBGP to the WAN routers. The WAN routers then advertise the default route to our MPLS providers via eBGP (just like all other routes are advertised). The MPLS providers receive the default route from several of our hub sites. Since the BGP AS path length on the default route is equal, the MPLS providers simply pick the best default route based on their own backbone IGP costs. Thus, sites on the East Coast get Internet access from our East Coast hub site. Sites on the West Coast get Internet access from our West Coast hub site. Same for EMEA. Same for Asia. All dynamically chosen by the MPLS providers based on BGP.
Failover is dynamic also. If we have a site with a single Internet circuit and that circuit goes down the Internet router no longer has a default route to advertise to the internal core routers. So, BGP removes the default. This change propagates through BGP to the MPLS providers. The carrier's MPLS backbone picks a default route from another hub site, reconverges, and Internet traffic flows to the other hub site. Now the local site whose single Internet circuit is down gets Internet access from another hub site. This reconvergence takes about 10 seconds. Users hardly notice the change. Furthermore, any field sites that were using this hub site for Internet access also reconverge to another hub site. This can happen globally also. For example, EMEA sites can get Internet access from the US if needed.
None of this could have been done without BGP. Its scalability, openness, and route policy features gave us the tools needed to implement this design. Consolidating and making our Internet access dynamic cut ISP circuit costs by 61%.
Michael Morris is a communications engineering manager at a $3 billion high-tech company. His background is in enterprise WANs working with telcos, and developing large-scale routing designs. He has worked on networks at government and corporate organizations, including networks at two Fortune 10 companies. In his current role, he leads large-scale IT networking projects and develops and maintains architectural standards for data networks, storage area networks, IP Telephony, and security. Michael is a CCIE and has 11 years experience in networking and communications, including four years as a paratrooper in the U.S. Army. He has a bachelor's degree in MIS from the University at Buffalo. Recently, he was awarded the Network Professional Association® (NPA) Professional Excellence and Innovation Award for his work on network architecture, templates and enterprise MPLS design.
|
|
Cut ISP circuit costs by 61%
Hi Michael,
Could you go into a little more detail regarding your last comment as pasted below? How has the change helped you to cut circuit costs so drastically?
"Consolidating and making our Internet access dynamic cut ISP circuit costs by 61%."
Thanks.
/rls
Cut ISP circuit costs by 61%
Hi Robert,
It was a combination of new contracts and cutting hub sites to single circuits now that access was dynamic with BGP.
Before we redsigned our routing with BGP we didn't have dynamic default routing. All major sites with local Internet access needed two, large circuits. Routing with the FWs was done with static routes, so they were unaware if an Internet circuit went down. Thus, we neeed two, large, circuits at these sites for redundancy.
By moving to BGP globally, we could now learn and advertise the default route dynamically across our whole network. Being dynamic, a major site now only needed 1 ISP circuit instead of two. If that circuit went down, that major site would just fail to another hub site temporarily. As I mentioned, that reconvergence takes about 10 seconds.
So, we came close to cutting the number of our large ISP circuits in half and we competitively bid the remaining circuits. Those two steps cut our costs 61%.
Mike
Thanks Mike, I currently
Thanks Mike,
I currently work for a healthcare organization where the infrastructure is completely reliant on statics to point traffic to/from firewalls for ingress/egress internet traffic. On top of that, we only have one ISP provider. I'm currently researching the best solution to incorporate not only another provider for redundancy, but also for better throughput. Our current connection is only a 50mb connection which I suspect we max daily given the amount of complaints of "slow internet" we get from end users. Like you, I believe BGP(currently running EIGRP) will be the best solution to distribute traffic more efficiently, and is a protocol that our firewalls already understand vs. having to run the firewalls in bridge mode to simply "pass" EIGRP.
That said, you have any opinion/suggestions for moving forward? I know I've left out quite a few details that might be of use but I'd be curious in hearing any thoughts you or anyone else might have.
Thanks again.
/rls
RE: Thanks Mike, I currently
Hi Robert,
So, BGP is nice since it scales well and has many route-policy features to control your routing. Those policy features can come in handy.
Now, you bring up a good point about EIGRP and FWs. No FWs, not even PIX, run EIGRP (yes, ASAs do, but please). So you have to go transparent, as you mentioned, or use GRE tunnels (worse). Here's where BGP could come in handy. You can use static routes with the FW still, but just to allow an eBGP multi-hop session between your internal network and Internet router. Via this eBGP session send a default route to the internal network. Now, if the circuit goes down or the FW has problems the eBGP session will go down and the default route will be dropped. Your internal network can then converge to another internal location. The static routes pointing to the FW do not come into play since they only point to the Internet router, not the actual default route.
This same idea can work if you have two circuits at this site (as you mentioned your thinking about getting). Same concept, but you will just have greater circuit redundancy.
As for a new circuit, definitely ask as many carriers as you can for quotes. There are a lot of ISPs out there that would love to sell you bandwidth. If you need a list send me a private e-mail, I can help. Also, if your Internet traffic is mostly user based and is high during the day, but very low over night and on weekends, think about getting a burstable circuit. That will keep the costs down and is billed on a 95th-percentile basis. The low utilization over night and on weekends will keep the 30-day average low and keep your burst charge low, but you get a ton of bandwidth during the day when you need it.
Good luck.
Mike
BGP on Firewall ;-)
Use a Juniper Netscreen Firewall and run BGP. ;-)
BGP on FWs - Tried That
We actually tried that. The problems is while sessions are HA on Juniper FWs, routing protocols are not. So when a FW failover occurs, all the routing protocols reset. This is a real pain. Yes, the session table is sent to the stanby FW, but routing protocol peers reset. By the time the RP peers re-establish most sessions have timed-out anyways. It's a nasty little problem.
We ended up doing OSPF on the FWs instead of BGP because OSPF would re-establish much faster than BGP. The BGP (multi-hop) between the Internet router and internal core router flows through the FWs.
Michael J. Morris
CCIE #11733, JNCIA
Need advise on choosing right Core for BGP?
Hi,
this is my first time and I found the BGP posting my Mr. Morris to be very useful to my current problem.
I am facing a problem after being on this work for 5 days only, a Web Hosting company with a flat network.
now they have a transit ring from datacenter to their disaster recover using dark fiber with a provider and then to the Internet through the same provider.
i have been given the task to create a whol new infrastruture using 4 Cisco 6506, two at each side of this ring (2 in datacenter or headquarter office and 2 in the disaster reovery site) and 2 Cisco 6509 as core swithes for connecting servers and customers to them.
now, I am confused on where to run BGP? on 4 6505s? or only at one of them at each site?
I have ben asked to also configure 6509s for EIGRP to take care of routing (IGP) and Private VLANs to separate each customers and server types?
i need some help on where and on what dvices I must configure BGP and what and on which devices EIGRP and where to redistribute?
I am very comfortable with BGP configuration and troubleshooting on providers netwrok as well as enterprise networks but where and how to approach this task from scratch in very limitted time Frame, I am lost! being very new also has contributed to my being lost!
please advise if possible and it means alot to me. i have not used this site before and don't know how to view your comments and recommendations?
can email notifications be possible so I know responses have been posted?
please advise.
Regards,
Mike
RE: Need advise on choosing right Core for BGP?
Mike,
E-mail me a diagram or two with a longer explanation of the problem and I'll see if I can help you.
mjmorris@yahoo.com
Michael J. Morris
CCIE #11733, JNCIA
Internet proxies failover
Hello,
I agree with the design - just the e-bgp multihop is kind of weird, it would be really better if the FWs support BGP, but....
The only drawback that could prevent most customers for using this type of internet failover over another site (or DC) is that usually there are proxies involved and their external IPs are usually routed only by the local ISP. (at least that is my EMEA experience). So, do you have any proxies in your design and how do you incorporate them into the design?
Internet proxies failover
Hello,
I agree with the design - just the e-bgp multihop is kind of weird, it would be really better if the FWs support BGP, but....
The only drawback that could prevent most customers for using this type of internet failover over another site (or DC) is that usually there are proxies involved and their external IPs are usually routed only by the local ISP. (at least that is my EMEA experience). So, do you have any proxies in your design and how do you incorporate them into the design?