At last week's FutureNet conference one of the three major topics areas was the future of the Internet. Nemertes Research, the host of FutureNet, brought a very impressive group of Internet experts to the conference to discuss the issue.
Johna Till Johnson, President of Nemertes and moderator of a spirited round table discussion about the Internet at FutureNet, wrote about the Internet's forecast earlier this week, predicting "snow days". A couple other stories were also written this week about issue, one using another weather metaphor and another using a Chicken Little-like sky-is-falling description.
I thought I'd give a simpler description of the issue, with a router-jockey view.
Two big issues:
#1 - We are running out of IPv4 space (we knew that).
#2 - The global Internet routing table is too big now and getting bigger fast.
First, we've heard for years about the lack of IPv4 space. This was the main driver for IPv6's development in the 1990s. However, this problem has been mitigated, for a while, by NAT. Pure network engineers don't like NAT, but let's be honest - it works! However, even with NAT, there is still a growing need for public IPv4 space for companies and organizations. And, while there are 4,294,967,296 IPv4 addresses, poor address allocations over the last 30 years have led to a much smaller pool. Enter IPv6 with its 340,282,366,920,938,000,000,000,000,000,000,000,000 addresses. Problem solved right....well....
But, for now, let's discuss problem #2, which is caused by problem #1. Since there is a smaller and smaller pool of IPv4 addresses, companies are getting smaller and smaller public IPv4 assignments. For example, let's say your company gets its own /20. But, that's for your whole company, and you have 16 global sites that need to use that public IPv4 space. So you breakup your /20 into /24s and give each site, in different parts of the world connected to different ISPs, a /24. Oh, and don't forget, you're a good network engineer so each of those sites is multi-homed to two carriers, so that /24 is advertised twice - once to each carrier. Your company, which had a nice, single /20 public IP range, has just created 32 more routes in the global Internet routing table. Ouch. Now, imagine thousands of companies and organizations around the world doing that. Double Ouch.
That leads to today's extremely large, and growing every day, Internet routing table:
Neighbor V AS Up/Down State/PfxRcd [removed] 4 [removed] 8w6d 247868
The point of the experts at the conference was 250,000 routes is so large the Internet doesn't fully converge today, which leads to constant fluctuations and brown-outs. And it keeps getting larger. At some point, core routers on the Internet will not be able to hold that many routes in the forwarding table silicon (TCAMs). So, no convergence and, even if it did converge, the router's couldn't hold the routing table anyways.
But, before we melt down the Internet, we need to discuss issue #3 - which is issue #1 + #2. As we move into the near future, public IPv4 space starts to run out even more. So, you can't get a /20 anymore, you get a /24 (maybe). But you still have 16 sites in your company. So now, instead of sending 32 /24s in the Internet routing table you start sending in 32 /28s. Now the routing table starts to grow faster...exponentially. Now those 250,000 routes becomes 1,000,000 routes....maybe more. Now there are more routes to converge and more to store in the TCAMs. At what point does it start to break down?
A Juniper engineer spoke a little later in the conference about Internet scalability and mentioned that Juniper's earlier core routers (from the late 1990s) could handle today's Internet routing table. So, it appears we have some time on that point. The question is how much time? And what about older, smaller routers that can't handle that size table?
So, I know what you're thinking. Just configure IPv6 which will bring balance to the force. I'm not going to delve into that, but we all know that a conversion to IPv6 is tough. Plus, NAT-PT is a joke. Customers - the people who pay the ISPs - don't want IPv6 yet, so it doesn't get built.
So, let's start a pool. How many routes in the global routing table before things start to break? I'm going for 6,130,808 routes. What do you think?
More >From the Field blog entries:
Clearing the Air on the CCDE Practical Beta Exam
CCDE Practical Beta Test Cancelled
Quick Thoughts on the New Nexus 5000
What Goes Into a Written Network Architecture?
I Can Fix Anything With a Tunnel
Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.
Michael Morris is a communications engineering manager at a $3-billion high-tech company. His background is in enterprise WANs working with telcos and developing large-scale routing designs. He has worked on networks at government and corporate organizations, including networks at two Fortune 10 companies. In his current role, he leads a team of 10 engineers responsible for large-scale IT networking projects and architectural standards for data networks, storage area networks, IP telephony, contact centers, and security. Michael is CCIE #11733 and recently became one of the first three Cisco Certified Design Experts (CCDE) ever (#20080002). He has 11 years experience in networking and communications, including four years as a paratrooper in the U.S. Army. He has a bachelor's degree in MIS from the University at Buffalo and is working on his MBA from NC State University. In 2008, he was awarded the Network Professional Association (NPA) Professional Excellence and Innovation Award for his work on network architecture, templates and enterprise MPLS design.
The Leader in Network Knowledge
IPv6
Hi Michael
Does IPv6 help fix problem #2 at all, or does it in fact have the possibility of making problem #2 even worse?
That is, at least with IPv4, you know the worlds going to run out of address space sooner rather than later.
Good Internet Citizen
Mike,
I remember back in 2001 we were using foundry routers and the memory allocated to BGP routing table was 128MB...we had to increase it as the table grew to 168MB the router crashed but that's another story...:)
Do you feel that as provider and good Internet citizen the onus is on us to ensure that cleaner networks and more summarized networks are announced? but then again that would just be delay the problem till we find a proper fix.
I feel that until the router prices are within the reach of the customer (either router vendor come up with low cost solution for TCAM or new technologies) the customer will not be bothered.
/Majid
RE: Good Internet Citizen
What's the business benefit of being a "Good Internet Citizen"?
Your last paragraph hit on the point. If it's not financially viable, customers are not going to do it.
Mike
RE: IPv6
Actually, yes...#2 could make things worse in the future since there are a lot more IPv6 subnets availaible and IPv6 does not fix multihoming.
Routing tables could be fixed if
#1 - carriers controlled all IP addressing
#2 - multihoming is outlawed.
However, neither of those are ever going to happen nor would we want to. Imagine if you could only get two Internet circuits for your critical site from the same carrier. And, if you wanted to switch carriers, you'd have to renumber your entire organization. Ouch.
IPv6 will be a slower growth though since there is more addresses to give out, so it will take longer to grow the Internet routing tables. And, by then, maybe quantum computing or something will fix the problem. ;-)
Mike
RE: IPv6
Routing tables could also be fixed by a re-design in way routes are advertised.
In IPv6 there is the wonderful route of 2000::/3.
If the router was to say, classify its interfaces as up (upstream provider where we get to the internet from) and down (our internal clients and clients networks).
Multi-homing becomes a matter of selectively advertising ONE route (2000::/3) to all 'down' clients. And if we are allowing our carriers to use us as a transit we advertise it 'up' to them as well.
Of course we already have either a preferred-route provider or a balancing act going on for our 2+ homed upstreams. Regardless of the table size.
Bang. That takes care of the about 2^32 external Internet routes that aren't directly ours.
Note the major side-effect of this: the smaller the ISP the lower budget and smaller routers they can afford. Which also matches closely the amount of clients they have and thus routes they need to store in their own internal table.
The remaining issue is what do we advertise to our direct upstream? And what happens at the level.
Well, mitigating factor #1 is that the upstream are a larger company with probably bigger pockets to afford the router needed to handle the table at their level.
Mitigating factor #2 is believe it or not RAD - Below Tier-3 the majority of support devices, end-devices, and end-customers don't actually need to have permanent inbound addresses. For all of those auto-assignment works perfectly well and is far less of a headache.
And Bang, the downstream route table we need to advertise 'up' has dropped from many to just 1 (ours).
If we are lucky and one or both of our upstream are hopefully offering us the same auto-assign service and we can re-use a section of the spaces (both) they give us. The needed 'up' routes goes from 1 to none!
Of course, there are those public-facing devices which need a permanent address and thus routes. They remain a problem. Along with some now circuitous packet routing paths for SYN packets (SYN-ACK and later follow other IPv6 routing improvements to find the shortest-path back).
So the sky may or may not be falling. But its at least doing so slowly.
The Problem with What's Left of IPv4, and How IPv6 Helps
Hi Michael,
You make the very good point (#2) that the Internet routing table is getting bigger by the day, and BGP is struggling to manage. You also shine a light, in your examples, of why this is: Right now more than 50% of the 250,000+ entries in the Internet routing table are /24s. That results partly from multihoming practice, as you say, and also partly due to simple laziness in using proper aggregation policies.
But a very large contributing factor is that as the remaining IPv4 addresses run out, the IPv4 address space is becoming more and more fragmented. There's alot of talk among some in denial about IPv4 depletion that there are still plenty of unused IPv4 addresses in possession of various institutions, and that they might extend the life of IPv4 by creating a market for those addresses.
There's some fact around that; there are indeed quite alot of allocated -- but unused -- IPv4 addresses. It's arguable how long putting them on the market will actually stave off IPv4 depletion (or whether such institutions can even legally do this; having an IPv4 allocation does not mean having ownership of those addresses), but the point that is often missed is that these addresses are highly fragmented. If they start showing up in the Internet tables, they are going to be as distinct /24s or longer, and contribute to accelerated table explosion in a short order. Along with increased convergence times and decreased stability.
The Internet tables will grow at a much slower rate under IPv6, even as it is used to address billions of new devices, because the IPv6 blocks being allocated, and advertised into BGP, are more aggregated than IPv4.
Sure, there will still be some longer IPv6 prefixes advertised than desirable because of poor multihoming practices, but all in all the IPv6 Internet table -- even though it will be representing far more devices than the current IPv4 table does -- will grow at a much lower rate than the increasingly fragmented IPv4 table.
--Jeff