The biggest outages of 2021 had one thing in common: they affected major infrastructure or services providers and, as a result, affected large numbers of enterprises and end users. The lesson? Companies need to be careful about putting all their infrastructure eggs in one basket, or, if they must, to prepare for downtime if that particular service goes down.\n"There needs to be a plan in place," says Angelique Medina, head of product marketing at ThousandEyes, a Cisco-owned network intelligence company that tracks internet and cloud traffic. "Organizations don't need to be at the mercy of the availability of any one particular service."\n\n Read more: Cisco's ThousandEyes can peer into SaaS performance\n\nTwo of last year's biggest outages included cloud providers AWS and Azure. Two involved Internet service providers Verizon and Azure. Four outages involved CDN and DNS providers Akamai, Cloudflare, and Fastly. And rounding out ThousandEyes\u2019 list of the top 10 outages of 2021 are two Facebook outages.\u00a0\nThe Facebook outages didn't just take down the social media network and other company services like Instagram and WhatsApp. Many enterprises use Facebook to authenticate users. When that service went down, users were no longer able to log into those enterprises' websites.\n"Authentication, like DNS, is often overlooked when people think about availability," Medina tells Network World.\u00a0(Read more about U.S. and worldwide outages in our\u00a0weekly internet health check.)\nAnother overlooked networking issue that showed up in the top outages this year is BGP routing. BGP \u2013 which stands for border gateway protocol \u2013 tells Internet traffic what route to take. Even if the DNS listings point to the right destination, if the routing information is incorrect then traffic can be diverted to a dead-end route or a route that doesn't have enough capacity to handle all the traffic.\n"BGP hijacking can be really, really scary," says Medina. "It can be a very challenging thing to control for and can have very damaging effects."\nHere's the list of the top outages:\n1. Facebook: Oct. 4\nThe biggest outage of the year in 2021 was October's Facebook outage. "It was a hard down for about seven hours," says Medina. "Seven hours is pretty significant."\nThe outage affected all of Facebook's services, including Instagram, WhatsApp and Oculus, as well as all the enterprises that use Facebook's authentication mechanism.\nA routine maintenance job went wrong, and both system servers and BGP routes were affected. Worse yet, not only did Facebook's public-facing services go down, but also the tools that the employees use to manage those services. As a result, staffers had to physically enter the data centers to manually restart systems.\nAccording to Facebook's VP of infrastructure Santosh Janardhan, a command was accidentally issued that took down all the connections in its backbone network, disconnecting all of Facebook's data centers.\n"Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command," Janardhan said in a report released the next day.\nThat mistake took down systems that respond to DNS queries. Since Facebook's DNS servers could no longer connect to the data centers, they automatically disabled the related BGP advertisements, and so those DNS servers became unreachable even though they themselves were still up and running.\n"All of this happened very fast. And as our engineers worked to figure out what was happening and why, they faced two large obstacles," Janardhan wrote. "First, it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we\u2019d normally use to investigate and resolve outages like this."\nThe lesson here, according to Medina, is that companies need to keep their management systems well siloed from their production environment.\n"And also consider diversifying in terms of who's providing your services," she adds. "Not necessarily just relying on your own internal services, but potentially considering external providers \u2013 or multiple external providers."\n2. AWS: Dec. 7\nAWS is the largest cloud-computing service provider in the world, and when its services go down, millions of enterprises can be affected.\nOn December 7, an outage that lasted for over an hour affected Amazon's own services, as well as consumer devices like Roomba and Ring and streaming services like Disney+ and Netflix, because of problems with AWS EC2 APIs in the US-EAST-1 region.\nThe outage highlighted the need for enterprises to monitor the health of all the APIs that are part of their applications and contribute to service delivery, customer experience, and the company's ability to build and deploy, says Chris Villemez, senior technical marketing engineer at San Francisco-based ThousandEyes.\nCompounding the problem, enterprise customers didn't see any information on their AWS status page for more than an hour.\n"It's never a good idea to rely solely on a provider for information," says Medina. "Having more of an independent view will give you more insight in real time."\n3. Fastly: June 8\nFastly is one of the smaller content delivery networks. According to Enlyft, the company has a 4% market share, compared to 39% for CloudFlare and 24% for Amazon CloudFront.\nStill, more than 100,000 companies use its services, including Reddit and the New York Times. Even Amazon and eBay use some of Fastly's services and were affected by Fastly's June outage.\nBut customers had greatly varying experiences of the outage, depending on the degree to which they relied on Fastly services and how they reacted to the outage.\nFor example, Reddit went down completely and stayed down for the whole duration of the outage, which lasted nearly an hour, according to a report by ThousandEyes.\nBut the New York Times was able to reduce downtime by sending users directly to its site servers, which were hosted in the Google Cloud Platform. It still took time to make the fix, and time for the updated DNS records to propagate.\n"Depending on how long-lived your DNS records are, that can influence how quickly you're able to help your users," says Medina.\nAmazon uses multiple content distribution networks, including its own Cloudfront CDN and Akamai. When Fastly went down, it was able to reroute requests to other CNDs, significantly reducing the impact of the outage.\nSimilarly, eBay used Fastly for only some content, specifically individual objects on web pages. The company used Akamai to deliver the web pages themselves. Over the course of the outage, eBay was able to redirect requests away from Fastly and was eventually able to reduce the impact of the outage even further.\n4. Akamai Edge DNS: July 22\nAkamai is a global content delivery network, similar to Fastly in number of users and market share. And, as with the Fastly outage, companies that used multiple CDNs saw less impact from the outage.\nIn the case of this particular outage, the Akamai DNS service, which directs users to Akamai's CDN network, went down for over an hour. According to the company, a software configuration update triggered a bug in their Secure Edge Content Delivery Network impacting that network's domain name service system.\nMany major websites were affected, including Steam, American Airlines, Fox News, and HSBC. Amazon, which uses multiple CDNs, was able to reroute traffic and spared users any impact.\n5. Akamai Prolexic Routed: June 16\nThe July outage wasn't the only major outage for Akamai last year. In June, the company saw a breakdown of its DDOS mitigation services, Prolexic Routed, because of an issue with BGP routing.\nSome customer websites were unreachable for varying amounts of time, according to ThousandEyes. But by quickly taking the action of removing routes, Prolexic minimized the impact to its customers, and customers were free to reinstate BGP announcements through other providers to route around the issue. "Once that action was taken, customers who quickly restored connectivity to their sites were the ones who had redundant processes already in place."\n"Organizations don't have to be at the mercy of the availability of any one particular service," says Medina.\nAccording to Akamai, there were about 500 customers using this DDOS mitigation service. Many were routed automatically, restoring operations within minutes. Most of the rest were manually rerouted soon afterwards. The outage was caused because a routing table was accidentally exceeded.\n6. Verizon: Jan. 26\nVerizon's outage was the first major outage of 2021 and hit users from Washington, D.C. to Boston. "A lot of folks may not recall this, but it was pretty significant," says Medina.\nTens of thousands of customers were left without service as Verizon's FIOS network went down, including companies and employees working from home.\nAccording to Verizon, the disruption was due to a "software issue" triggered during routine network management activities, and was unrelated to a cut fiber line in Brooklyn, which happened at the same time.\n7. Comcast: Nov. 9\nAnother major outage of an Internet service provider occurred in November, when Comcast's network backbone in the San Francisco area went down for nearly two hours, then was followed by a more widespread outage that lasted for over an hour across multiple cities in the U.S., including Chicago and Philadelphia and stretching into New Jersey and South Carolina.\nTens of thousands of home and business users of Comcast's Xfinity network were affected. "There was clearly some internal routing issue," says Medina.\nWith both the Comcast and Verizon outages, the lesson is that companies need to have backup connectivity plans, not just for their own services but also for their employees and other key users.\n8. Cloudflare Magic Transit: May 3\nThe May Cloudflare outages is another example of an outage due to a service that is specifically designed to protect companies against outages. Like Amazon's Prolexic, Cloudflare's Magic Transit service is meant to help protect customers against DDOS attacks by routing traffic through their network, inspecting it, scrubbing it, and sending it on to where it's supposed to go.\nThis particular outage affected Cloudflare infrastructure around the globe, with issues occurring at varying levels for about two hours.\n"There were certain customers who were very rapidly able to respond," says Medina. These customers caught that there was a problem with the BGP routing and quickly advertised new routes, she says. "Having early awareness of what's going on and also redundancy \u2013 even if the outage is ongoing \u2013 reduces impact on you."\n9. Azure AD: Dec. 15\nThe most recent of 2021's major outages was December's Active Directory outage. Azure's AD service went down for one and a half hours in mid-December, preventing users from signing into Microsoft services like Office 365.\nSome enterprises also use the service for authentication to their own systems and services, says Medina.\n"So even though the applications themselves might have been available, users weren't able to log in," she says. "Authentication is one of these dependencies that sometimes gets overlooked when you're thinking about availability."\n10. Facebook: April 8\nFinally, rounding out the list is Facebook again, with an April outage that lasted about 40 minutes.\n"What was really interesting about this particular incident is it really highlighted how Facebook uses DNS to route users to its service," says Medina.\nIn this outage, too many users were routed to just one data center, creating network congestion.\n"It took some time for them to normalize the routing across their CDN edge," she says.\nKnow your digital supply chain\nThe key takeaway from all these outages is that companies need to be aware of all the components and dependencies that go into making their systems work, both on the back end, supporting their application servers, and on the front end, delivering data to end users.\n"These components give us a lot of options and lots of flexibility and ultimately the power to deliver content across the Internet," Villemez says.\nBut it results in a complex set of interconnected services and dependencies, many of which are outside a company's direct control.\n"So, for ITOps teams, it is absolutely critical that we know not just our direct dependencies, but also those indirect ones," he says. Then, companies need to plan ahead for the failure of any of these critical components. "Know how you can work around the problem while providers are trying to resolve something," he says.