Around 4:30 a.m. EST on Feb. 11, AT&T suffered an outage centered in Washington, D.C., followed by issues in Tulsa, OK, and San Antonio, TX. that affected customers including some in the US, Germany, and Australia. Five minutes later only Washington, DC, and Tulsa, OK, nodes were involved and five minutes after that, just those in Washington, DC. Customer networks affected included J.P. Morgan Chase, Jeffries Group, Travelers Property Casualty, and ConocoPhillips. The outage lasted 14 minutes and was cleared at around 4:45 a.m. EST. Click here for an interactive view of the outage.
On Feb. 11, Cogent Communications experienced a series of outages over a period of 3 hours and 38 minutes that affected downstream providers as well as Cogent customers globally. The outage lasted a total of 43 minutes and was first observed around 1:07 a.m. EST centered on Cogent nodes in San Francisco, CA. This initial outage lasted around two minutes, and the Cogent environment was then stable for 43 minutes before experiencing a series of four-minute outages observed on Cogent nodes in Dallas, TX and Phoenix, AZ. An hour and a half after the initial outage was observed, a four-minute outage was observed centering on Cogent nodes located in Newark, NJ. The outage reappeared 25 minutes later, extending the list of affected nodes to locations including San Francisco, CA, Atlanta, GA, Cleveland ,OH, New York, NY, and Washington D.C. Less than an hour later, a 19-minute outage affected nodes including Miami, FL, Austin, TX, El Paso, TX, Houston, TX, San Jose, CA, Los Angeles, CA, and Las Vegas, NV. Customer networks affected included Ford Motor Company, Oracle, Home Depot, and TikTok. The outage was cleared at around 4:45 a.m. EST. Click here for an interactive view of the outage.
Updated Feb. 8
Globally outages in all three categories decreased from 278 to 267, a 4% decrease. In the US, outages decreased from 119 to 94, a 21% decrease.
ISP outages worldwide decreased from 214 to 192, a 10% drop. In the US, ISP outages decreased from 102 to 78, down 24%.
Cloud-provider network outages worldwide dropped from 30 to 21, down 30%. In the US they dropped from 10 to three.
Collaboration-app network outages globally decreased from five to two and stayed the same, at two, in the US.
There were two notable outages this week. Around 1:40 a.m. on Feb. 1, Hurricane Electric experienced an outage affecting countries including the US, New Zealand, and Brazil. The outage was centered on Hurricane Electric infrastructure in Los Angeles, CA. After five minutes, the number of interfaces affected there reduced and appeared to affect users in the US only. The outage lasted around nine minutes and affected customers including Disney Streaming and LinkedIn. The issue was cleared around 1:50 a.m. ET. Click here for an interactive view of the outage.
On Feb. 2, TATA Communications (America), experienced an outage that affected some of its downstream partners and customers in countries including the US. It was first observed around 1:30 a.m. ET as TATA nodes located in Los Angeles, CA, appeared to show outage conditions. After five minutes, other nodes located in Seville, Spain, and Singapore, were affected. As the number of affected nodes increased, so did the number of customer networks affected, including Wells Fargo, Reuters, Twitter, and Salesforce. The outage lasted around 18 minutes across a half hour period and was cleared around 2 a.m. ET. Click here for an interactive view of the outage.
Update Feb. 1
Outages in all three categories globally ticked up three from 275 to 278 during the week and dropped from 132 to 119 in the US, an 11% decrease.
Globally, the number of ISP outages decreased from 217 to 214. In the US, levels slightly decreased too, from 106 to 102.
Cloud-provider network outages jumped from 18 to 30 worldwide, a 67% increase. US outages increased from nine to 10.
Collaboration-app network outages remained flat both globally and in the US, with five worldwide and two in the US.
Two notable outages affecting Comcast Cable and Verizon occurred during the week.
On Jan. 26, Comcasts suffered a 24-minute outage first observed around 12 a.m. ET that appeared to be centered at Comcast nodes in Newark, NJ, affecting access to customer networks including Amazon, Bloomberg and CBS. At 20 minutes into the outage, disruption was also observed in a New York, NY, node. The outage affected multiple Comcast peers and customers, and it was cleared around 12:25 a.m. ET. Click here for an interactive view of the outage.
On Jan 12 Verizon experienced an outage that affected East Coast customers abilty to access services including Slack, Zoom, Amazon and Google Traffic disruption was observed around 11:30 a.m. ET across multiple nodes concentrated along the US Verizon backbone. During the outage Verizon indicated a fiber cut affected service delivery in the Brooklyn, NY, area but is not believed directly related to the larger outage. Network services started to stabilize around 12:30 p.m. ET. Click here for an interactive view of the outage.
Update Jan. 26
Outages in all three categories worldwide rose from 215 to 275, a 28% increase compared to the week before. In the US they rose from 105 to 132, a 50% increase.
Globally, the number of ISP outages increased from 160 to 217, a 36% increase and in the U.S., they increased from 88 to 106, a 20% increase.
Cloud provider network outages increased from 14 to 18 globally, a 29% increase, while in the US they jumped from two to nine, a 350% increase.
Collaboration app network outages worlwide increased from two to five and from one to two in the US.
There were two notable outages during the week. On Jan. 20, Level 3 Communications experienced an outage affecting downstream partners and customers in countries including the US, South Africa, the U.K., Turkey, Russia, New Zealand, and Australia. The outage was first observed around 12:20 p.m. ET and lasted about 34 minutes. It appeared centered on Level 3 nodes in Washington, DC, and affected customers including J.P. Morgan Chase, Visa International, and Oracle. IT was cleared around 12:55 p.m. ET. Click here for an interactive view of the outage.
On Jan.18, TATA Communications (America), experienced an outage affecting many of its downstream partners and customers in multiple countries including the US, the UK, Australia, India, Singapore, Germany, Hong Kong, Japan and Canada. The outage was first observed around 7:40 a.m. ET. TATA nodes located in Newark, NJ; New York, NY; Frankfurt, Germany; London, England; Singapore; Paris, France; and Buckinghamshire, England all appeared to show outage conditions. After five minutes, TATA nodes located in Chicago, IL; San Jose, CA; Seville, Spain; Tokyo, Japan; and Hong Kong were affected. As the number of TATA nodes affected increased, so did the number of customer networks affected, including Wells Fargo, Reuters, Oracle, and Amazon. The outage lasted around 34 minutes and was cleared around 8:15 a.m.ET. Click here for an interactive view of the outage.
Update Jan. 19
Worldwide outages in all three categories increased from 157 to 215 over the previous week, an increase of 37%, and were up from 88 to 105 in the US, a 19% increase.
The total number of ISP outages increased from 122 to 160, up 31%, while in the US the increase was 19%, from 88 to 105.
Cloud-provider network outages doubled from 7 to 14 worldwide and remained the same in the US at two.
Globally there were two collaboration-app network outages, up from one, and they remained at one in the US.
There were two notable outages. On January 13, AT&T experienced an outage that affected customers in multiple countries, including the US, UK, Japan, Germany, Canada, Australia, India, and the Netherlands. The outage started around 9:25 p.m. ET and centered on AT&T nodes located in Phoenix, AZ, and last 23 minutes, and was cleared around 9:50 p.m. ET. Click here for an interactive view of the outage.
On January 13, Microsoft experienced an outage that affected some downstream partners and access to services running in Microsoft environments. The 12-minute outage was first observed around 12:15 a.m. ET and occurred in three four-minute occurrences over a 30-minute period. IT centered in Microsoft nodes in Des Moines, IA. It was cleared around 12:45 a.m. ET. Given the timing and uniform pattern of the outage, it is likely to have been an automated maintenance process. Click here for an interactive view of the outage.
Update Jan 12
Global outages in all three categories increased from 96 to 157, up 39% from the week before, and in the US they jumped from 33 to 88, up 167%.
ISP outages worldwide went from 71 to 122, up 72%, and from 30 to 74 in the US, up 147%.
Cloud provider outages increased from two to seven, a 250% increase,
Globally, cloud provider network outages increased from 2 to 7, a 250% increase, and from zero to two in the US.
There was just one collaboration-app network outage and that occurred in the US. There were none the week before.
There were two notable outages during the week. On Jan. 4, Slack experienced an outage at 10 a.m. EST that lasted until after 1:40 p.m. It affected customers worldwide, with many users unable to login, send or receive messages, or to place or answer calls. Slack identified the cause as insufficient router capacity in its cloud-provider network to meet customer demand. Starting at 11:15 a.m. EST Slack implemented a fix, and many customers could use the service again by 12:15 p.m. Slack announced messaging service restoration at 1:40 p.m. EST, although it’s calendar integration features took longer to restore.
On Jan. 7, Cogent Communications experienced an outage at 4:40 p.m. that lasted just under an hour and that affected downstream providers and Cogent customers globally. It consisted of four outage occurrences over a two-hour period, the first of which centered on Cogent nodes in Amsterdam, the Netherlands, mainly affecting European countries. Five later, Cogent nodes in Washington, DC, also exhibited outage conditions. At this point the Amsterdam nodes recovered, but the Washington D.C. nodes stayed down for another 35 minutes. Thirty-five minutes after the first outage cleared, the second outage was observed, centering on nodes in Oakland, CA. It lasted four minutes and affected only customers in the US. This was repeated five minutes later, this time lasting around three minutes. Following a five minute break, a final four-minute outage was observed, this time centering on Cogent nodes in Las Vegas, NV, and Oakland, CA. The outage affected access to services including Amazon, Yandex (Russan based search engine), Oracle, and Sberbank (a state-owned Russian banking and financial services company). The outage was cleared around 6:35 p.m. EST. Click here for an Interactive view of the outage.
Update Jan. 5
Outages in all three categories decreased from 172 to 96, a 44% decrease compared to the week prior. In the US, they decreased from 80 to 33, a 59% decrease.
Globally, ISP outages decreased from 135 to 71, down 47%. In the US, they dropped from 74 to 30, a 59% decrease.
Cloud-provider network outages decreased from five to two, and in the U.S., from two to zero.
There were no collaboration app network outages the previous two weeks.
Update Dec. 21
Total outages across all three categories dropped vs. the previous week, from 252 to 193, a 23% difference. In the US they outages decreased from 115 to 89, also a 23% difference.
Globally, the number of ISP outages decreased from 180 to 145, a 19% decrease, and in the US they decreased from 97 to 75, a 23% drop.
Cloud-provider network outages worldwide decreased from 11 to four, down 64%, while in the US they fell from 2 to 1.
There were three collaboration-app network outages during the week, all in the US. The week before there were four outages, none of them in the US.
There were two notable outages. On Dec. 14 between 6:50 a.m. and 7:30 a.m. EST Google experienced a global outage. ThousandEyes tests measured elevated server wait times, indicating the application was taking longer to respond to service requests. During the service disruption, network paths connecting to Google’s edge servers did not show any traffic loss
The other notable outage hit NTT America and affected some downstream providers and NTT networks in multiple countries including the US, Germany, Brazil, the UK, and Canada. The outage was first observed around 8:30 a.m. EST and appeared to be centered on NTT infrastructure in Los Angeles, California, and Seattle, Washington. The outage lasted just over 19 minutes and was cleared around 8:50 a.m. EST. Click here for an interactive view of the outage.
Update Dec. 14
Total outages in all three categories were up 26%, from 200 to 252, over the previous week, and the were up 39% in the US, from 83 to 115.
ISP outages worldwide increased from 129 to 180, a 40% increase. In the US, ISP outages increased from 66 to 97, a 47% increase.
Worldwide cloud-provider network outages increased from eight to 11, up 38%. In the US they increased from one to two.
Globally, there were four collaboration-app network outages, up from zero. None of them were in the US.
There were two notable outages during the week. On Dec. 10, Hurricane Electric experienced a 17-minute outage that hit users in the US, Canada, Germany, Egypt, Sweden, France, and the UK. The outage was first observed around 2:11 p.m. EST centered on Hurricane Electric infrastructure in Atlanta, Georgia, and 10 minutes later just in Dallas, Texas. The last two minutes affected Hurricane Electric interfaces in both Atlanta and New York, New York. The issue was cleared around 2:38 p.m. EST. Click here for an Interactive view of the outage.