There were two notable outages this week. Around 1:40 a.m. on Feb. 1, Hurricane Electric experienced an outage affecting countries including the US, New Zealand, and Brazil. The outage was centered on Hurricane Electric infrastructure in Los Angeles, CA. After five minutes, the number of interfaces affected there reduced and appeared to affect users in the US only. The outage lasted around nine minutes and affected customers including Disney Streaming and LinkedIn. The issue was cleared around 1:50 a.m. ET. Click here for an interactive view of the outage.
On Feb. 2, TATA Communications (America), experienced an outage that affected some of its downstream partners and customers in countries including the US. It was first observed around 1:30 a.m. ET as TATA nodes located in Los Angeles, CA, appeared to show outage conditions. After five minutes, other nodes located in Seville, Spain, and Singapore, were affected. As the number of affected nodes increased, so did the number of customer networks affected, including Wells Fargo, Reuters, Twitter, and Salesforce. The outage lasted around 18 minutes across a half hour period and was cleared around 2 a.m. ET. Click here for an interactive view of the outage.
Update Feb. 1
Outages in all three categories globally ticked up three from 275 to 278 during the week and dropped from 132 to 119 in the US, an 11% decrease.
Globally, the number of ISP outages decreased from 217 to 214. In the US, levels slightly decreased too, from 106 to 102.
Cloud-provider network outages jumped from 18 to 30 worldwide, a 67% increase. US outages increased from nine to 10.
Collaboration-app network outages remained flat both globally and in the US, with five worldwide and two in the US.
Two notable outages affecting Comcast Cable and Verizon occurred during the week.
On Jan. 26, Comcasts suffered a 24-minute outage first observed around 12 a.m. ET that appeared to be centered at Comcast nodes in Newark, NJ, affecting access to customer networks including Amazon, Bloomberg and CBS. At 20 minutes into the outage, disruption was also observed in a New York, NY, node. The outage affected multiple Comcast peers and customers, and it was cleared around 12:25 a.m. ET. Click here for an interactive view of the outage.
On Jan 12 Verizon experienced an outage that affected East Coast customers abilty to access services including Slack, Zoom, Amazon and Google Traffic disruption was observed around 11:30 a.m. ET across multiple nodes concentrated along the US Verizon backbone. During the outage Verizon indicated a fiber cut affected service delivery in the Brooklyn, NY, area but is not believed directly related to the larger outage. Network services started to stabilize around 12:30 p.m. ET. Click here for an interactive view of the outage.
Update Jan. 26
Outages in all three categories worldwide rose from 215 to 275, a 28% increase compared to the week before. In the US they rose from 105 to 132, a 50% increase.
Globally, the number of ISP outages increased from 160 to 217, a 36% increase and in the U.S., they increased from 88 to 106, a 20% increase.
Cloud provider network outages increased from 14 to 18 globally, a 29% increase, while in the US they jumped from two to nine, a 350% increase.
Collaboration app network outages worlwide increased from two to five and from one to two in the US.
There were two notable outages during the week. On Jan. 20, Level 3 Communications experienced an outage affecting downstream partners and customers in countries including the US, South Africa, the U.K., Turkey, Russia, New Zealand, and Australia. The outage was first observed around 12:20 p.m. ET and lasted about 34 minutes. It appeared centered on Level 3 nodes in Washington, DC, and affected customers including J.P. Morgan Chase, Visa International, and Oracle. IT was cleared around 12:55 p.m. ET. Click here for an interactive view of the outage.
On Jan.18, TATA Communications (America), experienced an outage affecting many of its downstream partners and customers in multiple countries including the US, the UK, Australia, India, Singapore, Germany, Hong Kong, Japan and Canada. The outage was first observed around 7:40 a.m. ET. TATA nodes located in Newark, NJ; New York, NY; Frankfurt, Germany; London, England; Singapore; Paris, France; and Buckinghamshire, England all appeared to show outage conditions. After five minutes, TATA nodes located in Chicago, IL; San Jose, CA; Seville, Spain; Tokyo, Japan; and Hong Kong were affected. As the number of TATA nodes affected increased, so did the number of customer networks affected, including Wells Fargo, Reuters, Oracle, and Amazon. The outage lasted around 34 minutes and was cleared around 8:15 a.m.ET. Click here for an interactive view of the outage.
Update Jan. 19
Worldwide outages in all three categories increased from 157 to 215 over the previous week, an increase of 37%, and were up from 88 to 105 in the US, a 19% increase.
The total number of ISP outages increased from 122 to 160, up 31%, while in the US the increase was 19%, from 88 to 105.
Cloud-provider network outages doubled from 7 to 14 worldwide and remained the same in the US at two.
Globally there were two collaboration-app network outages, up from one, and they remained at one in the US.
There were two notable outages. On January 13, AT&T experienced an outage that affected customers in multiple countries, including the US, UK, Japan, Germany, Canada, Australia, India, and the Netherlands. The outage started around 9:25 p.m. ET and centered on AT&T nodes located in Phoenix, AZ, and last 23 minutes, and was cleared around 9:50 p.m. ET. Click here for an interactive view of the outage.
On January 13, Microsoft experienced an outage that affected some downstream partners and access to services running in Microsoft environments. The 12-minute outage was first observed around 12:15 a.m. ET and occurred in three four-minute occurrences over a 30-minute period. IT centered in Microsoft nodes in Des Moines, IA. It was cleared around 12:45 a.m. ET. Given the timing and uniform pattern of the outage, it is likely to have been an automated maintenance process. Click here for an interactive view of the outage.
Update Jan 12
Global outages in all three categories increased from 96 to 157, up 39% from the week before, and in the US they jumped from 33 to 88, up 167%.
ISP outages worldwide went from 71 to 122, up 72%, and from 30 to 74 in the US, up 147%.
Cloud provider outages increased from two to seven, a 250% increase,
Globally, cloud provider network outages increased from 2 to 7, a 250% increase, and from zero to two in the US.
There was just one collaboration-app network outage and that occurred in the US. There were none the week before.
There were two notable outages during the week. On Jan. 4, Slack experienced an outage at 10 a.m. EST that lasted until after 1:40 p.m. It affected customers worldwide, with many users unable to login, send or receive messages, or to place or answer calls. Slack identified the cause as insufficient router capacity in its cloud-provider network to meet customer demand. Starting at 11:15 a.m. EST Slack implemented a fix, and many customers could use the service again by 12:15 p.m. Slack announced messaging service restoration at 1:40 p.m. EST, although it’s calendar integration features took longer to restore.
On Jan. 7, Cogent Communications experienced an outage at 4:40 p.m. that lasted just under an hour and that affected downstream providers and Cogent customers globally. It consisted of four outage occurrences over a two-hour period, the first of which centered on Cogent nodes in Amsterdam, the Netherlands, mainly affecting European countries. Five later, Cogent nodes in Washington, DC, also exhibited outage conditions. At this point the Amsterdam nodes recovered, but the Washington D.C. nodes stayed down for another 35 minutes. Thirty-five minutes after the first outage cleared, the second outage was observed, centering on nodes in Oakland, CA. It lasted four minutes and affected only customers in the US. This was repeated five minutes later, this time lasting around three minutes. Following a five minute break, a final four-minute outage was observed, this time centering on Cogent nodes in Las Vegas, NV, and Oakland, CA. The outage affected access to services including Amazon, Yandex (Russan based search engine), Oracle, and Sberbank (a state-owned Russian banking and financial services company). The outage was cleared around 6:35 p.m. EST. Click here for an Interactive view of the outage.
Update Jan. 5
Outages in all three categories decreased from 172 to 96, a 44% decrease compared to the week prior. In the US, they decreased from 80 to 33, a 59% decrease.
Globally, ISP outages decreased from 135 to 71, down 47%. In the US, they dropped from 74 to 30, a 59% decrease.
Cloud-provider network outages decreased from five to two, and in the U.S., from two to zero.
There were no collaboration app network outages the previous two weeks.
Update Dec. 21
Total outages across all three categories dropped vs. the previous week, from 252 to 193, a 23% difference. In the US they outages decreased from 115 to 89, also a 23% difference.
Globally, the number of ISP outages decreased from 180 to 145, a 19% decrease, and in the US they decreased from 97 to 75, a 23% drop.
Cloud-provider network outages worldwide decreased from 11 to four, down 64%, while in the US they fell from 2 to 1.
There were three collaboration-app network outages during the week, all in the US. The week before there were four outages, none of them in the US.
There were two notable outages. On Dec. 14 between 6:50 a.m. and 7:30 a.m. EST Google experienced a global outage. ThousandEyes tests measured elevated server wait times, indicating the application was taking longer to respond to service requests. During the service disruption, network paths connecting to Google’s edge servers did not show any traffic loss
The other notable outage hit NTT America and affected some downstream providers and NTT networks in multiple countries including the US, Germany, Brazil, the UK, and Canada. The outage was first observed around 8:30 a.m. EST and appeared to be centered on NTT infrastructure in Los Angeles, California, and Seattle, Washington. The outage lasted just over 19 minutes and was cleared around 8:50 a.m. EST. Click here for an interactive view of the outage.
Update Dec. 14
Total outages in all three categories were up 26%, from 200 to 252, over the previous week, and the were up 39% in the US, from 83 to 115.
ISP outages worldwide increased from 129 to 180, a 40% increase. In the US, ISP outages increased from 66 to 97, a 47% increase.
Worldwide cloud-provider network outages increased from eight to 11, up 38%. In the US they increased from one to two.
Globally, there were four collaboration-app network outages, up from zero. None of them were in the US.
There were two notable outages during the week. On Dec. 10, Hurricane Electric experienced a 17-minute outage that hit users in the US, Canada, Germany, Egypt, Sweden, France, and the UK. The outage was first observed around 2:11 p.m. EST centered on Hurricane Electric infrastructure in Atlanta, Georgia, and 10 minutes later just in Dallas, Texas. The last two minutes affected Hurricane Electric interfaces in both Atlanta and New York, New York. The issue was cleared around 2:38 p.m. EST. Click here for an Interactive view of the outage.
On December 8, Cogent Communications experienced an outage that, though only lasting four minutes, affected multiple downstream providers, as well as Cogent customers globally. The outage was first observed around 4:50 p.m. PST across Cogent’s global infrastructure, with Cogent nodes in the US, Germany, UK, Spain, France, Switzerland, and Ireland all reflecting the outage. The outage affected access to services including Microsoft, Amazon, SAP, Disney Streaming, and Wells Fargo. The outage was cleared around 4:55 p.m. PST. Click here for an Interactive view of the outage.
Update Dec. 7
Worldwide outages in all three categories were up compared to the week before from 159 to 200. In the US they were up from 48 to 83, 73% increase.
Globally ISP outages increased from 119 to 129, up 8%. They were up 65% in the US from 40 to 66.
Overall cloud-provider network outages increased from five to eight, but in the US they dropped from four to one.
For the first time since late September, there were zero collaboration app network outages anywhere in the world.
A notable outage occurred Dec. 2 when Level 3 Communications experienced a 14-minute outage that affected several downstream providers as well as Level 3 customers in the UK and Canada. First observed around 1:10 a.m. PST centered on Level 3 nodes in Seattle, WA. Service was restored to many of the customers and providers after five minutes and the outage was cleared at 1:25 a.m. PST.
Update Nov. 30
During the last week outages worldwide across all three categories decreased from 306 to 159, a 48% drop. In the US, they decreased 75%, from 193 to 48.
Globally, the number of ISP outages decreased by 54%, from 256 to 119. In the US they dropped 77%, from 176 to 40.
Cloud-provider network outages decreased overall from eight to five, a 38% decrease. In the US, they went up one, from three to four.
Collaboration-app network outages decreased from 4 to 1 worldwide, and in the US, the number dropped from 3 to 1.
A notable outage occurred on Nov. 25 when Kinesis, a key AWS service, suffered a day-long outage that affected other AWS services and many of its customers who rely on these services to run their businesses (including iRobot’s Roomba vacuum cleaner app). The outage was not network related, and ThousandEyes tests did not detect an elevation in packet loss during the incident. AWS later described the root cause as related to an operating system configuration in a detailed incident post-mortem.
Update Nov 23