On July 4 about 5 p.m. PDT Comcast suffered a 33-minute outage affecting U.S. uses and those in multiple other countries trying to access services using the Comcast network. The outage was caused by two events over a 40-minute period and affected Comcast nodes on the U.S. east and west coast and the central region. The outage was cleared at 5:45 p.m. PDT.
Update June 29
The total number of global outages for the week of June 22-28 decreased by 29% from the week prior, reaching the lowest number of outages observed since early April. In the U.S., the number of outages was down by 20%.
ISP outages were also down to the lowest levels recorded in the past eight weeks. Globally, the number was down by 26% this week, dropping from 216 to 160. In the U.S., ISP outages were down by 20% compared to last week, from 96 to 77.
Globally, cloud-provider outages decreased by 39% this week from 46 to 28, with the bulk being attributed to South America. In the U.S. cloud provider outages were down by 55% from 11 to five compared to the week prior.
Globally, last week saw zero collaboration app network provider outages for the second week in a row.
Comcast Cable Communications suffered a 24-minute outage affecting users across the U.S. accessing services including Zoom, Visa and Bank of America. The outage was focused on Comcast infrastructure located in Seattle, Wash., and was cleared just after 2:30AM PDT.
Update June 22
Cloud provider outages spiked to new record-level highs for the week of June 15-21. Globally, the number of cloud provider outages increased from 20 to 46, a 130% increase. In the U.S., the number of outages increased 175%, from 4 to 11.
Last week also saw record-level lows. For the first time since the week of February 24, there were zero collaboration app network provider outages both globally and in the U.S.
Globally, the number of ISP outages decreased marginally last week, dropping from 221 to 216. In the U.S., however, the number of ISP outages increased by 22%, compared to the week prior.
From a total outage perspective, the number decreased marginally globally, from 287 to 282. The U.S., however, saw a 14% increase in outages relative to the week prior. From 99 to 113 outages.
An outage of note occurred June 18 at 2:45 PDT and lasted 23 minutes, affecting multiple countries including Australia, France, Germany and the U.K. The outage affected access to Microsoft services including some identity systems and appeared to originate in Microsoft nodes in Des Moines, Iowa. The outage was divided into two outages over two hours, concluding just after 5 p.m. PDT. Click here for an interactive view of the outage.
Update June 15
During the week June 8-14, worldwide the number of total outages in all three categories rose 35%, and jumped 34% in the U.S.
ISP outages globally increased 32% from 168 to 221 and rose 14% in the U.S. from 68 to 79.
Cloud-provider outages decreased from 23 to 20 (-13%) and doubled from two to four in the U.S.
Global collaboration-network outages quadrupled from one to four worldwide and remained flat in the U.S. with one outage.
Verizon Business suffered a two-minute outage June 10 at 2:50 p.m. PDT that affected users in multiple countries trying to access services from Microsoft, Zoom and Amazon, among others. The outage centered on infrastructure in Seattle. The brevity of the problem indicates the systems were reacting automatically to address a fault.
Update June 8
During the week of June 1-7 he number of outages across all three categories of providers decreased 15% (249 to 212) worldwide vs. the week before, and dropped 29% in the U.S. (104 to 74).
ISP outages worldwide fell from 196 to 168, and in the U.S. the decline was from 91 to 68.
Public-cloud outages globally dropped from 26 to 23, and from six to two in the U.S.
Outages for collaboration-app networks all took place in the U.S. over the past two weeks and fell from seven to one.
Cable operator Spectrum sustained an outage June 2 that affected multiple channels across the U.S., with East Coast viewers suffering the worst of it for about three hours from just after 9 p.m. Eastern to around midnight.
Update June 1
The total number of outages worldwide among all three categories decreased 11% from 280 to 249 during the week of May 25-31, and declined 10% in the U.S. from 115 to 104.
The number of ISP outages decreased 13% globally from 225 to 196. In the U.S. they dropped from 109 to 91, a 17% reduction.
Public-cloud outages dropped from 35 to 26 worldwide but increased in the U.S. from two to six.
Collaboration app network provider outages totaled seven worldwide, up from four the prior week, and in the U.S. jumped seven-fold from one to seven.
Amazon suffered an outage May 28 startinn about 12:12 p.m. PDT when its website because unreachable for some users globally due to a DNS failure, making it impossible for some users to reach Amazon’s servers. The DNS servers were available at the time, so the problem was likely a misconfiguration. The issue was resolved by 12:38 p.m. PDT.
Update May 25
ISP outages jumped by more than a third in the U.S. during the week ending May 24, while outages among all three categories of provider registered a small increase.
Total outages among all categories rose from 263 to 280 globally, and from 86 to 115 in the U.S.
ISP outages rose from 223 to 225 worldwide, with most of the increase due to outages in the U.S., which jumped from 80 to 109.
Public cloud outages overall rose from 24 to 35, with U.S. outages only ticking up from one to two.
Collaboration app network outages dropped from four to five compared to the week before, with a drop in U.S. outages from five to one accounting for the improvement.
There were two noteworthy outages during the week:
- Just after 3 a.m. EDT on May 20, Google suffered an outage in the East Coast part of its network that affected users accessing site such as Uber and Shopify that are hosted by the public cloud provider. The outage lasted nine minutes and was located in the New York City metro area, and since it was during off-peak hours, impact on users was likely minimal. Click here for an interactive visualization of the outage.
- About 8 a.m. EDT on May 22, Hurricane Electric suffered an outage that lasted more than an hour and affected several countries. The worst part lasted 44 minutes. The outage was observed at Hurricane Electric nodes across multiple global locations, and affected users reaching sites including Microsoft, Amazon, Workday and Credit Suisse. Click here for an interactive visualization of the outage.
Update May 18
The total outages globally leapt up 22% between the week of May 4-10 and the week of May 11-17, from 216 to 263.
ISP outages worldwide grew from 183 to 223, while those outages in the U.S. moved up from 74 to 80.
Public cloud outages grew from 13 to 24 worldwide, but dropped from three to one in the U.S.
Collaboration app providers dropped worldwide from six to five, while for the third week in a row the U.S. outages totaled five.
One noteworthy outage found users around the world unable to load content hosted by YouTube for about half an out on May 14 starting about 4 p.m. PDT. Users could connect to YouTube servers, but the service itself didn’t load properly. ThousandEyes said a critical object on the site was responding erroneously, indicating a web-application issue, perhaps due to a site update or change.
Update May 11
Overall outages dropped for the second week in a row, from 282 the week before last to 216 (23%) last week. In the U.S. outages fell from 98 to 83 (15%).
ISP outages worldwide were down 22% (from 236 to 183) and in the U.S. they fell 18% (from 90 to 74).
Cloud networking outages dipped from 13 to 12 (7%) worldwide, and in the U.S. dropped from 3 to 1 (66%)
Collaboration-application network outages fell worldwide from seven to six (14%) and stayed steady in the U.S. at five.
ISP Cogent Communications suffered a notable 38-minute outage starting on May 5 about 12 a.m. PDT, affecting its infrastructure in the U.S., U.K., Canada and France. Traffic terminated in Cogent’s network, and users were unable to reach sites such as Amazon, Microsoft, WorldPlay and Oracle Cloud.
Update May 3
Overall, the number of outages dropped across the board last week, from 313 to 282, a decrease of 31worldwide, mainly due to a drop in overall U.S. outages from132 to 98.
Most of that was driven by a decline by at least half of outages for both ISPs and collaboration providers.
Total ISP outages dipped from 250 down to 236 globally, while ISP outages in the U.S. went from 124 to 90. That’s the second week in a row of declines.
Worldwide, collaboration provider outages dropped from 14 to seven, but rose from three to five in the U.S.
Public-cloud outages also declined week-over-week from 26 to12 globally and from four to one in the U.S.
Virgin Media suffered a noteworthy outage during the evening of April 27 in the UK, Ireland and the Netherlands. It started at 5:15 p.m. local time and lasted 15 minutes, then again at 6:15 for another 15 minutes. The pattern repeated several more times, transitioning to briefer outages that ended by 1:30 a.m. April 28. Based on the pattern, Thousand Eyes suggests that an automation issue could have been the root cause, though no official reason had been made public.
Update April 27
Globally, outages hit a record high during the week of April 20-26 – 313 – up 11% from the week prior and up 77% from the temporary decrease the week of April 6. The number of outages is the most since the end of March, but two issues – fiber cuts in CenturyLink’s network and a broad Tata Communications outage – helped push that number up. Outages in the U.S. hit record numbers, too, at 132.
The ISP outages worldwide tallied 250, and in the U.S. spiked to 124, a new high including the CenturyLink and Tata problems.
After a two-week downward trend, cloud provider outages globally increased to 26, the same levels registered in late March and early April. In the U.S. they went down slightly from six to four. Overall, these numbers are still in the normal range and in general, cloud providers continue to hold up well.
Collaboration-application-network outages increased slightly over the previous week from 11 to 14 but continue to remain low relative to the peak of 29 observed in the week of March 30-April 5. U.S. outages dipped from four to three.
The major outage in the Tata Communications network on April 20 affected its infrastructure in the U.K., France, Germany, and India. Around 11 a.m. local U.K. time, traffic attempting to reach services such as Amazon, ServiceNow, and Oracle Cloud began terminating in its network, affecting local users, users in the U.S. and elsewhere. The outage lasted about 20 minutes, and affected more than 80 network interfaces across multiple regions and cities.
Another far reaching outage occurred the next day in the U.S., when at least one fiber cut within CenturyLink’s network in Southern California affected enterprises and consumer users up and down the West Coast, and as far away as Raleigh, NC. It affected the Level 3 part of CenturyLink’s network, a transit provider it acquired in 2017. Merrill Lynch reported a disruption to its business as a result of the network outage, during which its brokers were intermittently unable to access their workstations. The incident started hitting enterprises and their users around 10 a.m. ET, with most disruption resolved by around 11:30 a.m. ET.
Update April 20
Total outages spiked 58% during the week of April 13-19 fueled by one prolonged outage that had a significant effect on multiple ISPs.
That one outage affected TeliaNet, Level 3, AT&T, and other ISPs on April 13. TeliaNet was the most affected of the group, and it’s not clear whether it was the cause of the outage,ThousandEyes says. During the downtime, at least one application provider withdrew route through the TeliaNet network until the next day.
Had that one outage not occurred, the total number of outages for the week would have been in the low 200s, which ThousandEyes says is in the normal range. As it turned out, total outages rose 59% from 177 to 282.
ISP outages jumped from 141 to 243 week over week, up 72% worldwide, and from 56 to 98 (75%) in the U.S.
Public cloud outages dropped off from the week before, from 19 to 14 (down 36%) worldwide, and stayed steady at six outages in the U.S.
Global application-provider networks had a slight increase in outages worldwide, up from 9 to 11 (22%), but dropped from 9 to 4 in the U.S., down 55%.
In other major outages,ThousandEyes said it appeared that several banks effectively suffered denial of service conditions when customers apparently flooded their sites seeking to find out whether they’d received their pandemic-related stimulus checks. Content-delivery networks serving the banks didn’t have network issues yet were unable to return Web content for many banking sites, “likely due to bank origin servers unable to handle the high volume of requests,” ThousandEyes says.
Update April 13
During the week April 6-Apri 12, service outages for ISPs, cloud providers, and conferencing services dropped overall. They went from 298 down to 177 globally (40%, a six-week low), and in the U.S. dropped from 129 to 72 (44%).
Globally, ISP outages were down from 229 to 141 (38%), and in the U.S. were down from 100 to 56 (44%).
Cloud provider outages were also down overall from 25 to 19 (24%), ThousandEyes says, but jumped up from one to six (500%) in the U.S., which saw the highest rate of increase in seven weeks. Even so, the U.S. total was relatively low. “Again, cloud providers are doing quite well,” ThousandEyes says.