Weekly internet health check, US and worldwide

ThousandEyes, which tracks internet and cloud traffic, provides Network World with weekly updates on the performance of three categories of service provider: ISP, cloud provider, UCaaS.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 Page 3
Page 3 of 14

Around 9 a.m. EST on Feb. 22, Slack experienced disruption to its business-communication platform that lasted around 3 hours and 14 minutes and impacted users accessing its messaging services. During the interruption, a number of application-based errors were observed, indicating that network connectivity to Slack was intact, and the problem resided within the back-end architecture. This was later confirmed by Slack, which identified the cause as a configuration change that inadvertently led to a sudden increase in activity on the Slack database infrastructure. That left some databases unable to serve incoming requests. Slack applied a combination of rate limits and a temporary redirection of requests to replica databases, allowing the system to recover. The outage was cleared around 12:14 p.m. EST.

Around 10:06 p.m. EST on Feb. 24, PCCW experienced an outage impacting some of its ISP customers and networks in countries including, the US and China. It appeared to center on PCCW nodes located in Ashburn, Virginia, and was cleared around 10:20 p.m. EST. Click here for an interactive view.

Updated Feb. 21

Global outages across all three categories decreased from 271 to 205, down 24% from the week before, while US outages decreased from 119 to 87, down 27%.

Globally, ISP outages decreased from 191 to 156, an 18% decrease, and the US outages dropped from 96 to 70, a 27% decrease.

Cloud provider outages worldwide dropped from 10 to eight, and in the US from six to four. 

Globally, collaboration-app network outages decreased from 13 to 10, while in the US they increased from two to five.

There were two notable outages during the week.

On Feb. 17, Level 3 Communications experienced an outage that impacted multiple downstream partners and customers across the US for a total of 28 minutes, divided into two occurrences distributed over an hour and 35 minutes. The first occurrence was observed around 2:40 p.m. EST centered on Level 3 nodes in Salt Lake City, Utah. An hour and five minutes after appearing to clear, the Salt Lake nodes began exhibiting outage conditions again that lasted nine minutes. The outage was cleared around 4:15 p.m. EST. Click here for an interactive view.

On Feb.17, Oracle experienced an outage on its network that affected customers and downstream partners interacting with Oracle Cloud services in countries including the US, Hong Kong, Australia, and Brazil. The outage was observed around 3:05 p.m. EST centered on Oracle nodes in Phoenix, Arizona, and Sao Paulo, Brazil. Five minutes later, the Sao Paulo nodes appeared to recover, reducing the number of affected countries to the US, Hong Kong, and Australia. The outage lasted 18 minutes and was cleared around 3:25 p.m. EST. Click here for an interactive view.

Updated Jan. 17

Global outages all three categories last week increased from 225 to 271, up 20% while in the US they rose from 104 to 119, a 14% increase.

The number of ISP outages globally increased from 151 to 191, up 26%, and in the US they were up from 79 to 96, a 22% increase.

Cloud-provider network outages decreased from 11 to 10. In the US they decreased from seven to six.

Globally, collaboration-app network outages decreased from 14 to 13 while in the US they dropped from three to two.

On Feb. 9, Oracle experienced an outage affecting Oracle Cloud customers and partners interacting with those services in countries including the US, Japan, Germany, Ireland, India, Canada, Brazil, Belgium, the Netherlands, Malaysia, Finland, the UK, Sweden, Poland, Spain, Australia, New Zealand, South Africa, Hong Kong, Austria, Russia, Turkey, Hungary, Taiwan, Greece, Portugal, Ukraine, and China. The outage came in three waves over a space of 3 hours and 15 minutes. The first occurrence lasting about 14 minutes started around 8:20 p.m. EST and centered on Oracle nodes in San Jose, California; Phoenix, Arizona; Jerusalem, Israel; Hyderabad, India; and Tokyo, Japan. An hour and 14 minutes after that, nodes in San Jose; Hyderabad; Amsterdam, the Netherlands; and Sao Paulo, Brazil, exhibited outage conditions. After 15 minutes nodes in Sydney and Melbourne, Australia; Singapore; Ashburn, Virginia; Washington, DC; Frankfurt, Germany; San Francisco, California; Toronto, Canada; and Phoenix, Arizona; exhibited error conditions as well. That second occurrence, lasted an hour and 4 minutes, appearing to clear around 11:20 p.m. EST. After 10 minutes, nodes in San Jose and Hyderabad once again exhibited outage conditions for 4 minutes, affecting customers in the US, Poland, and the UK. That phase of the outage cleared at 11:35 p.m. EST, making the total time of the outages an hour and 22 minutes. Click here for an interactive view.

On Feb. 8, Time Warner Cable experienced a disruption that affected customers and partners in countries including the US, Canada, France, Hong Kong, the UK, India, Singapore, Australia, Germany, Ireland, Malaysia, Brazil, the Netherlands, Italy, Indonesia, Japan, Mexico, and Republic of Korea. Outages occurred in five periods over the course of an hour and 15 minutes. The first period started at 1:30 p.m. EST and lasted four minutes and appeared to center on nodes in Denver, Colorado. About 20 minutes after that a second, 24-minute occurrence was observed on nodes in Los Angeles, California, and 10 minutes into that second period, nodes in Las Vegas, Nevada, and Denver Colorado, also began exhibiting outage conditions. Around 2:15 p.m. EST, the Los Angeles, Las Vegas, and Denver nodes appeared to clear. Five minutes later, at around 2:20 p.m EST, the Los Angeles and Denver nodes exhibited outage conditions again in three, 4-minute bursts over a 40-minute period. It was cleared around 2:45 p.m. EST. Click here for an interactive view.

Updated Jan. 31

Global outages across all three categories last week decreased from 243 to 198, down 19% compared to the week prior. In the US, outages increased from 81 to 86, up 6%.

Globally, the number of ISP outages dropped from 188 to 144, down 23% while in the US they decreased from 68 to 66, down 3%.

Cloud provider network outages globally to increased from six to seven and from four to five in the US.

Globally, collaboration-app network outages dropped from 11 to three and from three to one in the US.

On Jan. 27, Hurricane Electric experienced an outage that impacted customers and downstream partners across countries including the US, the UK, France, Hong Kong, India, Japan, Germany, Singapore, China, Malaysia, and Canada. The outage was divided into three occurrences over a five-minute period. The first period, lasting around 9 minutes, was observed around 12:25 a.m. EST, centering on Hurricane Electric nodes in New York, New York, and Chicago, Illinois. Around 15 minutes after the first occurrence appeared to clear, the second was observed on nodes in Chicago. Around 1 a.m. EST, the Chicago nodes appeared to recover, but 10 minutes later, they and nodes in New York began exhibiting outage conditions again. This occurrence lasted around a minute before appearing to clear. The total outage lasted around 26 minutes and was cleared around 1:25 a.m. EST.

Updated Jan. 24

Global outages across all three categories increased from 236 to 243 last week, a 3% increase compared to the week prior. In the US, outages decreased from 86 to 81, down 6%.

Globally, the number of ISP outages increased from 173 to 188, up 9%, and in the US they increased from 66 to 68, up 3%.

Global cloud-provider outages decreased from 10 to six, and in the US they dropped from five to four.

Collaboration app network outages worldwide increased from six to 11, while in the US they remained at three for the third week in a row.

There were two notable outages last week.

On Jan. 20, Cogent Communications, experienced a series of outages over a period of an hour and 28 minutes that impacted customers globally and multiple downstream providers. The 40-minute outage was first observed around 10:08 p.m. EST and initially centered on Cogent nodes in Denver, Colorado, and Salt Lake City, Utah. This initial outage lasted around a minute and the Cogent environment remained stable for 15 minutes before experiencing a 9-minute outage observed on Cogent nodes in Dallas, Texas, and Phoenix, Arizona. An hour and a half after the initial outage, a 4-minute outage was observed on Denver nodes. Five minutes after that outage appeared to clear nodes in Denver began exhibiting outage conditions again. Fifty-two minutes after first being observed, the outage reappeared, affecting nodes in Sacramento and Oakland, California, Salt Lake City, and Denver, affecting more and more customers and providers. This occurrence lasted 17 minutes before appearing to clear around 11:20 p.m. EST. Five minutes later, Salt Lake City nodes exhibited outage conditions before clearing and being replaced by nodes located in Sacramento. The outage was cleared around 11:35 p.m. EST. Click here for an interactive view of the outage.

On Jan. 17, PCCW, a Hong Kong-based Tier 1 ISP, experienced an outage impacting some of its customers and networks in countries including the US, Singapore, Thailand, and China. The outage was first observed around 9:15 p.m. EST and appeared to center on PCCW nodes in Singapore. It lasted 34 minutes and was cleared around 9:50 p.m. EST. Click here for an interactive view of the outage.

Updated Jan. 17

Global outages across all three categories increased from 206 to 236 last week, up 5% compared to the week before. In the US they increased from 78 to 86, up 10%

Globally, ISP outages increased for the second consecutive week, rising from 150 to 173, up 15%, while in the US they increased from 57 to 66, up 16%.

Cloud-provider network outages worldwide jumped from two to 10, and from two to five in the US.

Globally, collaboration-app network outages doubled from three to six, and remained the same at three in the US.

Notable outages

On Jan. 10, Cogent Communications experienced an outage that affected some of its downstream providers and customers in countries including the US and Mexico. The outage was first observed around 10:20 p.m. EST and appeared to center on nodes in Phoenix, Arizona. The outage lasted 14 minutes and was cleared around 10:35 p.m. EST. Click here for an interactive view of the outage.

On Jan.12, Hurricane Electric, experienced an outage affecting customers and downstream partners across regions including the US, Hong Kong, Japan, Canada, Australia, the Philippines, and Thailand. The 14-minute outage was first observed around 1 p.m. EST and appeared to center on Hurricane Electric nodes in Minneapolis, Minnesota, and Seattle, Washington. Five minutes later, the Minneapolis nodes appeared to recover and those in Tokyo, Japan, exhibited outage conditions. This coincided with an increase in the number of downstream partners and countries impacted. Around 1:10 p.m. EST, the Tokyo nodes appeared to clear. The outage was cleared around 1:15 p.m. EST. Click here for an interactive view of the outage.

Update Jan. 10

Global outages across all three categories last week increased from 180 to 206, up 14% compared to the week prior. In the US, outages increased from 48 to 78, a 63% increase.

Globally the number of ISP outages increased from 121 to 150, up 24%, and in the US they increased from 40 to 57, up 43%.

Cloud-provider network outages dropped from five to two worldwide, but in the US they increased from zero to two.

Globally, collaboration-app network outages decreased from five to three, and rose from two to three in the US.

There were two notable outages during the week.

On Jan. 3, Oracle experienced an outage on its network affecting Oracle Cloud services in the US. The outage was divided into two occurrences over a 25-minute period. The first occurrence was observed around 5:20 a.m. EST centered on Oracle nodes in Austin, Texas, and lasted about a minute. Five minutes after appearing to clear, they began exhibiting outage conditions again that lasted 12 minutes. The outage was cleared around 5:35 a.m. EST.

On Jan. 5, Microsoft experienced an outage on its network that affected access to services running on Microsoft environments. The outage, first observed around 11:20 p.m. EST, lasted nine minutes and appeared to be centered on Microsoft nodes in Chicago, Illinois. Five minutes later, a number of the Chicago nodes appeared to recover, reducing the number of affected partners. The outage was cleared around 11:30 p.m. EST. Given the duration, timing, and location of the nodes the cause is likely to have been a maintenance exercise.

Update Dec. 13

Global outages across all three categories last week increased from 287 to 356, up 24% from the week prior. In the US, outages increased from 103 to 130, up 26%.

Globally, the number of ISP increased from 209 to 261, up 25%, while in the US, they increased from 85 to 108, up 27%.

Cloud-provider network outages worldwide dropped from 28 to 27. In the US, they increased from two to five.

Globally, collaboration-app network outages jumped from five to 13 outages, while in the US they increased from three to seven.

On December 7, AWS experienced an outage disrupting users and customers accessing its services in regions across the globe. The outage was first observed around 10:40 a.m. EST and appeared to be centered on infrastructure in the AWS US-EAST-1 region, located in Northern Virginia. It initially affected services relied upon by non-Amazon apps and services, across regions including in the US, Europe, and APJC. The impact varied depending on the user's IP address. At around 12:37 p.m. EST, Amazon announced it had identified issues related to an application programming interface (API) and were working on recovering services. At 5:43 p.m. EST, Amazon announced it had mitigated the underlying issue, and services began to return to normal. Around 6:03 p.m. EST, most services had been restored, with some disruption still being experienced on the AWS API gateway service. The disruption can be divided into two occurrences, with the first appearing to be the most prominent and lasting around an hour and four minutes; many of the services were restored around 11:44 a.m. EST. The second, lasting around 8 hours, was cleared around 8 p.m. EST.

Related:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 Page 3
Page 3 of 14