Level 3 Communications has cited a "configuration error" as the root cause of its nationwide network outage on Tuesday.
Here's the public statement issued by the Broomfield, Colo., service provider:
On October 4, our voice network experienced a service disruption affecting some of our customers in North America due to a configuration error. We know how important these services are to our customers. As an organization, we’re putting processes in place to prevent issues like this from recurring in the future. We were able to restore all services by 9:31 a.m. Mountain time.
(UPDATED on Oct. 14, 2016) Level 3 got more specific with customers, issuing a Reason for Outage (RFO) Summary (shared by a Network World reader) headlined "Repair Area: Human Error Occurrence" and that read in part: "Investigations revealed that an improper entry was made to a call routing table during provisioning work being performed on the Level 3 network. This was the configuration change that led to the outage. The entry did not specify a telephone number to limit the configuration change to, resulting in non-subscriber country code +1 calls to be released while the entry remained present. The configuration adjustments deleted this entry to resolve the outage."
Social media sites such as Reddit and Twitter erupted on Tuesday morning with inquiries and complaints about the outage from Level 3 customers, as well as customers of other big carriers like AT&T and Verizon that were affected by the outage. Speculation for the outage ranged from possible fiber cuts to more outlandish theories.
Level 3 acknowledged Tuesday morning that it was aware of the issue and was attempting to fix it, and did in fact restore the network during the late morning EST. Though reports of continued issues did trickle out throughout the rest of the day, with organizations affected pointing to Level 3 as the cause of their connectivity problems. Level 3 has done no such finger pointing itself.
Level 3 was brief in its public explanation for the outage -- we're not talking the sort of transparent explanation that some cloud vendors, like AWS, for example, have gotten into the habit of providing during/after outages.
Meanwhile, Level 3 has now moved on to an even more pressing concern: Hurricane Matthew.
Twitter