Azure and other Microsoft cloud services were disrupted for five hours when IP routes were accidentally removed from networking equipment.
Microsoft cloud and Azure services hosted on the West Coast of the US went down for hours on Thursday when network connectivity failed. Although services running entirely within Microsoft’s West US cloud region were unaffected, any traffic entering or leaving the facilities was affected.
Microsoft has now published a Preliminary Post Incident Review (PIR) of the incident, reporting that connectivity was lost for five hours between 14.44 UTC (7.44 a.m. Pacific Time) and 19.41 UTC on July 23. The problem was caused when a set of IP routes was removed in error while isolating a device for routine maintenance.
Before starting the maintenance work, Microsoft checked that at least one of the two redundant paths to the facility remained operational. When it came to starting the work, however, automated systems included some additional devices in the perimeter to be isolated, and removing some IP routes that had not been included in the initial assessment.
Customers discovered the problems very quickly, and engineers identified the issue within the first hour and started to reconnect services. Microsoft said the disruption had been caused by some “recent fiber maintenance activity”.
To minimize the risk of disruption from such errors in the future, Microsoft advised organizations handling mission-critical data to consider a multi-region approach.
The Azure outage was the second significant one to hit Microsoft this year. In February, there was a 10-hour disruption to US West and US East regions.
