Australia's telecommunications regulator is investigating an outage at Optus after an upgrade went wrong and network monitoring failed to spot the problem.
Australian telecom provider Optus is facing a government investigation after a network upgrade left Australians in three states unable to dial emergency services.
The outage on Thursday hit Australia’s Triple Zero (000) service, the equivalent of the US 911, European 112, or UK 999 emergency numbers, and went undetected for 13 hours, leaving residents of South Australia, West Australia, and Northern territories with no means of contacting an ambulance. A reported three people died after calls to emergency services failed to connect.
The outage was caused by a technical failure in relation to a firewall upgrade, Optus CEO Stephen Rue said in a written statement issued Saturday.
“When the upgrades and changes were implemented, initial testing and monitoring did not indicate there were any issues with calls connecting — normal calls were connecting as they should and call volumes at a national level did not raise any red flags. There was a technical failure in the system, and further, there were no alarms to alert us that some emergency calls were not making it through to emergency services,” he wrote.
On Sunday, he provided more details on the cause of the failure, writing, “Early investigations show that it appears that established processes were not followed. In order to establish why this occurred, we are speaking with the individuals involved.”
The company is conducting an internal investigation to uncover the technical details of the failure, he said, and to address the failure of network monitoring, “I have put in place an immediate halt to further changes in our network system until we have a broader understanding of the events that have occurred so we can also introduce greater monitoring, testing and compliance and reviews of our change processes. Further, our technical teams are monitoring Triple Zero call volumes and failure rates state by state 24 hours, seven days a week.”
Optus, which is owned by Singapore Telecommunications (Singtel), faces a barrage of criticism, particularly because this was not the first such incident. In November 2023, there was a similar episode, resulting in a A$12 million (US$7.9 million) fine. The Australian Communications and Media Authority is now investigating the company again, and its statement pulled no punches.
“Australians must be able to contact emergency services whenever they need help. This is the most fundamental responsibility every telco provider has to the public. When an emergency call fails to connect, the consequences for public health and safety can be devastating,” the regulator said.
After the 2023 outage, the Australian government ordered a review of services, chaired by former deputy chair of the Australian Communications and Media Authority Richard Bean. The subsequent report made 18 recommendations to ensure that there would be no repeat of the outage.
The first of these recommendations mandated network operators to ensure that, in the event of loss of connectivity, calls to Triple Zero would be carried by other networks.
Further recommendations included the requirement that network operators establish a Triple Zero custodian, with responsibility for the efficient functioning of the Triple Zero ecosystem, including monitoring its end-to-end performance. In addition, providers had to conduct six-monthly end-to-end testing of all aspects of the ecosystem within and across networks, which must include the examination of network functionality during different types of outages.
Telecommunications consultant Paul Budde said that it wasn’t just about emergency services. “There have been a number of outages — not just the 2023 one — and it’s not only been Optus that has been struggling.” He is calling for a thorough investigation into the underlying infrastructure in Australia. “It should be an engineer-led investigation, politicians shouldn’t be anywhere near it. The government should set it up but there needs to be a deep technical dive into the infrastructure to find out what’s there and how robust it is.”
Brian Jackson, principal researcher at Info-Tech Research Group, said that it was a striking example of the risks of critical infrastructure failure on society. “While many IT outages only result in lost productivity from downtime, there are some industries where communications are a life and death matter. All telcos can be affected by upgrades to network infrastructure, even if properly executed,” he noted. “[Whether they’re] changes in configurations for a particular purpose or a cascading failure of many systems, other outages will occur at some point without adequate safeguard measures.”
The Optus outage will undoubtedly concern other operators across the world, all of whom will be hoping that their procedures are robust enough to ensure they don’t suffer a similar fate.
Jackson pointed out that such issues were not just about technology, but about operations too. “Employees must be properly trained on how to handle infrastructure upgrades, maintenance, and other activities, and proper oversight needs to ensure failsafes are in place. Even when things are done by the book, there can still be unforeseen risks related to technical glitches, misconfigurations, or system overloads. You just can’t test for everything.”
And companies should not forget their partners, he added. “It’s impossible to do business these days without relying on telcos, cloud service providers, and other digital connection arbitrators. We’ve seen many recent examples of how a misconfiguration with one of these providers can have trickle-down effects through large swaths of the economy. Enterprise leaders must consider how to build resilience against supply chain outages like this one.”
More Optus news:
