The airline confirmed that the failure of a third-party piece of hardware grounded flights, underscoring the fact that even systems that are multiply-redundant can go down.
An outage that grounded flights at Alaska Airlines for three hours on Sunday wasn’t the result of a cyberattack, but a hardware failure in one of the company’s data centers.
“A critical piece of multi-redundant hardware at our data centers, manufactured by a third-party, experienced an unexpected failure,” affecting several key systems, Alaska Airlines told Network World in an email.
Initial speculation, dismissed by the airline, was that the outage was caused by a security incident, particularly as Hawaiian Airlines, which Alaska Airlines acquired last September, was hit with a cyber attack in June, and as cybercriminals increasingly have set their sights on the aviation industry.
The outage underscores the fact that even multi-redundant systems aren’t foolproof, and can go wrong all on their own.
“I think they emphasized redundancy to make it clear that they believe they invested appropriately in the solution: ‘We anticipated this with redundant systems and were still impacted!’” said Jeremy Roberts, senior director of content and research at Info-Tech Research Group.
Hardware failure halts flights for three hours
Alaska Airlines announced the IT outage at around 8 p.m. Pacific Time on July 20 (Sunday), saying it was affecting its operations and had resulted in a “system-wide ground stop of flights.” It lifted the stop three hours later, at 11 p.m.
The airline had to cancel more than 150 flights since Sunday evening, including 64 on Monday. It said additional flight disruptions are likely as it repositions aircraft and crews throughout its network.
On Monday, the airline said it was working with its vendor to replace the faulty equipment at the data center.
Alaska Airlines emphasized that the safety of its flights was never compromised, and that “the IT outage is not related to any other current events, and it’s not connected to the recent cybersecurity incident at Hawaiian Airlines.”
The airline did not provide additional information to Network World about the specifics of the outage.
“There are many redundant components that can fail,” said Roberts, noting that it could have been something as simple as a RAID array (which combines multiple physical data storage components into one or more logical units). Or, on the network side, it could have been the failure of a pair of load balancers.
“It’s interesting that redundancy didn’t save them,” said Roberts. “Perhaps multiple pieces of hardware were impacted by the same issue, like a firmware update. Or, maybe they’re just really unlucky.”
While Alaska Airlines hasn’t released the total cost of this outage (and likely hasn’t even determined it yet), “it’s easily in the millions,” said Roberts.
Of course, he noted, there’s really no definitive answer until (or if) Alaska Airlines releases a post-mortem.
Redundancy, redundancy, redundancy
Investment in redundancy is a critical business practice, and the Alaska Airlines incident emphasizes the importance enterprises should place on specifying, managing, and maintaining such equipment, said Info-Tech’s Roberts.
“Redundancy is (or should be) ubiquitous,” he said. “It should not be possible to run an enterprise service without some form of redundancy pretty much all the way down the stack.”
Multiple switches, load balancers, and internet connections are all standard when an enterprise has a service level to maintain, he said. “If the system is critical, you should have an active/active configuration so you can fail over in real time.”
He pointed out that redundancy is why, when Elon Musk got frustrated with the speed of a data center move and began unplugging servers at Twitter, the website didn’t go down. And some enterprises actively test their multi-redundant systems: Netflix, for one, uses and open-sources a tool called Chaos Monkey that randomly kills production instances to make sure systems are fault tolerant.
Ultimately, as evidenced by Alaska Airlines, “there’s a lot at stake,” said Roberts.
