A simple database permission change triggered a global failure, highlighting how self-inflicted software errors now cause more disruptions than cyberattacks.
A massive outage at content delivery network Cloudflare, which took down services including ChatGPT, Canva and parts of AWS, was traced to a routine change in database permissions linked to Cloudflare’s Bot Management system, which inadvertently knocked offline a significant portion of the web.
The November 18 outage, which lasted approximately six hours, has once again highlighted how ordinary software updates have become one of the leading triggers of large-scale internet outages in recent years.
The issue was triggered by a change to one of Cloudflare’s database systems’ permissions that caused the database to output multiple entries into a feature file used by the Bot Management system, Matthew Prince, co-founder & CEO of Cloudflare, explained in a blog post.
As this feature file propagated across Cloudflare’s global network, its size doubled, exceeding the limits the Bot Management system was designed to handle and causing the system to fail. After identifying the core issue, Cloudflare replaced the larger feature file with an earlier version of the file, which brought the system back to normalcy.
What went wrong
Initially, Cloudflare wrongly suspected the outage to be a hyper-scale DDoS attack as the volume of the 5xx error HTTPS status code served by the Cloudflare network started to fluctuate, which was quite unusual for an internal error.
“The explanation was that the file was being generated every five minutes by a query running on a ClickHouse database cluster, which was being gradually updated to improve permissions management. Bad data was only generated if the query ran on a part of the cluster which had been updated. As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network,” Cloudflare noted in the blog post.
Prince explained that every request to Cloudflare takes a well-defined path within the company’s network. The requests are first terminated at the HTTP and TLS layer, then flow into the core proxy system, and finally through Pingora, which performs cache lookups or fetches data from the origin if needed.
As traffic moves through the core proxy, it applies each customer’s unique configuration and settings, from enforcing web application firewall rules and DDoS protection to routing traffic to Cloudflare Developer Platform and Cloudflare R2 Storage. These configuration and policy rules are enforced by domain-specific modules. One of those modules, Bot Management, resulted in the outage.
The Bot Management module uses a machine-learning model to assign bot scores to every request. The model relies on a feature configuration file, which is a collection of traits that help determine whether a request is automated. This feature file is refreshed every few minutes, allowing Cloudflare to react to new types of bots and new bot attacks.
A change in query behaviour of the database software, ClickHouse, which generates this feature file, created large numbers of duplicate feature rows, increasing the size of the previously fixed-size feature configuration file, causing the bots module to trigger an error. As a result, Cloudflare’s core proxy began returning HTTP 5xx errors for any traffic relying on the bots module. Services like Workers KV and Access were also impacted, as they depend on the same core layer.
Once the underlying issue was identified, Cloudflare halted the generation and propagation of the bad feature file and manually inserted a known good file into the feature file distribution queue, and then forced a restart of its core proxy.
“This is like the butterfly effect in modern interconnected systems. A butterfly can flap its wings to fly, and its wing-flapping is the basis for the butterfly effect, a concept in chaos theory where a tiny change can have massive, unpredictable consequences. Same thing we are seeing here,” said Pareekh Jain, CEO of Pareekh Consulting.
Jain noted that such deep dependencies between security layers and AI modules for traffic routing are now common in large-scale cloud environments. Integrating powerful, dynamic modules directly into request processing pipelines means a flaw anywhere in the path can disrupt everything downstream.
More outages are now self-inflicted
This Cloudflare incident highlights a broader trend as some of the largest platforms now experience more frequent failures due to routine system updates than external attacks. In the last month alone, Microsoft Azure experienced a global outage due to an inadvertent tenant configuration change in Azure Front Door (AFD), and AWS suffered due to a latent defect within the service’s automated DNS management system.
“The real tension sits between the speed at which modern cloud platforms deploy changes and the maturity of the mechanisms that are meant to validate those changes,” said Sanchit Vir Gogia, chief analyst at Greyhound Research. “Cloud and network providers push updates constantly, often through automated pipelines that touch hundreds of services. Yet the dependency maps, failure simulations, and test harnesses behind these pipelines are still catching up to the complexity of the systems they support.”
Gartner’s senior principal analyst Bhuvie Chhabra said that the Cloudflare outage underscores that operational errors arising from any kind of configuration changes in cloud services, not cyberattacks, represent the most significant systemic risk today. “The cascading failure, where a bot-management feature broke core traffic processing, exposes the inherent critical dependency within interconnected layers.”
Cloudflare patches the faultline, but bigger work remains
Apologizing for the disruption caused to its customers and to the Internet at large, Cloudflare said that beyond fixing the immediate problem, it has begun work on preventing similar incidents in the future.
The company is hardening the ingestion of Cloudflare-generated configuration files in the same way it validates user-generated input. It is also enabling more global kill switches for features, eliminating the ability for core dumps or other error reports to overwhelm system resources and reviewing failure modes for error conditions across all core proxy modules.
“These actions address the immediate root cause. These are good first steps. Cloudflare’s fixes are necessary but not sufficient. The outage exposed a far deeper issue,” said Gogia. “The broader challenge sits with enterprises. They must design systems that operate cleanly even when a major intermediary stumbles. That means more isolation between critical application flows and optional security modules, more graceful degradation modes, and more explicit regional containment of feature rollouts. It also means treating multi-provider architectures as strategic investments rather than discretionary costs.”
Chhabra added that while Cloudflare’s remediation is crucial, the wider industry must resist overreacting by partitioning applications or providers and instead prioritize pragmatic resilience. They should apply diversification sparingly and only for critical systems where downtime has a material business impact.
