Table of contents of the article:
On November 18, thousands of websites protected by Cloudflare suffered severe and widespread outages, giving the impression that “half the Internet” was down . On the surface, it might have looked like a cyberattack or a global infrastructure issue, but the reality was far more startling: a change to a database's permissions triggered a domino effect that led to the repeated crashes of one of Cloudflare's most critical services. The database in question was ClickHouse , a high-performance columnar database used for real-time analytics, chosen for its ability to run complex queries on large volumes of data in extremely short times. It's a powerful technology, but—like all global and shared components—it can become a point of vulnerability if something changes unpredictably.
What exactly happened
Cloudflare Bot Management is based on a machine learning model that analyzes each HTTP request by evaluating a set of features collected from internal systems. These features are extracted via a query on ClickHouse and then included in a file that is updated every 5 minutes and distributed across the global Cloudflare network. It is therefore a highly dynamic and distributed mechanism : any error in the generation or distribution step propagates within minutes.
Changing database permissions
The Cloudflare team was implementing an internal security process improvement: moving away from shared accounts in favor of dedicated user accounts and explicit permissions. This step involved updating permissions on the ClickHouse database used by Bot Management. The operation seemed completely harmless. No one would have imagined that a change in access could alter the behavior of a query that had been operating for years. And yet, that's exactly what happened.
The hidden side effect: duplicate data
Before the change, the query returned approximately 60 features from the default database. After the permission change, ClickHouse began reading data not only from the “default” database, but also from the “r0” database. The result was an output of over 200 features . The Bot Management module had a hard-coded limit of 200 items . When the list exceeded the threshold, the service did nothing, did not log any errors, did not trigger any fallbacks or controlled degradations: it simply crashed everything . A theoretically unlikely condition, never seen before, suddenly became reproducible every five minutes, depending on the node generating the feature file.
Because it looked like a DDoS attack
The configuration file was being regenerated and redistributed every five minutes. This meant that some instances on the Cloudflare network were receiving a correct file, while others were receiving a corrupt one. The result was intermittent and erratic behavior : services crashing and restarting, nodes asynchronously cycling in and out of error conditions. This pattern is also typical of a distributed DDoS attack , especially when it affects multiple independent regions. To make matters worse, Cloudflare's status page also went offline during the incident , due to a completely unrelated issue. A deadly coincidence, which led engineers down the wrong path for over two hours.
Diagnosis and resolution
It wasn't until around 14:30 PM that the team identified the source of the problem: a configuration file containing more than 200 features, generated by queries modified following ClickHouse's permission review. Once the cause was identified, engineers stopped the propagation of the faulty file and deployed a version known to be fixed . Full recovery was completed at 17:06 PM, approximately two and a half hours after the fix.
Technical Lessons: What Engineers Need to Take Home
1. Unwrap() kills systems
The most serious problem was not the number of features, but the way the software reacted to the anomaly. The offending module used .unwrap() in Rust: a convenient choice for rapid development, but extremely dangerous for mission-critical systems. Unwrap() assumes that the operation always succeeds. If something goes wrong, it causes a panic which interrupts the entire service, without logs, without countermeasures, without diagnosis. If the system had logged a simple error message instead of crashing, the problem would have been identified within minutes. A single .unwrap() in a global pipeline can cause a international incident.
2. Global database changes are fragmentation grenades
A permissions change seemed harmless. But in a complex, distributed system, even what seems impossible can generate side effects in subsystems that had no direct connection to the change. Staging environments never perfectly replicate a global infrastructure. The idea that a simple change could cause duplicate query results wasn't among the anticipated risks. Yet that's exactly what happened. The lesson is simple: any global change can generate unexpected emergent behavior.
3. Coincidences are more misleading than bugs
When two unrelated problems occur at the same time, the mind creates connections that don't exist. The Bot Management crash and the offline status page made the hypothesis of an external attack credible. Cloudflare is bombarded daily with attack attempts: starting from that hypothesis is natural. However, it cost 2,5 hours of investigation into a nonexistent problem. The lesson: data always trumps narrative.
4. CDNs are real single point of failure
The incident highlighted a difficult fact to accept: the internet is massively dependent on a few global players. When a CDN like Cloudflare fails, much of the web is immediately impacted. Most companies don't have a second CDN ready, nor do they have an autonomous infrastructure to absorb traffic. Multi-CDN redundancy exists, but it's expensive, complex, and often impractical. The dependency is real, and the November 18 incident made it clear again.
Conclusion
The November 18th outage was a textbook case showing how seemingly irrelevant details can have enormous effects in global systems. A change in database permissions changed the behavior of a single query. The unexpected output exceeded a hard limit. A critical module relied on .unwrap() and reacted with total panic rather than a managed error. The global distribution of the faulty file every five minutes amplified the impact. An external coincidence led to a prolonged misdiagnosis. And so, from a single technical detail, tens of millions of websites have gone offlineAnd that's why, in complex architectures, it's not the attacks that are the most scary: it's the small, unexpected changes. Or worse yet, it's the .unwrap() hidden in the code.
