- Cloudflare admits to causing its own outage – it wasn’t a cyberattack
- Fluctuating error reports made the problem difficult to identify at first
- “Unacceptable” outage led to learning opportunities
Cloudflare shared more details about its November 18 outage – its worst outage since 2019 – confirming that it was not the result of an attack or any other type of malicious activity.
In a blog post, the company’s co-founder and CEO Matthew Prince explained that a database permission change triggered the system to generate a “features file” that doubled in size, before propagating to all machines on its network, causing the software to fail.
After Cloudflare was able to identify what went wrong, normal operation resumed just over three hours after the outage, with full recovery a few hours later.
Cloudflare confirms outage was not an attack
“The main traffic was largely flowing normally at 2:30 p.m.,” Prince wrote, as confirmed by a chat showing a sharp drop in 5xx error HTTP status codes around that time.
However, Cloudflare had to dig a little deeper to find out what exactly was happening due to a fairly high fluctuation range in error reporting. In fact, the problematic file was generated every five minutes.
“In addition to returning HTTP 5xx errors, we observed a significant increase in response latency from our CDN during the impact period,” Prince added, noting that “large amounts of CPU” were being used for debugging and observability.
Cloudflare’s status page was also taken down during the attack – a page completely independent of Cloudflare’s infrastructure. Apparently it was little more than a coincidence.
Still, the outage at least served as a learning opportunity for Cloudflare, which now promises to enable more global kill switches for features.
“An outage like today is unacceptable,” Prince concluded, before putting a slightly positive spin on it: “When we have had outages in the past, it has always led us to build new, more resilient systems.”
Follow TechRadar on Google News And add us as your favorite source to get our news, reviews and expert opinions in your feeds. Make sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, unboxings in video form and receive regular updates from us on WhatsApp Also.
