Feed: Slashdot
Feed Link: https://slashdot.org/
---

Title: Cloudflare Explains Its Worst Outage Since 2019

Link: https://tech.slashdot.org/story/25/11/19/1447...

Cloudflare suffered its worst network outage in six years on Tuesday,
beginning at 11:20 UTC. The disruption prevented the content delivery network
from routing traffic for roughly three hours. The failure, writes Cloudflare
in a blog post, originated from a database permissions change deployed at
11:05 UTC. The modification altered how a database query returned information
about bot detection features. The query began returning duplicate entries. A
configuration file used to identify automated traffic doubled in size and
spread across the network's machines. Cloudflare's traffic routing software
reads this file to distinguish bots from legitimate users. The software had a
built-in limit of 200 bot detection features. The enlarged file contained
more than 200 entries. The software crashed when it encountered the
unexpected file size. Users attempting to access websites behind Cloudflare's
network received error messages. The outage affected multiple services.
Turnstile security checks failed to load. The Workers KV storage service
returned elevated error rates. Users could not log into Cloudflare's
dashboard. Access authentication failed for most customers. Engineers
initially suspected a coordinated attack. The configuration file was
automatically regenerated every five minutes. Database servers produced
either correct or corrupted files during a gradual system update. Services
repeatedly recovered and failed as different versions of the file circulated.
Teams stopped generating new files at 14:24 UTC and manually restored a
working version. Most traffic resumed by 14:30 UTC. All systems returned to
normal at 17:06 UTC.

Read more of this story at Slashdot.

---
VRSS v2.1.180528