The internet was brought to its knees by a massive outage on Tuesday. Now we know exactly what the cause was, and how the problem was fixed.
Tuesday will be remembered as the day the internet broke — before swiftly being fixed again. Early in the morning, websites including Amazon, Reddit, Spotify, eBay, Twitch, Pinterest and, unfortunately, CNET went offline due to a major outage at a service called Fastly. Everywhere you looked, there were 503 errors and people complaining they couldn’t access key services and news outlets. Within 24 hours, we found out the root cause of the outage.
After an investigation into what went wrong, Fastly published a blog post describing exactly what went down — and it turns out the whole incident was triggered by just a single, unnamed Fastly customer.
In mid-May, Fastly issued a software deployment that contained a bug, which if triggered in specific circumstances could take down vast swaths of its network. The bug lay dormant until June 8, when one Fastly customer inadvertently triggered it during a “valid configuration change,” which caused 85% of the company’s network to return errors.
“We detected the disruption within 1 minute, then identified and isolated the cause, and disabled the configuration,” said Nick Rockwell, Fastly’s senior vice president of engineering and infrastructure, in the blog post. “Within 49 minutes, 95% of our network was operating as normal. This outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them.”
What happened during the Fastly outage?
At around 2:58 a.m. PT, Fastly’s status update page noted an error, saying “we’re currently investigating potential impact to performance with our CDN [content delivery network] services.” Shortly thereafter, reports emerged on Twitter of major news publications including the BBC, CNN and The New York Times being offline. Twitter itself was still running, although the server that hosted its emojis went down, leading to some odd-looking tweets. ReadMore
Source : cnet