UPDATE 6/9: Fastly today provided more details on Tuesday's outage, placing the blame on an "undiscovered software bug" that was triggered by a "valid customer configuration change."
The issue dates back to May 12 when Fastly rolled out a "software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances." On June 8, one of its customers made that configuration change, which "triggered the bug [and] caused 85% of our network to return errors," says Nick Rockwell, SVP of Engineering and Infrastructure.
Fastly is now deploying a bug fix and conducting a post mortem on why this happened.
"Even though there were specific conditions that triggered this outage, we should have anticipated it," Rockwell says. "We provide mission critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority. We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support."
Original Story 6/8:If you were trying to check the news this morning and got a series of 404 errors on the web, you’re not alone. A problem with content delivery network (CDN) Fastly temporarily took major online publications and services offline, including The New York Times and The Guardian.
Recommended by Our Editors
“We're currently investigating potential impact to performance with our CDN services,” Fastly wrote on its status page just before 6 a.m. ET this morning. Several minutes ago, it updated to say “The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.”
It later said a "service configuration...triggered disruptions across our POPs [Points of Presence] globally." That configuration was disabled.
Affected sites are slowing coming back online; certain elements of the New York Times’ homepage are not loading, for example, but it—as well as sites like Reddit—are now live.
With sites offline and unable to publish, a number of editors took to Twitter to livestream the outage and other news. As The Guardian’s Alex Hern explained, Fastly “runs an ‘edge cloud,’ which is designed to speed up loading times for websites, protect them from denial-of-service attacks, and help them deal with bursts of traffic. That technology inherently requires Fastly to sit between most of its clients and their users, meaning that if the service suffers a catastrophic failure, it can prevent those companies from operating on the net at all.”
Sites switched away from Fastly, he said, to get back online.
Downdetector.com confirms that outage reports with Fastly appeared around 5:52 a.m. ET. It shows trouble with services like Twitch, HBO Max, CNN, fuboTV, Target, and more.
Disclosure: Downdetector is owned by Ookla, which is owned by PCMag parent Ziff Davis.