UPDATE 10:45 p.m. ET: Facebook on Monday blamed a "faulty configuration change" for today's outage.
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication," says Santosh Janardhan, Facebook's VP of Infrastructure. "This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt."
There is "no evidence that user data was compromised as a result of this downtime," he says.
The outage was so extensive because it "also impacted many of the internal tools and systems we use in our day-to-day operations, complicating our attempts to quickly diagnose and resolve the problem," according to Janardhan. "We apologize to all those affected, and we’re working to understand more about what happened today so we can continue to make our infrastructure more resilient."
UPDATE 8:30 p.m. ET: Facebook's services started to come back online around 6:30 p.m. ET. In a tweet, Facebook apologized for the downtime. CEO Mark Zuckerberg also posted an apology on his Facebook page.
"Sorry for the disruption today -- I know how much you rely on our services to stay connected with the people you care about," Zuckerberg wrote.
Downdetector says "the global Facebook outage is now one of the largest ever tracked on Downdetector in terms of the total number of reports (over 14 million as of 3:30 p.m. PDT) and duration." Earlier this afternoon, the US had the most reports at over 1.7 million followed by Germany (1.3 million), the Netherlands (915,000), the United Kingdom (789,000), and Italy (400,000), according to the site.
What Is BGP Routing?
As Cloudflare outlined in a blog post, "It was as if someone had 'pulled the cables' from [Facebook's] data centers all at once and disconnected them from the Internet." Cloudflare's Celso Martinho and Tom Strickx point to what's known as the Border Gateway Protocol.
"BGP stands for Border Gateway Protocol. It's a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn't know what to do, and the Internet wouldn't work.
The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable."
Their post goes into full technical details for those who want a deep dive. For everyone else, it's back to arguing with your high school classmates on Facebook, peeping your ex's Instagram Stories, and staying connected through WhatsApp messages.
Original Story:Facebook and Instagram are having a rough Monday; both apps are down for many people.
Facebook confirmed the issue on its Twitter feed, and said it's "working to get things back to normal as quickly as possible." Outgoing CTO Mike Schroepfer also tweeted "sincere apologies" and pointed to "networking issues" that teams are working to debug.
It said the same for Instagram:
On Downdetector.com, outage reports spiked just before noon ET for Facebook...
... and Instagram:
Facebook-owned apps WhatsApp and Facebook Messenger were also having stability issues this morning, as is Oculus.
According to security reporter Brian Krebs, the DNS records for Facebook and Instagram were withdrawn today:
Cloudflare CEO Matthew Prince suggests this is a costly mistake versus a cyberattack:
This is probably not how Facebook wanted the start the week, after a bruising night that featured a 60 Minutes interview with a whistleblower who handed over tons of internal documents to the Wall Street Journal and regulators. The former Facebook employee, Frances Haugen, argued that the social network is not "willing to actually invest what needs to be invested to keep Facebook from being dangerous." Facebook denies any wrongdoing and says its research has been mischaracterized.
Twitter, at least, is having a laugh over the whole incident.
(Editors' Note: Downdetector is owned by Ookla, which is owned by PCMag publisher Ziff Davis.)