Facebook blames ‘faulty configuration change’ for six-hour outage
Image credit: Image Source
Facebook Inc has blamed a “faulty configuration change” on its routers for causing a widespread, nearly six-hour outage yesterday (4 October), which affected the 3.5 billion users of its platforms Facebook, Instagram, WhatsApp and Messenger, as well as taking out Facebook's own internal Workplace software for its employees.
Tens of thousands of people reported outrages yesterday evening, prompting Facebook to confirm using its Twitter account that it was aware of the issues and working to resolve them. It apologised for the outage, particularly for its effect on businesses that depend on Facebook platforms for sales and thanked its users for “bearing with us”.
According to reports, other tools which depend on Facebook services also suffered, including employees’ work passes and email, Oculus VR, and Pokémon Go, which requires a Facebook login.
According to around 50,000 reports on Facebook submitted to DownDetector before 5pm, most problems (72 per cent) were with the website while some were with the server connection and app. More than 75,000 complaints were submitted regarding WhatsApp and 30,000 regarding Instagram, with many citing the server connection. The outage was the largest ever tracked by DownDetector.
Facebook and Instagram eventually returned late on Monday evening, while WhatsApp returned in the early hours of Tuesday morning. Ad measurement firm Standard Media Index estimated that Facebook lost out on approximately $545,000 in ad revenue for every hour of the outage.
Facebook has offered the most likely cause of the outage in a statement: “Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centres caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.
“We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.”
It did not provide further details about the configuration change, such as whether it was planned. It added that it is working to better understand the outage to improve the resilience of its infrastructure.
The outage prompted speculation on rival platforms such as Twitter on the cause of the outage, with suggestions ranging from feasible technical faults to the conspiratorial.
Earlier in the day, anonymous Facebook employees told Reuters they believed the outage was caused by an internal mistake in how traffic is routed to its systems, compounding other failures. Security experts commented that human error or sabotage were plausible.
The outage came one day after a former Facebook employee-turned-whistleblower gave up her anonymity, revealing herself as the source of a vast trove of sensitive internal documents passed to the Wall Street Journal, which appear to show the company consistently and knowingly profiting from hate speech and misinformation. Frances Haugen, a former product manager in Facebook’s civic misinformation team, will this week give evidence to a US Senate.
According to prepared testimony seen by Reuters, Haugen intends to compare Facebook to tobacco companies that for decades denied the link between smoking and lung cancer. She will describe her former employer as “one of the most urgent threats” facing the US.
Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.