Facebook

More Details on Today's Outage

Facebook Engineering

Early today Facebook was down or unreachable for many of you for approximately 2.5 hours. This is the worst outage we’ve had in over four years, and we wanted to first of all apologize for it. We also wanted to provide much more technical detail on what happened and share one big lesson learned.

The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition. An automated system for verifying configuration values ended up causing much more damage than it fixed.

The intent of the automated system is to check for configuration values that are invalid in the cache and replace them with updated values from the persistent store. This works well for a transient problem with the cache, but it doesn’t work when the persistent store is invalid.

Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.

To make matters worse, every time a client got an error attempting to query one of the databases it interpreted it as an invalid value, and deleted the corresponding cache key. This meant that even after the original problem had been fixed, the stream of queries continued. As long as the databases failed to service some of the requests, they were causing even more requests to themselves. We had entered a feedback loop that didn’t allow the databases to recover.

The way to stop the feedback cycle was quite painful - we had to stop all traffic to this database cluster, which meant turning off the site. Once the databases had recovered and the root cause had been fixed, we slowly allowed more people back onto the site.

This got the site back up and running today, and for now we’ve turned off the system that attempts to correct configuration values. We’re exploring new designs for this configuration system following design patterns of other systems at Facebook that deal more gracefully with feedback loops and transient spikes.

We apologize again for the site outage, and we want you to know that we take the performance and reliability of Facebook very seriously.

Angie Perham

623pm nevada still down fb had me make new password and one of the feeds are posted

Daniel Watts

IT STILL ON WORKING ON MOBILE..ONLY WORKING ON LAPTOP FOR ME

Daniel Watts

NOT WORKING ON MOBILE

David Baer

still having issues in Illinois.

Kathy Keebaugh

Having short list of posts as well. Sad

Linda Clark

4:07 PST - Having problems in San Diego - UGH!

Armindo Souza

Still with problems for me.

Margaret Ferguson

Can’t get on been trying all day weird
Is there still probs

Emke Kay Marilyn

Went down yesterday and I'm still down.
Waiting......!!!!!!

Karen Frazer

facebook is currently DOWN now 9/28 causing huge problems as I need evacuation status on current wildfires; east Santa Rosa evacuated and the fire is moving fast

Karen Frazer

not a bandwidth issue as I can watch movies and check all other websites

Trick Lattie

Down in Florida? 🙄

Kay Sucy Mason

Down in Colorado!

Nancy Russell

Yeah, I wondered where everyone was.

Mohammed Awwad

Cyprus here...A long article post I was writing for one hour was suddenly lost...
unable t o retrieve it...the problem is still going on!!

Frank Moran

5th october, facebook down again why?.last few weeks it only works sometimes..

Lin Banks

FACEBOOK STILL DOWN AT 12:50 PST ON THE CALIFORNIA COAST.

Frank Moran

Hhthjvjk

Narmeen Walid

I’m having an unexpected error in logging in my account from the Facebook app and messenger !!

Adam Griffiths

down in Cairns, Australia

Rowena Hover

Down in FL.

Marye Steckley

Down in Oregon.

Mary Radcliff

Not fixed at 9:42 mountain Time

Paula Prindle-Epps

My Facebook is still down it's been over 12 hours

Patricia Webb

Facebook is so slow and unresponsive 😡

Michael Ferguson

My Facebook is still not working

Kyle Lambert

@[100000493248837:2048:Michael Pica] Oof.

Ed Wieczorek

still having trouble in knoxville tenn with sharing

Karen Smith

Not been working for 3 days now

Betty Hulin

Mine is still not working in Gilman, Iowa. I hope it's back working soon

Tony Goodman

I have been telling you Facebook to stop trying to fix something that is not broken. When are you going to cut your losses? I believe everybody knows that Russia, and every other nation that doesn't like the U.S. is trying to take you over, or take you down! Stop trying to run the company like a dictatorship, and things might get better for you. Classic Facebook will solve your problems, go back to it. You're welcome!

Eva A. Combs

Why is it out now10 years later?

Judith Pool

DOWN TODAY

Wendy Oakes

Still down in PA

Carole Daniels

Still not loading notifications from friends post can’t read comments. Just get the symbol it’s searching

Saipin Jirasakhirun

Still down in Atlanta

Joe Kotera

It's down now on my end not showing anything on my timeline and won't refresh but I can go look into anything else

Mike A. Sekosky

Should notify if there’s an outage

Erena Aman

Facebook is down what happened in Virginia area

John O'Brien

Haven’t had a updated post in a week! Anybody working at censorship central?

Valentin Quitral

FB connection has been awful for weeks now and just keeps going to the kill or wait message

Diane Virginia Cunio

Not fixed here in NC. It's 5:41 pm and it's been down ALL DAY.

Lorraine Summey Emch

Not fixed in Illinois either

Doc Cook

I can’t read the notifications comments from my private group. Why!!! I need to read them!!

Sherri Jameson Faulk

Facebook is not working now, October 26,2020

Ryan Thomas

lol at everyone who's saying it's "still" down and don't realize this post is from TEN YEARS AGO. 🤣

Mary Kozlowski

Unfortunately Facebook has stopped is the response I am still getting. All day long and into tonight!

Maria Hartman

My Facebook is still down. It’s 11 pm!

Joseph Matthias Young

those are certainly https://www.facebook.com/groups/380017965778389/?hc_location=ufi

Tony Tokyo

all good no love lost

Maria Lombardo Ditchek

A message on my FB page says it doesn't recognize my browser. My browser is Chrome and it's been fine up until now.

Susie Garcia

Why do I have to keep signing in??? Ridiculous

Susie Garcia

Hmmmm political anyone

Marsha Goodhue Hatfield Feucht

I can't sign on to FB....everwhere else, yes...no with FB....everyhing was fine, then bingo, no FB.....period. Anyone know what to do? Today is Oct 30, 2020 10:30 a.m.

Jennifer Bless

Tanks

Hriday Behari Bajpai

Facebook not working properly please correct

Ashish Jatav

yeah call on the support number FACEBOOK HEADQUARTERS
+1 855 971 OO7O Facebook headquarters phone number
+1 855 971 OO7O __ or email to complaintsandhelp @ outlook. com.. . . / . ... . - - - tormund . . .,. ,. .,

Jeffrey Cline

Issues are currently affecting multiple ads accounts - disapproving ads and in some cases suspending accounts. For anyone running an ad account, check your ads and account performance - hoping issues are resolved soon.

Christina Ianniello

I'm having the same issue with my ad accounts. They've completely disabled one of them.

Bob King

Using iPhone app and was logged out and cannot log back in with the app. Just says no Internet connection. Well posting here via mobile browser so obviously have internet. This is 3:25pm Eastern.

Ed Wieczorek

this new facebook is a joke , nothing works like the old version , fix this or go back to the old version

Shubham Soni

yeah call on the support number FACEBOOK HEADQUARTERS
+𝟭 𝟴𝗢𝟱 𝟮𝗢𝟯 𝟴𝗢𝗢𝟭Facebook headquarters phone number
+𝟭 𝟴𝗢𝟱 𝟮𝗢𝟯 𝟴𝗢𝗢𝟭 __ or email to complaintsandhelp @ outlook. com.. . . / . ... . - - - tormund

Larry Peck

I HATE THE NEW VERSION OF FACEBOOK. NOTHINIG BUT PROBLEMS. AT LEAST GIVE PEOPLE THE OPTION TO GO BACK. AND YES....I KNOW THESE ARE ALL CAPS!!!

Ashish Jatav

Lila Bricklin

I can't Post ‼️ I use chrome and CANNOT Post a photo or an article. It just spins. I've rebooted, cleared cache and don't know what to do. Can somebody please let me know what to do? I'm not very tech savvy and just want to know when this is going to be fixed.

Madelene V. Sebaldt

Good luck to those woeking with the Facebook site. It is back now. :-)

Leigh Coffey

I still can't log in. I open my browser, click on the Facebook icon/blue rectangle, and get a window that says I'm not logged in. (Right!) I click on Log in, and the same window just keeps reappearing. Now I find that I'm somehow automatically getting in just by going to this website. Hmm.

Jan Lutz

ME TOO!

Sheryl Davis Dear

Same here but I found something on the AOL page to post to FB and I was able to get into FB.

Ashish Jatav

Alison Dalrymple

I keep getting the same box that says login and it wont let me! Same box just keeps popping up! So frusterating!

Kimberley C Russell

I am having the same problem as Alison Dalrymple and it's been like it since last night! I only have a desktop so not sure if it is happening with phones.

Arthur Brown

mine will come up but says page unresponsive

Valerie Joy Kohlmeyer

Mine is still not working!!! It just sits there and flashes. Guess it is time to leave facebook and join Parler or MeWe!!

Mae Erickson

can't sign in page just keeps jumping all over the screen.

Suraj Shetty

yes same with me the login page is jumping

Lorraine Punton

Same here driving me crazy!!!

Chic Diores Robles

Yes..same here login page is jumping...

Frisco Papacostidis

Im very dissapointed with your new platform. What the hell is this. Why change if it aint broke. Keep it simple.

Shubham Soni

Phillip Heard

@[100024030765597:2048:Mae Nebril Obina] thats what was going on

Ken Fagers

pati messenger nag loloko ,
ako nalang yata talaga hindi pa nagloloko till now XD

Chris Nelson

@[108824017345866:274:Facebook] Facebook is still not working correctly on mobile.... using the FB app it will load between 2-5 stories and then state "Error Loading News Feed" at which point it **WILL NOT LOAD** any further content. Closing the app and restarting it does not work, only the same stories are shown. Focring the app to stop completely and then restarting from scratch does not, again only the same stories are shown.

Before this outage it was possible to view messages on mobile by using a browser and navigating to a specific message URL (ie facebook . com / messages / t / 123456789 ) and then selecting "view desktop site" from the mobile device browser menu.... however since this outage, it is no longer possible to view messages on a mobile browser; navigating to the message URL provides "this content cannot be found right now" error message, and a graphic of a thumb with a bandage on - clicking the messages notification icon no longer brings up the list of messages but merely changes the colour of the bandage on the graphic of the thumb. The error message of "this content cannot be found" is bunkum, as I have had the messages I wanted to view on my desktop PC, which were displayed correctly, and I copied the exact URL address to the mobile browser of which then it did not work.

Steve Crouch

Still not working on my desktop.

Garry Bradshaw

Knackered In UK Also

Guna Shekar Reddy

Deborah Hairston

FRUSTRATED

Guna Shekar Reddy

David Malone

It's still down. It's been down here for 18 hours.

Sanam Bewafa

Carol Mcdonald

Cannot get on Facebook. It says we're having problems with setting up Facebook in English United kingdom.

Ravi Saini

Rob Edlin-White

Why is there no date on this post?

Ravi Saini

Elmarie Kleinhans

When was this, I see no date

Laurence Dutch Knappert

Access to my business page is not available. If I try it just turns facebook off.

Frances Bugsbunny

I can’t sign out

Luke Lim

Facebook bug with Android? I can't open My Page. It keep on closing. Happened on my other android devices too.

Dan Shriver

I keep having people who friend me
completely disappear from the site,
Boom , there gone

Lynette Brooker

I kinda lost Facebook, my name and picture.

Ravi Saini

Yvonne White-Simon

2/5/2021, my Facebook is not working, from Ames, Iowa. What is wrong?

Priyanka Thakur

My Facebook is not working properly! First my account is disabled. And now I can’t make Request to review and Ads Account Overview is coming bank page? I am not getting any help

Ravi Saini

Aqua Vein

When was this?

Ravi Saini

Phyllis Smith

Mine told me to update but won't let me so can't use Facebook at all

Todd Webster

I am still getting error messages "Sorry, Something went wrong, please try again later"

Manjur Khan

Arnaldo Attallah Montis Alti

I am the founder and admin of the Facebook group “Virtual Museum”. Half of the albums with more than 3000 artworks --HD pictures framed with their descriptions and the biography of their related artists-- have disappeared from the group or, in other words, nobody can access them anymore! Neither me, the admin, nor the general public. This issue started a few weeks ago. I have written tot Facebook support and they have not been helpful at all. I go from one generic response to the next with no end is sight as to when these issues would be fixed. I don't know who to contact so these issues can be resolved once and for all. I am the founder of the group, and have been dedicated to this group from almost four years. The group in question “Virtual Museum” has over 130,000 members, who are now being penalized because they have no way of accessing more than 50% of the archived contents, that held albums containing more than 3000 Masterpieces.

Is anybody able to please HELP?

Leonel Auditor

Yeah. I understand what you're going through. Go Facebook Team!

Manjur Khan

Genomiel Gines

My FB won’t loud it just tells me something went wrong and kept on saying it will be fixed but it was almost 3 weeks now. I had a hard time logging in 😩what’s going on ?

Manjur Khan

PS Core Asiacell

what is state now ?

Manjur Khan

Rodger Wearne

When I open marketplace items it seems to close them again after a couple of seconds..
This is in Australia

Brenda Logo - Macale

Cant access

Ed Scott

Session ended log back in its frozen screen now

Cassie Daly

Sooooo I can't post on facebook or look at birthdays.

Cassie Daly

I have started my computer multiple times and nothing ..... I can post from my Phone tho.

Cassie Daly

Don't what ya guys did but NOW it finally working after 3 days.

Aleck Chirwa

Yes it will fine

Aleck Chirwa

Image plumber
😓Block drain R500
😭 Leaking toilet
😲Burst geyser replacement
🤔Gas geyser installation
😝Solar geyser installation
🤣We specialised in plumbing maintenance and construction
👩‍🔧 Call 0795717155

Manjur Khan

John Weeks

I believe during this time someone stole a profile of someone that I accept a friend request. About 10 seconds after I posted about PSYOPS lurking my account a fake Danny Mallott tried to swindle me out of money for a fake program to get grant money in return. I believe they stole a picture from Danny’s account as their profile pic. When I declined he disappeared. I made a report and tried to block him and it was a gost account. Just that pic came up with no account details. I was only able to block on messager and not Facebook, and I can’t get messenger notification of message to go away.

TouKir Ahmed

I Can't log in at my account with Messenger & Facebook lite app... Its shows #Service_temporarily_unavailable
#Login_failed_due_to_an_unsatisfied_error

Please Facebook Let's fixed it.. Don't vanish my account...

Thandi Tchala

Now??

Aria Brown

My fb has been down for about 3 days...still doesnt work...keep saying errorr

Harsh Chaturvedi

Great..atleast its working now

Tahera Shah

Its very boring now

Jwngsar P Daimari

I still can't use fb app. Its still down

Roman Rds

Now I can`t Open Facebook app and messenger

Abraham Rahman Sifat

Why Facebook are not open my mobile

James Jim

Why is it when I delete a post on Market place it is there again when I log back in. ? Also the page looks like it doesn't fit the screen. I have forced stopped, deleted cashed data still no joy.

John Boulogne

Test

Fratena Tena Dawson

Still not working on my iPhone for over 3 days now.

Emma Louise

Facebook app still down in UK

John Michael

App (android) not working (I'm in the UK). Won't let me login. Hasn't been working for 3 or 4 days. 5/4/2021

Alison Cairns

Android app not working properly in UK for over a week, unable to read comments or see most recent

Judith Gilbert

What's going on with FB today I'm in cali and get nothing even after installing and reinstalling

Debra Edwards

It's this happening now?

Refkinscallv

now

Ann Coppell

What’s going on

Lauren Exe

FB is shut down completely. so is Messenger. FIX THE ISSUE.

Ulrike Jung

It is down for me via app right now? Not loading?

Sami Chafchak

April 8 5:44 PM Eastern Toronto time. Facebook is not connected. Can't comment or post.

Taylor Redmond

FB not working for me 4/8/21 Been like this for 30-40 mins

Loraine Sabangan

April 8 in fresno, ca down approximate 2:47pm

Brian Griff Griffiths

down for about 30 min in Kentucky...

Angela Powell

We just wanted the low views on Marketplace fixing not a whole site down 🤦🏽‍♀️

Melissa Moghadam

facebook, workplace, whatsApp, and instagram they all went down altogether at 5:33 P.M eastern time!

Debbi Sobotka

FB DOWN 4/7 4:15PM SOUTHERN UTAH

Oussama Ziani

There are so many things to be fix on Facebook not just the outage

Khaled Mohamed Khaled

i cannot open any pàge .. i cañnot do any thinģ with the personal àccount

Khaled Mohamed Khaled

april 9. 5 am +2

Kyleen Fury

The issue was caused by a configuration error change

Tony Doyle

Looks like Prod changes not being tested properly in DEV

Hope Trachtenberg-Fifer

Facebook crashes every time I attempt to post.

Roanoke, VA
4-9-21

Tamie Dragone

Not working in Kansas. Crashing everytime I am trying to post in an album.
since 8 am April 10

Khaled Mohamed Khaled

facebook crashes when i open any page 11/4

Ḍexter Ḳatrinạ

I can't access Creator Studio, Ad Account or Facebook Developer since last week. Please help. I've raised two tickets with no acknowledgement or response.

Kaivalya Madhav Behere

Not working in Pune- India

Wolvie Michael R Vaughn

Its still not working !
It is currently April 12, 2021 2:40 PM EST.

Wolvie Michael R Vaughn

It's still not working !!!!!
It is currently April 12, 2021 5:38 PM EST.

John van den Broek

Cant post anything. Max about 10 minutes today the lockup. Over crap service . 13th April 2021 NZ

Wolvie Michael R Vaughn

It's STILL NOT WORKING !!!
It is currently April 13, 2021 Tuesday 8:10 AM EST. USA !!!!!!!!

Wendy Robertson Cain

Not working in southern Indiana

Hafnas

April 14 wed. I can't login fb.

Larry Jerre Anne Wagner

Not working in MissouriCity Texas
April 15

Madhu Chadda

Not working in India

Dannielle Judd

april 17, still all messed up.. south florida

Antonia Papadopoulou

it goes down every day for some hours 14-17/4/21 in crete, greece...... what is going on? every day? why? PLS INFORM THE PUBLIC

Rachelle Van Duser

Can't login in Las Vegas, NV

Michael Olmsted

Facebook not working.

Gaynor Wright

Not working in the UK. Not enabling me to add photos and text to business page. Extremely annoying.

Jim Davis

Just as well.

Janet M Keegan

Is this information current, as in 24 April, 2021? I can’t see a date on this information 👀

Singh Ranu

Fb is not working properly.... requesting to login again and even there is a glitch...

Eason Ng II

Is this on 26 april?

Stuart Dowling

April 25th,,,can't post or comment....page says it needs toreload...working on errors....

Phillip Longley

Today is 28 April 2021 & Facebook was working a I cannot see any photos nor videos.

Peter Ingram

It is still down in Sydney..Australia.
Saying problems ..reload page but still down
It is the 28th April at 1.05pm here

Jen E. Jen

Down 4/30 315am est

Jack Simpson

black screen Vancouver, BC Fri Apr 30th @ 12:23am

Krash Leigh

Down Fri, April 30th Indianapolis, IN @ 3:43pm EST

Gail Marie Reilly

North Carolina it is still not working it is now 4:50 am

Elise Grant Lauinger

Hilton Head SC down 4/30/21 6:23 am

France Garrido

NYC Facebook is a black screen.

Jim Ottney

game apps still not working in Stoughton WI

Jim Ottney

Now 7:00am CST and STILL Best Fiends, Words w/Friends and other game apps won't initialize.

Jim Ottney

and still not able to access apps from laptop at 8:00am CST.

Ann Murray

Still down in UK Scottish borders 15.22

Judith Mortimore

Site fine but no games loading!!!

Linda Dewhurst

30.04.21.. site ok but games not loading, anyone have an answer

Saffie Storch

it's not called faces**t for nothing! - games still not working after 8 hrs - 4hrs AFTER they fixed it!

Jim Ottney

"fixed" lol

Nikki Ourand Lambert

30 April 2021, currently 3:00pm...games still not loading since at least 8:00am. Please fix!

William Minerva

Facebook is down?

Kathy Hall

2 May 2021 - 09:07 GMT -- For two days, I've been having query issues, notifications not refreshing or opening, comments not showing until I refresh the page ... if that works. Often having to shut down FB altogether, which is a pain if I'm in the middle of something. What the heck, FB?!!!