• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

Facebook down

If the story is true that they needed to bring in a welder who used an angle grinder to get them in the server cage I am amused.

The big narrative seems to be that a pretty bad but not completely catastrophic outage was made catastrophic by Facebook's decision to run everything internally so essentially when Facebook went down Facebook couldn't get on Facebook to fix Facebook.

Running a process that internally for like your day to day stuff... still a bad idea in my opinion but you could at least argue there's some upsides to it.

But not having at least one engineer with a big keyring so they can physically access the building and servers and like a simple phone/e-mail tree for notifications and communications as part of your disaster recovery plan is like massively stupid.

This is "I lock to the keys in the safe to keep them safe" level of "Okay how does this go in your head?"
 
Does no-one teach/learn the basic skills any more? In my day with XXXXXXX, YYYYYY and ZZ we learned to bypass such annoyances as locks.
 
Does no-one teach/learn the basic skills any more? In my day with XXXXXXX, YYYYYY and ZZ we learned to bypass such annoyances as locks.

In a recent thread I criticized the film and book Jurassic Park because it wouldn't matter how egotistical John Hammond was, any engineer would have basic disaster recovery plans for systems reboots and power outages. The whole "Oh crap our computer system went down and we have ZERO PROCEDURES FOR THIS I GUESS WE'LL HAVE TO WING IT!" plot device that both the book and film (in slightly different ways) use just didn't make sense to me.

Maybe I was being a big optimistic.
 
One quote from a 20-ish nephew that will stay with me -- "Facebook is for old people."

I was told that would happen when everyones parents got on FB. All the cool kids would leave.

One thing that irks me about FB is that Zuckerberg is a dolt and he has a lot of power, but he's no Wozniak. He made an offensive app in college that was a simple vote yes or no when looking at someones face. Now it's FB.

He got lucky, and now he appears in front of congress, makes decisions that impact the world, and he does not seem like a very bright guy to me. At all.
 
Zuckerberg's comment on the outage and the whistleblower:

https://www.facebook.com/zuck/posts/10113961365418581

Some of it:

First, the SEV that took down all our services yesterday was the worst outage we've had in years. We've spent the past 24 hours debriefing how we can strengthen our systems against this kind of failure. This was also a reminder of how much our work matters to people. The deeper concern with an outage like this isn't how many people switch to competitive services or how much money we lose, but what it means for the people who rely on our services to communicate with loved ones, run their businesses, or support their communities.

Second, now that today's testimony is over, I wanted to reflect on the public debate we're in. I'm sure many of you have found the recent coverage hard to read because it just doesn't reflect the company we know. We care deeply about issues like safety, well-being and mental health. It's difficult to see coverage that misrepresents our work and our motives. At the most basic level, I think most of us just don't recognize the false picture of the company that is being painted.

Many of the claims don't make any sense. If we wanted to ignore research, why would we create an industry-leading research program to understand these important issues in the first place? If we didn't care about fighting harmful content, then why would we employ so many more people dedicated to this than any other company in our space -- even ones larger than us? If we wanted to hide our results, why would we have established an industry-leading standard for transparency and reporting on what we're doing? And if social media were as responsible for polarizing society as some people claim, then why are we seeing polarization increase in the US while it stays flat or declines in many countries with just as heavy use of social media around the world?
 
If we wanted to ignore research, why would we create an industry-leading research program to understand these important issues in the first place?

Yeah been there in 2 big corporates.
Top guy to subordinate "Here is Bad Thing. Here is money. Use money fix bad thing"
Subordinate who knows FA about Bad Thing assembles team, runs around screaming "There is Bad Thing", spends money, returns to Top guy "Bad thing addressed. Hear me gloat"
Top guy "Here is glory and pay rise."
Bad thing continues.
 
They have - they all went to Instagram.

Which is owned by Facebook.


Telegram got 70 million new users that day.

Which is owned by a Russian residing in Dubai who withstood attempts of the Russian government to shut the service down.

It's where the cool kids (like me) hang out these days.
 
Oh grandad - you are so out of date - the kids went to Snapchat and now to TikTok.

Not really. They're in the same demographic, but in terms of permanent pages and messaging, Instagram still rules among teens.
 
In a recent thread I criticized the film and book Jurassic Park because it wouldn't matter how egotistical John Hammond was, any engineer would have basic disaster recovery plans for systems reboots and power outages. The whole "Oh crap our computer system went down and we have ZERO PROCEDURES FOR THIS I GUESS WE'LL HAVE TO WING IT!" plot device that both the book and film (in slightly different ways) use just didn't make sense to me.

Maybe I was being a big optimistic.

I don't remember what happened in the book, but in the film, the engineer deliberately sabotaged the system. Disaster recovery plans can't mitigate the effects of insiders with enough power deliberately taking everything out.

It was a significant weakness of the system that Nedry couldn't effect his plan without taking down everything. You'd think he could program it in such a way as to just leave his path clear. It's also somewhat bizarre that nobody considered the possible effects of a power outage on the effectiveness of the fences.
 
I figure most folks here already know how DNS works, but Ben Eater did a YT video on the event. It goes into a lot of detail for DNS, but it has chapters so you can skip ahead. This is a tech background video, not anything about problems/procedures at FB during the event.



Ben has other videos on building an 8-bit computer from discrete logic chips.
 
Last edited:

Back
Top Bottom