Arpit’s Newsletter

Home
Career Growth
System Design
Deep Dives
Papers and Musings
Outage Dissections
About

Outage Dissections

So, the outage is mitigated, now what?
Getting the service up is P0 during the outage, but that is not all. There are a few other things that we need to take care of once the issue is…
 • 
Arpit Bhayani
Control an outage by localizing the failures
Outages are inevitable; but we should design our architecture and ensure that if a component is down, it should not lead to a complete outage. What…
 • 
Arpit Bhayani
Dissecting GitHub Outage - Multiple Leaders in Zookeeper Cluster
The split-brain problem in Distributed Systems is not theoretical. GitHub had an outage because their Zookeeper cluster ended up having two leaders…
 • 
Arpit Bhayani
Dissecting GitHub Outage - How databases are managed in production
Managing databases in production is not easy and it does require a lot of tooling to keep it up and running. GitHub had an outage that gives us a…
 • 
Arpit Bhayani
Dissecting GitHub Outage - Downtime due to Rate Limiter
Rate Limiters are supposed to avoid downtimes, but what if they turn out to be the root cause of a major outage? A large chunk of GitHub users saw…
 • 
Arpit Bhayani
Dissecting GitHub Outage - When master failover failed
Master failover failed for GitHub leading to a 5-hour long incident, let's see what happened. Incident Summary For five hours, GitHub users observed…
 • 
Arpit Bhayani
Dissecting GitHub Outage - Downtime GitHub thought they avoided
GitHub thought they avoided an outage by fixing a possible root cause 6 months in advance, but fate had different plans. Check Suites and Workflows When…
 • 
Arpit Bhayani
Dissecting GitHub Outage Downtime due to creating an Index
Imagine you created an index on a table and instead of boosting the performance, it lead to an outage 🤦‍♂️ GitHub ran a migration to reverse an index…
 • 
Arpit Bhayani
Dissecting GitHub Outage - Repository Creation Failed due to secret scanning service
Just imagine you trying to create a repository on GitHub and it is not working, and this happened to GitHub in April 2021 when their users were not able…
 • 
Arpit Bhayani
Dissecting GitHub Outage: Downtime due to an Edge Case
An edge case took down GitHub 🤯 GitHub experienced an outage where their MySQL database went into a degraded state. Upon investigation, it was found…
 • 
Arpit Bhayani
ALTER TABLE taking down GitHub
Can an ALTER TABLE command take down your production? 🤯 It happened to GitHub on 27th November 2021 when most of their services were down because of a…
 • 
Arpit Bhayani
Engineering deep-dive into Atlassian's Mega Outage of April 2022
In April 2022, Atlassian suffered a major outage where they "permanently" deleted the data for 400 of their paying cloud customers, and will take them…
 • 
Arpit Bhayani
© 2024 Arpit Bhayani
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great culture