Devon Ashcraft
🇺🇸Production is down: incident response and on-call sanity
I have run infrastructure and on-call rotations for sixteen years in the Bay Area, most recently as a staff site reliability engineer. I take evening and night sessions because that is when production breaks and because this is when I am awake anyway. Two kinds of calls. The first is right now: something is down, you are the only person on, and you need another pair of eyes on what you are seeing. I can be that, and I will tell you what to check in what order and when to stop debugging and roll back, which is the decision people get wrong under pressure. The second is calmer and more valuable: your on-call is destroying your team. Pages that are not actionable, no runbooks, the same incident every three weeks with no follow-up. That is fixable and it is mostly not a technical problem. I will also tell you honestly when the answer is that you are understaffed and no amount of tooling will cover it.
