There's a particular kind of confidence that comes from having a good incident response playbook. You've documented the escalation paths. You've defined severity levels. You've run tabletop exercises. And then production breaks in a way that doesn't fit any of the boxes, and you watch your carefully designed process dissolve in real time.
This is the gap that doesn't show up in postmortems often enough: the difference between the incident you planned for and the incident you actually got.
Your Mental Model of the System Is Already Wrong
Every engineer carries a model of how their production system works. That model was accurate at some point — probably around the time they last touched that part of the codebase. Since then, the system has drifted. New dependencies got added. A configuration change went in without a corresponding runbook update. A service that used to be stateless now has a cache that matters.
The model in your head is a snapshot. The system is a film.
This is why the first five minutes of an incident are often the most dangerous. You're not debugging the actual system — you're debugging your mental model of it, and the two aren't the same. The engineer who's most confident about what's wrong is frequently the one who's most wrong, because confidence correlates with how recently you worked in that area, not with how accurately you understand the current state.
The practical consequence: during an incident, your first hypothesis should be held loosely. Treat it as a direction to investigate, not a conclusion to confirm. The teams that recover fastest tend to be the ones that stay genuinely curious about what the system is telling them rather than hunting for evidence that they were right.
The Failure Mode Nobody Practices
Tabletop exercises are useful. Chaos engineering is useful. But both share a structural limitation: they test your response to failures you imagined in advance. The failure modes that actually hurt you are the ones that fall between the categories.
Consider what happens when two systems degrade simultaneously but independently. Your monitoring catches both signals. Your on-call engineer sees two separate alerts and starts working the more familiar one. The second issue — the one that's actually causing the customer-visible problem — gets treated as noise or a downstream effect of the first. By the time the team realizes these are separate incidents, the blast radius has expanded.
This isn't a process failure in the traditional sense. The runbooks were followed. The escalation happened. The problem is that the incident didn't fit the mental model of "one root cause, one fix," and nobody had practiced the coordination overhead of running parallel investigations.
The teams that handle this well have usually been burned by it before. They've developed informal norms — someone explicitly owns the question "is this one incident or two?" early in the response. That's not in most runbooks. It lives in the team's operational memory, which is exactly why it evaporates during turnover.
Boring Reliability Is a Skill, Not a Personality Trait
There's a tendency to treat operational excellence as something certain engineers have and others don't — the "reliability mindset," the "SRE instinct." That framing is comfortable because it makes the problem about hiring rather than about systems and practice.
The engineers who are genuinely good at incident response aren't running on instinct. They've built habits: they write down their hypotheses before they start investigating, so they can track when they were wrong. They narrate their actions in the incident channel, not because they're performing for an audience, but because articulating what you're doing forces you to notice when you're not sure why you're doing it. They resist the pull toward the familiar fix.
These habits are learnable. They're also fragile — they erode under pressure, during handoffs, and when the team is understaffed. Which means the real operational question isn't whether your team has good instincts. It's whether your environment makes good habits easier than bad ones.
A runbook that requires ten steps before you can restart a service is a runbook that will get skipped at 3am. An alert that fires without context about what to check first is an alert that trains engineers to dismiss alerts. The environment shapes the behavior, and the behavior determines the outcome.
What to Actually Watch
The next time you run a postmortem, add one question to the standard template: At what point did our mental model of the system diverge from what was actually happening, and how did we find out?
That question surfaces the gap between your planned response and your actual response. It tells you whether your monitoring is giving you accurate information or a filtered version of it. And it points toward the specific places where your team's operational knowledge has drifted from the system's current reality.
The incident you survived last month taught you something about the incident you planned for. The one that will hurt you next is already diverging from that model. The teams that stay ahead of it are the ones that treat the gap itself as the thing worth measuring.