The most dangerous postmortem is the one where the team leaves the room saying "we still don't know why it happened."
Not because the unknown is inherently dangerous — production systems are full of unknowns, and experienced engineers learn to live with them. The danger is what happens next: the incident gets closed, the ticket gets marked resolved, and the system goes back to doing whatever it was doing before. The mystery becomes wallpaper. Six months later, it happens again, and someone says "oh, we've seen this before" — and still nobody knows why.
Irreproducible incidents are a specific failure mode that most teams handle badly, and the way they handle it reveals something important about how they think about operational knowledge.
What "We Can't Reproduce It" Actually Means
When a team can't reproduce an incident, there are really only a few explanations. The failure was triggered by a transient state — a race condition, a cache in a specific configuration, a third-party dependency that briefly misbehaved. Or the failure was triggered by a combination of conditions that no longer exist — a traffic pattern, a data shape, a deployment artifact that's since been overwritten. Or the team is looking for the wrong thing, because the mental model of the system is wrong.
That third case is the one worth sitting with. The inability to reproduce isn't just a technical problem; it's an epistemological one. You can't reproduce what you don't understand, and you don't understand it because your model of the system doesn't include the mechanism that failed. The gap between your mental model and the actual system is exactly the size of the incident.
This is where most postmortem processes fail. They're designed to answer "what happened" — to reconstruct a timeline, identify a proximate cause, and assign action items. That's useful when the failure is legible. When it's not, the same process produces a postmortem that says "root cause: unknown" and action items that amount to "add more logging." Which is fine, as far as it goes. But it treats the irreproducibility as a dead end rather than as a signal.
The Irreproducible Incident as a System Probe
Here's the reframe: an incident you can't reproduce is a probe into your system's actual behavior, not a failure of your debugging process. The fact that it happened tells you something real — that the system has a failure mode that exists outside your current observability envelope. The fact that you can't reproduce it tells you something equally real — that you don't have enough visibility into the conditions that trigger it.
Both of those are actionable, even without knowing the root cause.
The question to ask in the postmortem isn't "why did this happen" — it's "what would we need to see to know if this was about to happen again?" That reframe shifts the output from "we need to understand the past incident" to "we need to instrument the system better for the next one." The incident becomes a specification for your observability gaps.
This is a different kind of action item. Instead of "fix the thing that broke," it's "build the capability to see the thing that broke." It requires admitting that you're not going to close the loop on this particular incident — and that's a harder conversation to have in a postmortem room where everyone wants resolution.
Running the Incident Forward, Not Just Backward
The practical version of this looks like a specific exercise at the end of the postmortem: before you close the incident, write down what you would need to observe — in logs, metrics, traces, or system state — to detect this failure mode in the next five minutes if it were happening right now. If you can't write that down, you're not done with the incident. You've just stopped working on it.
That exercise forces the team to confront the observability gap directly. Sometimes it reveals that the gap is small — you need one more metric, one more log line, one more alert. Sometimes it reveals that the gap is architectural — you'd need to instrument something that's currently a black box, or add tracing to a path that's never had it. Either way, you've converted an unresolved mystery into a concrete engineering problem.
The teams that handle irreproducible incidents well share one habit: they treat the incident as open until they can answer that question. Not open in the sense of blocking other work, but open in the sense of tracked, revisited, and connected to the observability work that follows from it. The incident ticket stays linked to the instrumentation work. When the new logging ships, someone goes back and checks whether it would have caught the original failure.
Most teams close incidents when the system stops being on fire. The better practice is to close them when you've closed the knowledge gap. Those are different finish lines, and the distance between them is where the next incident lives.