There's a particular kind of confidence that sets in after a deployment goes smoothly a dozen times in a row. The pipeline is green, the metrics look clean, and the rollback procedure sits in the runbook like a fire extinguisher you've never had to use. You know it's there. You're pretty sure you know how it works.
Then something breaks in production at 11pm, and you discover that "pretty sure" is doing a lot of heavy lifting.
Rollback is the recovery path most teams treat as a given — the escape hatch that's always available, always reliable, always faster than pushing forward. In practice, it's often none of those things. And the gap between assumed and actual rollback capability is one of the more reliable sources of extended incidents I've seen.
The Assumption That Rollback Is Symmetric
The core problem is that deployments and rollbacks are not mirror images of each other. Deploying new code is a forward operation — you're adding capability, changing behavior, writing new data. Rolling back is a backward operation on a system that has already moved forward.
That asymmetry matters most when your deployment touched data. A schema migration that ran successfully during the deploy doesn't un-run when you roll back the application code. A new event type written to a queue doesn't disappear because you reverted the service that wrote it. The downstream consumers of that data — the ones still running the old code — now have to handle records they were never designed to process.
This is the failure mode that bites teams who've tested their rollback procedure in staging but not against a realistic production data state. Staging doesn't have three weeks of accumulated records in the new schema format. Staging doesn't have the downstream service that's been running for six hours against the new API contract. The rollback works fine in staging because staging is a simplified version of the problem.
What "Tested Rollback" Actually Requires
The teams I've seen handle this well share a few practices that are less common than they should be.
First, they treat rollback as a deployment in its own right — with its own checklist, its own verification steps, and its own definition of success. Not "did the old version start?" but "is the system behaving correctly given the state it's actually in?" Those are different questions.
Second, they think explicitly about data compatibility before deploying, not after something breaks. The question to answer before you ship is: if we roll this back in two hours, what does the system look like? If the answer involves manual data cleanup, compensating transactions, or "we'd need to call the database team," that's a risk that should be visible before the deployment, not discovered during an incident.
Third — and this is the one most teams skip — they actually exercise rollback on a schedule. Not chaos engineering, not a tabletop exercise, but a real rollback of a real recent deployment in a production-like environment, with someone on the clock. The goal is to find out what the runbook gets wrong before you need the runbook to be right.
The Organizational Pattern Behind the Gap
There's a reason rollback procedures go untested even at teams that care about reliability. Rollback is a recovery capability, and recovery capabilities are easy to defer because they only matter when something goes wrong. The deployment pipeline gets investment because it runs every day and its failures are immediately visible. The rollback procedure gets a wiki page because its failures are hypothetical — until they aren't.
This is the same dynamic that produces escalation paths nobody tested (covered here back in July) and recovery plans that look solid until the actual incident reveals the gaps. The common thread is that we tend to invest in the systems we use constantly and underinvest in the systems we use only under pressure. Under pressure is exactly when you can't afford to discover that your assumptions were wrong.
The fix isn't complicated, but it requires treating rollback as a first-class operational concern rather than an afterthought. That means rollback steps in the deployment checklist, not just the incident runbook. It means data compatibility review as part of the deployment review. It means someone on the team can answer, from memory, what a rollback of last week's release would actually involve.
If the honest answer to that last question is "I'd have to look it up," you have a gap worth closing before the next 11pm page.
The rollback you've never practiced is the rollback you'll fumble when the pressure is real. The time to find out what it actually takes is before you need it to work.