The rollback plan that got used

War story. Serial numbers filed off, lessons intact.

A finance-adjacent system, a planned Saturday cutover to a new platform, months of parallel running behind it, diffs at zero for six weeks. The checklist was honest, the rehearsals were real. At hour four, the new estate's integration with a partner system started timing out under real Saturday volume, a load shape the mirror had never carried because the partner throttled test-tagged traffic differently. Nobody had known. Nobody could have, short of this exact Saturday.

The no-go person, who was not the project's sponsor on purpose, called it at hour five against the plan's hour-six limit. Rollback executed in forty minutes, Monday opened on the old system, customers noticed nothing. Three weeks later the partner issue was fixed and the second attempt was the most boring cutover of the year.

What made the rollback work is the entry's whole cargo:

1. It was rehearsed that month. Not written, rehearsed. The team had run it end to end fourteen days earlier, which is why forty minutes was forty minutes and not four hours of discovering that a restore script assumed a hostname that no longer resolved.
2. The data had a plan of its own. Four hours of writes had landed on the new estate. The plan pre-decided the answer: a reconciliation export, replayable into the old system, tested with fabricated Saturday data during rehearsal. "What about the writes" is the question that turns rollbacks into all-nighters; it had been answered in daylight.
3. The abort criteria were numbers with a clock. Error rate over threshold for thirty continuous minutes inside the final two hours: abort. Nobody had to win an argument at 2 a.m.; the plan had already voted.
4. Rolling back was reputationally safe. The sponsor had said, in writing, in advance, that an executed rollback was the process working, not failing. So the room reported the timeout at first sight instead of nursing it hopefully for an hour. That sentence in the kickoff deck was worth more than any script.

The counterfactual estates are the ones we usually meet: rollback plans written to satisfy a checklist, never rehearsed, aging in a wiki. A plan like that fails exactly when consulted, which converts a bad Saturday into a famous one.

Inventory first, always, but rollbacks second: before any window, ask the team when the rollback was last actually run, by whom, and how long it took. If the answer contains the word "should," the window moves. Nobody has ever regretted rehearsing a rollback. The alternative gets written up, occasionally by regulators, always by us.