The Migration You Cannot Roll Back

Modern deployment has trained an assumption into all of us: every change can be undone. Canary, blue-green, feature flag, revert the commit and let the pipeline run. The rollback button is right there in the interface, and for code it genuinely works.

Then someone drops a column, and the button is still there, and it does nothing useful at all.

Code is an artefact, data is state

The asymmetry is simple once you say it out loud. Rolling back code means running a previous artefact — it exists, it is immutable, it is sitting in a registry. Rolling back data means restoring a previous state, and the previous state no longer exists. What you have instead is a backup from some minutes or hours ago, and restoring it means discarding every write since.

So the rollback for a bad deploy costs a few minutes of degraded service. The rollback for a destructive migration costs whatever your customers did between the backup and now. Those are not the same operation and they should not be described with the same verb, but they usually are — right up until the incident where it matters.

Expand and contract, and where it actually hurts

The standard way through this is to split a schema change into phases that are individually safe. Add the new thing without depending on it. Write to both. Fill in the history. Move readers. Only then remove the old thing.

Every row in that table is reversible except the last one, and the last one is where teams get hurt — for a reason that is mostly psychological.

The contract step lands days or weeks after the change everyone was nervous about. The risky-feeling deploy already went out, watched closely, and it was fine. By the time the drop ships, the migration is considered done, the person running it is tidying up, and the change goes out on a Friday with a one-line description. The only irreversible step in the whole sequence is the one that ships with the least attention on it.

The old code you forgot was running

The contract step also has a blast radius most people underestimate, because dropping a column breaks anything still expecting it — and "anything" is a longer list than the deployed service version.

There is a batch job that runs monthly and has not run since before the switch. There is a worker that has been up for eleven days and never picked up the new build. There is a queue holding messages serialised in the old format, which will be consumed at some point after the column is gone. There is an analytics pipeline nobody in the room owns. There is a mobile client on a version from March that you cannot force anyone to upgrade. And there is the rollback path itself: if the current release turns out to be bad and you revert, you revert to code that reads the column you just dropped.

That last one is worth sitting with. Dropping a column does not just remove a one-way door from this change. It removes the escape hatch from every change you ship afterwards, until the old code is genuinely gone everywhere.

The defence is a read monitor rather than confidence. Instrument reads of the old field, ship the switch, and then wait — through a full monthly cycle, past the longest-lived worker, beyond your mobile support window — until the counter has been flat at zero for long enough that you would be surprised to see it move. Then drop. The waiting is the work.

Dual-writes lie, and backfills race

The middle phases are safer, not safe.

A dual-write inside a single transaction is genuinely atomic. A dual-write across two systems — old table and new service, database and search index, primary store and event stream — is two operations that can disagree. One succeeds, the other fails or is retried out of order, and you now have divergence that nothing will detect, because both systems believe they are correct. If you are dual-writing across a boundary, you need a reconciliation job comparing the two and reporting drift, and you need it running before you trust the new side enough to read from it.

Backfills have their own hazard: they race with live traffic. Your job reads a batch of rows, computes the new values, and writes them back — and somewhere in that gap a user updated one of those rows through the normal application path. Write the whole batch unconditionally and you have just reverted a customer's change with an UPDATE nobody will ever find in a log. The fix is to make the backfill refuse to overwrite: fill only where the target is still null, or condition the write on the version you read. Make it resumable and idempotent too, because it will be interrupted, and restarting from the beginning on a large table is how a two-hour job becomes a two-day one.

"We'll roll back" is not a plan

If your migration plan's contingency section says roll back, and the change includes a destructive step, you do not have a contingency section. You have a sentence.

A real plan names which step is one-way and says what happens if the problem is discovered after it. Usually the honest answer is that there is no undo, only a forward fix — re-expanding the column, re-deriving the values from another source if one exists, or accepting the loss. Knowing that in advance changes the decision, because it forces a question that is otherwise never asked: is removing this old column worth giving up the ability to reverse anything for the next two weeks?

Frequently it is not, and the column stays a while longer at a cost of nothing but a little untidiness. The migrations that go badly are rarely the ones that were too cautious.

Judge a migration on whether its irreversible step can be delayed, observed and deliberately chosen — not on whether it applied cleanly. Reversibility is not a button on a dashboard. It is a property you either build into the sequence or discover you never had.