Make the next data change reversible

Why data changes feel different from deploys

You can roll back a bad binary. You cannot always roll back a bad UPDATE. Data has customers attached to it. It has reports that were already sent, invoices that were already paid, and a second system that already consumed the event. That is why a “simple migration” becomes the riskiest work in the company.

The goal is not to be brave. The goal is to make the next change small enough that being wrong is cheap. That is an engineering problem, not a character problem.

Name the cutover before you write the script

Before anyone alters production data, write down the answers to five questions. If you cannot answer them, you are not ready to change the data. You are ready to investigate.

• What must remain true during the change? Old readers, new writers, a downstream warehouse, a legal export.
• How will you know the new shape is correct? Not “the job finished,” but a comparison you trust.
• What is the unit of rollback? A feature flag, a second column, a dual-write you can stop, a backup you have actually restored.
• Who can pause it? Name the person and the signal they will use.
• What will you not try to undo? Some writes are better compensated than reversed. Decide that while you are calm.

Prefer expand, then move, then contract

The safest production data changes add capacity before they remove it. Add the new column or table. Write to both shapes. Read from the old one until the new one matches. Shift reads. Only then remove the old shape. This is slower to describe than “run the migration.” It is much faster than an incident.

Dual writes are not free. They create a period when two truths must be kept in agreement. That period is the point. It is when you measure drift, replay the backfill, and find the records that do not fit the story you told yourself about the data.

The backfill is where you meet the business as it actually is, not as the schema promised it would be.

Validate like you mean it

A finished job is not evidence. Evidence is a comparison: row counts, checksums, sampled records, and the handful of ugly cases you already know about — the duplicate email, the user in two tenants, the order with no line items.

Build the validation before the backfill. Run it on a copy. Run it again in production with writes going to both shapes and reads still on the old one. Only flip a reader when the disagreement is understood, not merely small.

• Automate the comparison so it can run more than once. You will need it after the first surprise.
• Keep a quarantine for records that cannot be migrated automatically. Silent drops are how finance finds you later.
• Log the decision for each quarantined record. Future you will not remember why row 1842 was special.

When a rewrite of the data is the wrong move

Sometimes the request is “put this in a new database.” That can be right. It is often a way of avoiding a narrower question: which queries are slow, which writes are unsafe, which reports are lying. Measure the path that hurts. A new store with the same undifferentiated model will reproduce the same pain with a different bill.

We would rather make the next change reversible than make the next architecture impressive. If you have a data change that feels too risky to attempt, that feeling is information. It usually means the rollback is not designed yet.