SDLC, DevOps & Developer Experience · Chapter 8 of 15
Large-scale migrations
The highest-value engineering skill nobody teaches, why rewrites fail for structural reasons, and the step everyone skips.
A large migration — a framework version, a database, a cloud, a language runtime, an authentication system — is one of the most common things a senior engineer is asked to lead, and one of the least written about. It is also where careers are made and quietly ended, because a migration that stalls halfway is worse than not starting.
Why the rewrite fails, structurally
It is worth being precise about this, because “rewrites fail” is repeated as folk wisdom and the reasons are specific and predictable:
- The old system does not stand still. It keeps receiving features and fixes, so you are chasing a moving target — usually with fewer people than are maintaining it.
- The requirements were never written down. They are encoded in twelve years of accumulated edge cases, and the only reliable specification is the old code, which nobody wants to read.
- No value ships until the end, which makes the first budget review impossible to survive. “What have we got for six months?” has no good answer.
- The cutover is one enormous unrehearsed risk concentrated on a single night.
- The team learns nothing until late, so every wrong assumption compounds silently until the assumptions meet reality all at once.
- And it outlasts its sponsor. The executive who approved an eighteen-month rewrite will frequently not be in the role when it needs its second budget, and the successor has no attachment to it.
The pattern that works
The strangler: put an interface in front of the old system, build the new one behind it, and move traffic across incrementally until the old one is doing nothing. The name comes from a vine that grows around a tree and eventually stands on its own.
- 1 · Facade. Route all access through an interface you control. Nothing has moved and you have bought the ability to move anything.
- 2 · Shadow. Run the new implementation alongside the old on real traffic, compare the results, and discard the new one's output. This is the most valuable and most skipped step — it finds the behaviours nobody documented, at zero user risk, using production traffic you cannot synthesise.
- 3 · Trickle. Send 1% of traffic to the new path, then 5, 25, 50. Every step is reversible with a configuration change.
- 4 · Cut over. 100% to the new path, with the old one kept warm and the rollback still one flag away.
- 5 · Delete. Actually remove the old system.
Step 5 is the one that gets skipped, and skipping it is the whole failure
Once traffic is on the new system, the organisation's attention moves on. The old one is still running, still being patched, still in the on-call rotation, still costing money.
You are now paying for two systems and reasoning about two systems, and you have captured none of the benefit that justified the project. This is the single most common way a technically successful migration becomes a commercial failure.
So: put the deletion in the plan as a dated deliverable with an owner, not as a follow-up. Track the number of remaining callers as the project's headline metric, because it is the only one that goes to zero. And be ruthless about the last 5% — it is usually one team with one unusual dependency, and it will hold the whole thing open for a year if nobody goes and sits with them.
Data migration, which is a different and harder problem
Moving code is reversible. Moving data usually is not, and it needs its own plan:
- Dual-write before you dual-read. Write to both stores, read from the old. Then compare continuously and fix the discrepancies. Only then start reading from the new.
- Reconciliation is the whole job. A background job that compares the two stores and reports differences is what turns ‘we think it's correct’ into evidence. Expect the first run to find problems — if it finds none, suspect the reconciliation rather than celebrating.
- Backfill is where the surprises are. Historical data has years of format changes, nulls that shouldn't exist, encodings nobody remembers, and records written by code that was deleted in 2019. Budget for the backfill separately from the migration; it is routinely the larger half.
- Decide what to do about the un-migratable. There will be records that cannot be transformed cleanly. Deciding early — fix, drop, or carry in a quarantine table — is much better than discovering it at 80% and stalling.
Automating the change itself
For a migration that touches hundreds or thousands of call sites — a framework upgrade, an API change, a language version — doing it by hand is slow and produces inconsistent results. The alternative is worth knowing about because it converts a year of grinding into a few weeks:
- Write a codemod — a program that parses the code into a syntax tree, transforms it, and writes it back. Unlike a regular expression, it understands the code's structure, so it does not break on formatting or comments.
- Run it, review the diff, iterate. The first pass will get 80% right; you fix the transform rather than the output, and re-run.
- Land it in many small pull requests, one per team or per module, so review is possible and a problem affects one area rather than everything.
- And add a lint rule that prevents the old pattern returning. Without it, the old form reappears within weeks in new code, and you will do the migration twice. This is the step that makes it stick and it costs almost nothing.
Running it as a programme
Publish a burn-down of remaining call sites, and make it the headline number. ‘414 remaining, down from 3,200’ is visible progress that survives a sponsor change; ‘phase 2 in progress’ does not.
Do the first three yourself. Migrating three real call sites teaches you what the guide needs to say, and a migration guide written by someone who has not done one is always wrong in the same places.
Then make it self-service. A guide, a codemod, an example pull request, and a channel where someone answers within a day. If every migration needs your involvement, the throughput is your calendar.
Decide the forcing function early and say it out loud. A deprecation date, a build warning that becomes an error, or the old path simply being switched off. Without one, the last 20% never happens — and it is kinder to announce the date at the start than to discover in month nine that you need one.
And name the cost of not finishing. Migrations lose funding to features constantly, and the argument that works is not technical purity — it is the concrete cost of running two systems, the security exposure of the unsupported one, and the engineering time being spent twice.
How to talk about one in an interview
This is a common senior interview question and most answers are a project narrative. The structure that lands: what made it hard, what you did to make each step reversible, how you got other teams to move, and whether you actually finished.
That last part is worth being honest about. “We got to 95% and the last five were three teams who never prioritised it, so I got a deprecation date agreed and it completed the following quarter” is a much better answer than an unbroken success, because everyone who has run one recognises it.
Check yourself