Lakshya

SDLC, DevOps & Developer Experience · Chapter 8 of 15

Large-scale migrations

The highest-value engineering skill nobody teaches, why rewrites fail for structural reasons, and the step everyone skips.

8 min read1 diagramAll 15 chapters

A large migration — a framework version, a database, a cloud, a language runtime, an authentication system — is one of the most common things a senior engineer is asked to lead, and one of the least written about. It is also where careers are made and quietly ended, because a migration that stalls halfway is worse than not starting.

THE ONLY MIGRATION SHAPE THAT RELIABLY FINISHES 1 · facade put an interface in front of the old nothing has moved yet 2 · shadow new system runs too results compared, discarded zero user risk 3 · trickle 1% of traffic to new then 5, 25, 50 reversible at every step 4 · cut over 100% to new old kept warm still reversible 5 · delete actually remove the old the step that gets skipped only now is it done Step 5 is the one organisations skip, and skipping it is how you end up running both systems for three years — paying twice andreasoning about two. WHY BIG-BANG REWRITES FAIL — AND IT IS NOT INCOMPETENCE the old system keeps changingyou are chasing a moving target with fewer peoplethe requirements were never written downthey are encoded in twelve years of edge casesno value ships until the endso the first budget question is impossible to answerthe cutover is one enormous riskand there is no rehearsal for itthe team learns nothing until lateevery wrong assumption compounds silentlyand it takes longer than the sponsorwho approved it will be in the role Incremental migration is slower in the plan and faster in reality, because every step delivers value and every step is reversible —which is what lets it survive a change of sponsor.
The strangler pattern, and why the alternative fails. Step 5 is the one organisations skip, and the reasons in the lower box are structural rather than a matter of competence.

Why the rewrite fails, structurally

It is worth being precise about this, because “rewrites fail” is repeated as folk wisdom and the reasons are specific and predictable:

  • The old system does not stand still. It keeps receiving features and fixes, so you are chasing a moving target — usually with fewer people than are maintaining it.
  • The requirements were never written down. They are encoded in twelve years of accumulated edge cases, and the only reliable specification is the old code, which nobody wants to read.
  • No value ships until the end, which makes the first budget review impossible to survive. “What have we got for six months?” has no good answer.
  • The cutover is one enormous unrehearsed risk concentrated on a single night.
  • The team learns nothing until late, so every wrong assumption compounds silently until the assumptions meet reality all at once.
  • And it outlasts its sponsor. The executive who approved an eighteen-month rewrite will frequently not be in the role when it needs its second budget, and the successor has no attachment to it.

The pattern that works

The strangler: put an interface in front of the old system, build the new one behind it, and move traffic across incrementally until the old one is doing nothing. The name comes from a vine that grows around a tree and eventually stands on its own.

  • 1 · Facade. Route all access through an interface you control. Nothing has moved and you have bought the ability to move anything.
  • 2 · Shadow. Run the new implementation alongside the old on real traffic, compare the results, and discard the new one's output. This is the most valuable and most skipped step — it finds the behaviours nobody documented, at zero user risk, using production traffic you cannot synthesise.
  • 3 · Trickle. Send 1% of traffic to the new path, then 5, 25, 50. Every step is reversible with a configuration change.
  • 4 · Cut over. 100% to the new path, with the old one kept warm and the rollback still one flag away.
  • 5 · Delete. Actually remove the old system.

Step 5 is the one that gets skipped, and skipping it is the whole failure

Once traffic is on the new system, the organisation's attention moves on. The old one is still running, still being patched, still in the on-call rotation, still costing money.

You are now paying for two systems and reasoning about two systems, and you have captured none of the benefit that justified the project. This is the single most common way a technically successful migration becomes a commercial failure.

So: put the deletion in the plan as a dated deliverable with an owner, not as a follow-up. Track the number of remaining callers as the project's headline metric, because it is the only one that goes to zero. And be ruthless about the last 5% — it is usually one team with one unusual dependency, and it will hold the whole thing open for a year if nobody goes and sits with them.

Data migration, which is a different and harder problem

Moving code is reversible. Moving data usually is not, and it needs its own plan:

  • Dual-write before you dual-read. Write to both stores, read from the old. Then compare continuously and fix the discrepancies. Only then start reading from the new.
  • Reconciliation is the whole job. A background job that compares the two stores and reports differences is what turns ‘we think it's correct’ into evidence. Expect the first run to find problems — if it finds none, suspect the reconciliation rather than celebrating.
  • Backfill is where the surprises are. Historical data has years of format changes, nulls that shouldn't exist, encodings nobody remembers, and records written by code that was deleted in 2019. Budget for the backfill separately from the migration; it is routinely the larger half.
  • Decide what to do about the un-migratable. There will be records that cannot be transformed cleanly. Deciding early — fix, drop, or carry in a quarantine table — is much better than discovering it at 80% and stalling.

Automating the change itself

For a migration that touches hundreds or thousands of call sites — a framework upgrade, an API change, a language version — doing it by hand is slow and produces inconsistent results. The alternative is worth knowing about because it converts a year of grinding into a few weeks:

  • Write a codemod — a program that parses the code into a syntax tree, transforms it, and writes it back. Unlike a regular expression, it understands the code's structure, so it does not break on formatting or comments.
  • Run it, review the diff, iterate. The first pass will get 80% right; you fix the transform rather than the output, and re-run.
  • Land it in many small pull requests, one per team or per module, so review is possible and a problem affects one area rather than everything.
  • And add a lint rule that prevents the old pattern returning. Without it, the old form reappears within weeks in new code, and you will do the migration twice. This is the step that makes it stick and it costs almost nothing.

Running it as a programme

Publish a burn-down of remaining call sites, and make it the headline number. ‘414 remaining, down from 3,200’ is visible progress that survives a sponsor change; ‘phase 2 in progress’ does not.

Do the first three yourself. Migrating three real call sites teaches you what the guide needs to say, and a migration guide written by someone who has not done one is always wrong in the same places.

Then make it self-service. A guide, a codemod, an example pull request, and a channel where someone answers within a day. If every migration needs your involvement, the throughput is your calendar.

Decide the forcing function early and say it out loud. A deprecation date, a build warning that becomes an error, or the old path simply being switched off. Without one, the last 20% never happens — and it is kinder to announce the date at the start than to discover in month nine that you need one.

And name the cost of not finishing. Migrations lose funding to features constantly, and the argument that works is not technical purity — it is the concrete cost of running two systems, the security exposure of the unsupported one, and the engineering time being spent twice.

How to talk about one in an interview

This is a common senior interview question and most answers are a project narrative. The structure that lands: what made it hard, what you did to make each step reversible, how you got other teams to move, and whether you actually finished.

That last part is worth being honest about. “We got to 95% and the last five were three teams who never prioritised it, so I got a deprecation date agreed and it completed the following quarter” is a much better answer than an unbroken success, because everyone who has run one recognises it.

Check yourself

Why do big-bang rewrites fail?Structurally: the old system keeps changing, requirements exist only as accumulated edge cases, no value ships until the end, the cutover is one unrehearsed risk, and the project outlasts its sponsor.
What is the most valuable and most skipped migration step?Shadow mode — running the new implementation on real traffic and discarding its output. It finds undocumented behaviour at zero user risk, using traffic you cannot synthesise.
Which step do organisations skip, and what does it cost?Deleting the old system. You then pay for two systems, reason about two, and capture none of the benefit that justified the project.
How do you make a code migration stick?A codemod for the bulk change, small per-team pull requests, and a lint rule preventing the old pattern from returning. Without the lint rule you will do it twice.
← Day zero to the first pull requestThe CI/CD platform landscape →