SDLC, DevOps & Developer Experience · Chapter 1 of 15
Getting code into production without breaking it
Five strategies, what each one genuinely costs, and the version-skew problem that makes most of them harder than they look.
How you deploy determines how confidently you can change things, which determines how fast you can move. Every strategy below is a different answer to one question: how much of your traffic are you willing to expose to a change you have not yet proved?
The problem underneath all of them: version skew
With the exception of a full-stop replacement, every strategy here runs two versions of your code simultaneously — during a rolling update, throughout a canary, and across a blue-green cutover while connections drain. That single fact generates most of the real difficulty.
- Database schema changes must be backward compatible for the duration. The old code must keep working against the new schema. This is why the expand-and-contract pattern exists and why it is non-negotiable at any scale.
- API changes must be additive. A new required field breaks the old callers still running beside you.
- Message formats must be readable by both. A queue holds messages written by the new version and consumed by the old one.
- Cached data must be compatible. A serialisation change means the new version cannot read what the old one cached, or worse, misreads it.
Expand and contract — the only safe way to change a schema
It is three deploys, and skipping the middle one is how teams cause outages they describe as ‘the migration went wrong’.
1 · Expand. Add the new column as nullable. Deploy code that writes to both the old and the new, and reads from the old. Nothing breaks, because the old code still sees everything it expects.
2 · Migrate. Backfill the new column. Deploy code that reads from the new and still writes to both. Now you can verify correctness with a real rollback available.
3 · Contract. Once nothing reads the old column and you are confident, stop writing it, then drop it — and leave a gap of days or weeks before the drop, because a dropped column is the one change you cannot roll back.
Renaming a column is the same problem in disguise, and it is where people slip: a rename is a drop and an add executed atomically, which means there is a moment when the running old code sees neither. Add, dual-write, migrate, remove.
Choosing between them
| Situation | Strategy | Why |
|---|---|---|
| Stateless service, good metrics, real traffic volume | Canary | Small blast radius, and real traffic finds what staging cannot |
| You need instant, certain rollback | Blue-green | Flip the router back in seconds. Costs double infrastructure while both run |
| Ordinary service, no strong rollback requirement | Rolling | No extra infrastructure. Accept the version-skew discipline |
| The risk is in the behaviour, not the code | Feature flag | Deploy dark and release to a cohort. Separates the two decisions entirely |
| Low traffic, cannot justify the machinery | Rolling, with a fast rollback path | A canary needs enough traffic to produce a signal — below that it is theatre |
| Stateful system, database, or a broker | None of the above alone | Needs its own plan. Data has no rollback |
The last row deserves emphasis. Stateless services roll back; data does not. You can redeploy yesterday's binary in a minute, and you cannot un-write the rows it produced. Any change that mutates data needs a separate, explicit reversal plan — and it should be written down before the change, because writing it during an incident is how bad decisions happen.
Canary analysis, done properly
A canary that nobody watches is a slow deploy. The value is entirely in the comparison, and there are three ways to get it wrong:
- Compare canary against a baseline, not against history. Run a small control on the old version at the same moment and compare the two. Comparing this hour's canary against last hour's fleet confounds your change with time of day, load and everything else that moved.
- Gate on the right signals: error rate, latency at the tail rather than the mean, saturation, and at least one business metric. A deploy that halves your conversion rate while every technical metric stays green is exactly the case a technical-only gate misses.
- Bake long enough for the failure mode you fear. Five minutes catches a crash loop. It does not catch a memory leak, a connection-pool exhaustion, or anything that only appears at a daily peak. Match the bake time to the slowest failure you are trying to detect, and be honest that a fast canary is only screening for fast failures.
Feature flags, and the debt they create
Flags are the most powerful of these techniques because they separate deploying code from releasing behaviour. Code ships continuously, dark; the release is a configuration change with instant reversal and no deploy.
They also accumulate. A codebase with four hundred flags has an unknowable number of code paths — combinatorially, it is untestable — and every one is a branch someone must reason about. The disciplines that work:
- Every flag has an owner and an expiry date at creation. Not a policy, a required field.
- Alert on stale flags the same way you alert on anything else that is overdue.
- Distinguish the two kinds. A release flag is temporary and must be removed. An operational flag — a kill switch, a rate limiter, a degradation toggle — is permanent infrastructure and should be treated as such. Conflating them is why flag clean-up projects stall: people are reluctant to delete flags that turn out to be load-bearing.
- Flags are a control-plane dependency. If your flag service is down, what do your services do? The answer must be a safe default cached locally, not a failure — a flag system that can take down every service that reads it has inverted the whole point.
What to measure
Four numbers describe a delivery system, and the useful thing about them is that they trade off against each other in ways that expose whether you are actually improving:
| Metric | What it measures | The trap |
|---|---|---|
| Deployment frequency | How often you ship | Easy to game by deploying nothing |
| Lead time for change | Commit to production | The number people quote; measure the whole path including review and waiting, not just the pipeline |
| Change failure rate | Proportion of deploys causing an incident | Rises if you only chase frequency — which is the point of tracking both |
| Time to restore | How fast you recover | The most under-rated of the four. A team that recovers in five minutes can afford to take risks a team that recovers in five hours cannot |
The pairing is what matters: frequency and failure rate together, lead time and restore time together. Optimising either one alone produces a predictable pathology, and a team reporting only the flattering half of each pair is telling you something.
Check yourself