Lakshya

SDLC, DevOps & Developer Experience · Chapter 3 of 15

Designing a pipeline people do not route around

Order stages by how fast they fail, cache the right thing, and measure the number that actually matters to a developer.

7 min read1 diagramAll 15 chapters

A continuous integration pipeline has one job: tell a developer whether their change is safe, as fast as possible. Everything else — the stages, the caching, the parallelism — is in service of that, and a pipeline that takes forty-five minutes has stopped doing it regardless of how thorough it is.

ORDER STAGES BY HOW FAST THEY FAIL, NOT BY WHAT THEY TEST lint / format syntax, style no build needed < 10s typecheck contracts cheap, catches a lot < 60s unit tests logic no I/O, parallel < 2 min build artefact cached deps < 3 min integration real deps containers < 5 min security scan SCA, SAST, secrets in parallel < 3 min e2e / smoke the whole thing slow, flaky post-merge A developer who learns their mistake in 40 seconds fixes it in context. One who learns after 25 minutes has started something else —and the cost is the context switch, not the wait. WHERE A 45-MINUTE PIPELINE'S TIME ACTUALLY GOES — MEASURE BEFORE YOU OPTIMISE dependency install, uncached the single biggest win — cache it full test suite on every push run affected tests on push, full suite on merge queueing for a runner invisible in the build log — measure it separately sequential stages that need not be lint, typecheck and tests do not depend on each other the actual compilation usually the smallest slice, and where people look first And measure the number that matters to a developer: time from push to merged — which includes queueing, flaky reruns and waiting fora reviewer, and is usually dominated by the parts that are not the build.
Order by failure speed, and measure where the time actually goes. The bottom bars are the typical distribution, and the compilation everyone optimises first is usually the smallest slice.

Order by how fast a stage fails, not by what it tests

If a change has a syntax error, the developer should know in ten seconds — not after a four-minute build and a six-minute test run. The ordering principle is simple and frequently violated: cheapest and most likely to fail, first.

The reason this matters more than the raw duration is context switching. A developer who learns about a mistake in forty seconds is still holding the change in their head and fixes it immediately. One who learns after twenty-five minutes has started something else, and now pays to reload the context. The cost of a slow pipeline is not the waiting; it is the fragmentation.

Where the time actually goes

Instrument the pipeline per stage and look at the distribution rather than the mean — a median of forty-five minutes and a 95th percentile of forty-five minutes are different problems. Then work in order of cheapest-to-try:

  • Caching, which is almost always the biggest single win. Most pipelines reinstall dependencies from scratch on every run. A warm dependency cache and a build cache keyed on content hash routinely removes half the duration for a day's work.
  • Parallelism. Find what is sequential and need not be. Lint, typecheck and unit tests almost never depend on each other, and running them concurrently is usually a configuration change.
  • Test selection. Run the tests affected by the diff on every push, and the full suite on merge to the main branch. This is where the largest wins live and it needs a dependency graph, which is why build systems that compute one exist.
  • Bigger runners. The least elegant fix and frequently the most cost-effective — an engineer waiting fifteen minutes costs considerably more than a larger machine.
  • And measure queueing separately. Time spent waiting for a runner does not appear in the build log, so it is invisible in every dashboard and can be a large share of the total.

Measure time from push to merged, not pipeline duration

Pipeline duration is the number the platform reports and it is not the number a developer experiences.

Time from push to merged includes queueing for a runner, flaky reruns, waiting for a reviewer, and the merge-queue wait. In most organisations it is dominated by the parts that are not the build — which means a team that spends a quarter optimising build speed can move the metric that matters by almost nothing.

Set a budget and alert on it, because pipelines regress silently as people add steps. Without a budget you will be back here in six months with a pipeline nobody deliberately made slower.

Flaky tests, which destroy the pipeline's purpose

A flaky test does something worse than fail: it teaches people that a red build might mean nothing. Once developers habitually re-run rather than investigate, the pipeline has stopped being a signal and a genuine regression will be re-run away with the rest.

  • Quarantine rather than tolerate. Move a flaky test out of the blocking suite immediately, into a non-blocking one, with a ticket and an owner. Leaving it in place to be re-run is the decision that erodes trust.
  • Track flakiness as a rate — the proportion of runs where a test's result differs on the same commit — and make it visible. What is not measured stays.
  • The common causes are worth knowing, because they repeat: shared mutable state between tests, dependence on execution order, real time and timezones, real network calls, and insufficient waiting in end-to-end tests. Four of those five are fixed by isolation.
  • Automatic retries hide the problem, and there is a defensible narrow use: retry once at the end-to-end layer only, and record every retry so the rate stays visible. Retrying silently at every layer means you no longer know what your test suite is telling you.

Branching, and the honest trade

ModelHow it worksFits
Trunk-basedShort-lived branches, merged daily; incomplete work behind flagsContinuous delivery, one deployed version, mature testing. Requires feature flags and a fast pipeline
Short-lived feature branchesBranch, PR, merge within daysThe pragmatic majority. Most of the benefit of trunk-based with a lower bar
Git flowLong-lived develop and release branchesVersioned software with supported releases — installed products, mobile apps, firmware
Release branchesCut a branch, stabilise, cherry-pick fixesAnything you must patch for customers on an older version

The honest position: trunk-based development is right for continuously deployed web services and is not universal advice. If you ship a product customers install and you must patch version 4.2 while 4.3 is in development, you need release branches and the trunk-based literature does not describe your situation. Being able to say that is more useful than repeating the recommendation.

What is close to universal: the longer a branch lives, the more expensive the merge, and the cost grows faster than linearly because the conflicts interact. A three-week branch is not three times harder to merge than a one-week branch.

Where developer experience actually is

Developer experience work is often mistaken for tooling. The measurable version is friction in the inner loop — the cycle of change, build, test, observe that a developer repeats dozens of times a day.

  • Local environment setup time. If a new engineer takes three days to run the system locally, that is three days per hire, forever, plus the ongoing cost every time it breaks.
  • Inner-loop latency. How long between saving a file and seeing the result. Under two seconds and you stay in flow; over thirty and you check your messages.
  • Time to first production commit for a new joiner. A single number that captures documentation, environment, pipeline and review culture at once.
  • The proportion of a developer's day spent on their own work rather than waiting, fixing infrastructure, or chasing a flaky test. This is what all of it is for, and it is the number worth reporting upward because it converts to money without a translation step.

Check yourself

How should CI stages be ordered?By how fast they fail — cheapest and most likely to fail first. The cost of a slow pipeline is the developer's context switch, not the waiting.
What usually gives the biggest pipeline speedup?Caching dependencies and build outputs. Most pipelines reinstall from scratch every run, and warming that routinely halves the duration.
Why is a flaky test worse than a failing one?It teaches people that red might mean nothing. Once re-running is habitual, a genuine regression gets re-run away with the rest.
Is trunk-based development always right?No. It fits continuously deployed services with mature testing and feature flags. If you must patch an older version for customers, you need release branches.
← Infrastructure as code, and the state file that runs everythingDevSecOps and the software supply chain →