SDLC, DevOps & Developer Experience · Chapter 5 of 15
A testing strategy that survives a real codebase
The pyramid as a cost heuristic rather than a law, the layer almost everyone is missing, and how to decide what to test where.
Testing advice is unusually dogmatic for something so context-dependent. The useful version is a single principle: put each test at the cheapest level that can actually catch the bug you are worried about — and no lower. Everything else follows.
What each layer can and cannot see
- Unit tests catch logic errors in a function. They are effectively free, they run in milliseconds, and they cannot catch anything about how pieces fit together. A suite of unit tests over heavily mocked code can be entirely green while the system does not work at all, because every mock encodes an assumption that was never checked.
- Integration tests use a real database, a real queue, a real HTTP call. They catch the things unit tests structurally cannot: wrong SQL, serialisation mismatches, connection and configuration problems, transaction behaviour. Containers made these cheap enough that the old argument for mocking the database has largely evaporated.
- End-to-end tests exercise the whole system as a user would. They catch integration of everything and they are slow and flaky, so a small number of high-value paths is right and a hundred is a mistake.
- Manual and exploratory testing catches what nobody thought to assert. This is the layer people write off as unprofessional, and it is the only one that finds the unanticipated. A skilled tester with an hour and no script finds a class of problem no suite will.
The two opposite failure modes, and both are common
The ice-cream cone — hundreds of slow browser tests, very few unit tests. The suite takes forty minutes, fails randomly, and nobody trusts it. This is what the pyramid warns against, and it is real.
The inverted problem the pyramid causes — thousands of unit tests over code so heavily mocked that the tests assert the mocks rather than the behaviour. Coverage says 95%. Nothing in the suite would notice if two services stopped agreeing on a message format.
So the diagnostic question is never ‘what shape is our pyramid?’ It is: what bugs actually reached production in the last six months, and which layer should have caught each one? That question produces a specific, defensible testing plan; the shape question produces an argument.
Contract testing, the layer most teams are missing
In a system of several services, the expensive failures are almost never inside a service. They are at the boundary: a producer changed a field and a consumer broke, and neither team's tests noticed because each tested itself in isolation.
Contract testing fixes exactly this. The consumer declares what it needs from the producer, in an executable form. That expectation is shared, and the producer's pipeline verifies that it still satisfies every declared consumer contract before it can deploy.
- The value is that it runs in the producer's pipeline. The team making the change learns immediately, in their own build, rather than the consumer learning in production. That is what turns a coordination problem into a test failure.
- It does not require end-to-end tests. Both sides are tested in isolation against the shared contract, so you get boundary safety without the slowest layer.
- Its limit is honest: it verifies the shape and the semantics you declared, not that the overall business flow is correct. It is a boundary check, not a system check.
- Where it earns its cost: more than a handful of services, teams deploying independently, and a history of breaking each other. Below that, it is ceremony — a two-service system does not need this machinery.
Test doubles, which are four different things
“Mock” has become a catch-all and the distinctions matter, because choosing the wrong one is what produces tests that assert their own setup:
| Kind | What it is | Use it when |
|---|---|---|
| Stub | Returns canned answers | You need the collaborator to return something and do not care that it was called |
| Mock | Asserts it was called, how, and how often | The interaction is the behaviour you are testing — an email was sent, a payment was charged |
| Fake | A real working implementation, simplified — an in-memory repository | The most under-used and usually the best. Real behaviour, no infrastructure |
| Spy | A real object that records how it was used | You want real behaviour and to assert on the interaction |
The rule that prevents most trouble: mock at the boundary you own, not at every seam. Mocking your own internal classes couples the test to the implementation, so any refactor breaks a hundred tests without a single behaviour changing — and a test suite that punishes refactoring is a suite that prevents improvement. Mock the third-party payment gateway; use a real database in a container.
Coverage, and what it is actually for
Coverage measures which lines executed during the test run. It does not measure whether anything was asserted, and a test that calls a function and asserts nothing produces identical coverage to a thorough one.
- Low coverage is a genuine signal. If a module is at 10%, it is untested. That is useful information.
- High coverage is not a signal. 95% tells you lines ran, not that behaviour was checked.
- A coverage target becomes a target. Mandate 80% and you get tests written to reach 80%, which are the tests that assert least.
- What to use instead: coverage on the diff, as information rather than a gate — ‘this change added 200 lines and 40 are untested’ is a useful thing for a reviewer to see and a bad thing to block on.
- And if you want to know whether your tests are real, use mutation testing. It changes the code deliberately — flips a comparison, removes a call — and checks whether any test fails. A surviving mutant is a line that is covered and unprotected. It is slow, so run it periodically on critical modules rather than in the pipeline, and be prepared for the result to be uncomfortable.
Where AI-generated code changed the calculation
Two effects, pulling in opposite directions, and both worth naming.
Writing tests got much cheaper, which removes the most common excuse for not having them. And the value of tests went up, because more code is now written by something that does not understand the system's invariants and produces plausible-looking code that compiles.
But a caution that follows directly: tests generated from the implementation assert what the code does, not what it should do. If the implementation is wrong, the generated test enshrines the bug and gives you confidence in it. The practice that survives this is to write or review the assertions against the requirement, not against the code — which is the same discipline as before, applied with more suspicion.
Check yourself