Software Engineering · Chapter 11 of 31
API design and caching
Contracts you cannot take back, the four cache invalidation strategies, and the two failure modes that turn a cache into an outage.
An API is a promise to people you will never meet, and a cache is a copy of the truth that is wrong some of the time. Both are places where a decision made in an afternoon constrains you for years, which is why they are worth thinking about explicitly rather than by default.
Versioning, and what actually counts as breaking
The useful distinction is not major-versus-minor; it is whether an existing client keeps working.
| Change | Breaking? | Why |
|---|---|---|
| Adding an optional response field | No | Provided clients ignore unknown fields — which they must, and which you should state in your contract |
| Adding a required request field | Yes | Existing callers do not send it |
| Making an optional request field required | Yes | Same reason, and it is easy to do accidentally through validation |
| Removing or renaming any response field | Yes | Something is parsing it |
| Changing a field's type | Yes | Including int to string, and including a numeric ID that grows past a 32-bit range |
| Adding a new enum value | Usually yes | Strict clients reject unknown values. This one is routinely underestimated |
| Tightening validation | Yes | Requests that used to succeed now fail — a breaking change with no schema change at all |
| Changing default sort order or pagination size | Yes in practice | Someone depends on it, whether or not you documented it |
The last two are worth dwelling on because they do not look like API changes and they cause real outages. Anything observable becomes a contract whether you intended it or not — which is the practical content of Hyrum's law, and it is why the honest approach is to decide what you are promising and to make everything else deliberately unstable and visibly so.
Pagination, and why offset breaks
- Offset pagination —
LIMIT 20 OFFSET 4000. Simple, supports jumping to a page, and has two real problems. It gets slower as the offset grows, because the database must produce and discard those 4,000 rows. And it is incorrect under concurrent writes: if a row is inserted before your offset between page 2 and page 3, you will see an item twice; if one is deleted, you will skip one silently. - Cursor pagination —
WHERE (created_at, id) < (:ts, :id) ORDER BY created_at DESC, id DESC LIMIT 20. Constant time regardless of depth, stable under concurrent modification, and it cannot jump to an arbitrary page. Note the tuple comparison: you need a tiebreaker column or rows with identical timestamps will be skipped or repeated, which is the bug that makes people conclude cursor pagination is broken. - The rule: cursor for anything infinite-scrolling, anything large, and any API a machine consumes. Offset only where a user genuinely needs page numbers over a small, stable set.
Errors, which are part of the contract
Return a machine-readable code alongside the human message, because clients need to branch on something and the message will be reworded eventually. Distinguish clearly between the client's fault (4xx) and yours (5xx), and tell the caller whether retrying is worthwhile — a Retry-After header on a 429 or 503 is worth a great deal more than a generic error and it is trivially cheap to add.
And never leak internal detail in an error. A stack trace or a SQL fragment in a response is an information-disclosure finding, and it is one of the most common findings in any external penetration test.
Caching: the four strategies
| Strategy | How it works | Where it fits | The catch |
|---|---|---|---|
| Cache-aside | Read cache; on a miss read the source and populate | The default — most systems, most of the time | Every miss pays full latency; a burst of misses can stampede the source |
| Read-through | The cache itself fetches on a miss | Cleaner application code | The cache becomes a dependency in the write path of your read |
| Write-through | Write to cache and source together | When reads immediately follow writes | Slower writes, and you cache data that may never be read |
| Write-behind | Write to cache, flush to source asynchronously | Very high write throughput | You can lose data. Only for things you can afford to lose |
Invalidation, which is genuinely the hard part
- TTL — simplest, and it means you serve stale data for up to the TTL. Frequently that is completely acceptable and the whole discussion is unnecessary; ask how stale is actually tolerable before engineering anything cleverer.
- Explicit invalidation on write — correct when you can enumerate every key affected by a change. The failure mode is missing one, and the resulting bug is permanent, silent and reported by a user weeks later.
- Versioned keys — include a version or an updated-at timestamp in the key, so a write naturally orphans the old entry rather than requiring you to delete it. Elegant, and the orphans must expire on their own.
- Never invalidate — make the key content-addressed, a hash of the inputs. The best option when it is available, because there is nothing to get wrong.
The two ways a cache causes an outage
Stampede (or dogpile). A popular key expires. A thousand concurrent requests all miss, all hit the database simultaneously, and the database falls over. The cache was protecting the database and its expiry is now what killed it.
Fixes: a lock or single-flight so only one request recomputes while the others wait; probabilistic early expiry so recomputation is spread out rather than synchronised; or serving stale content while refreshing in the background, which is usually the best answer because it is also the best user experience.
Cold start. You flush the cache, or restart it, or fail over to an empty one. Now 100% of traffic goes to a database sized for 5%. This is why ‘just clear the cache’ is a dangerous remediation during an incident, and why cache restarts should be staggered rather than simultaneous.
Both are worth naming in a design review, because both are entirely predictable and neither is usually planned for.
HTTP caching, which you get for free and mostly ignore
Cache-Control: max-agefor how long,s-maxagefor shared caches specifically,privatefor anything user-specific — and getting that last one wrong means a CDN serves one user's data to another, which is a genuine and recurring class of incident.ETagplusIf-None-Matchgives you a 304 with no body. The server still computes the response to know the ETag, so it saves bandwidth rather than work — unless you can compute the ETag cheaply, which is worth designing for.stale-while-revalidateserves the stale copy immediately and refreshes behind it. This is the single most under-used directive in HTTP and it solves the stampede problem at the edge for free.Varytells caches which request headers change the response. Getting it wrong either breaks correctness or destroys the hit rate — aVary: User-Agentmeans essentially every client gets its own cache entry.
The general point about cache keys is worth stating separately because it is where CDN projects fail: a cache key that includes something irrelevant has a hit rate near zero, and the team concludes the CDN does not help. Normalising the key — stripping tracking parameters, varying only on what genuinely changes the response — is usually the difference between a 40% and a 95% hit rate.
Check yourself