Software Engineering · Chapter 10 of 31
Distributed systems: the failures that only happen in production
Retries that cause the outage, idempotency, the two-generals problem in a payments API, and why 'just add a queue' moves the problem rather than solving it.
A distributed system is one where a machine you have never heard of can make your program fail. The move from a single process to several introduces a small number of failure modes that do not exist locally, and they are the ones that produce the incidents nobody could reproduce.
The one fact everything follows from
When a call fails, you cannot tell whether it happened. A timeout means you did not get a response. It does not mean the work was not done — the request may have been processed and the response lost, or it may have been dropped on the way in. From your side these are indistinguishable, and no amount of engineering makes them distinguishable.
This is the two-generals problem, and it is not a theoretical curiosity: it is why a payment can be taken twice, why an email can be sent twice, and why ‘did the order go through?’ is a genuinely hard question. Everything below is a response to it.
Idempotency, which is the actual answer
If an operation can be safely repeated, then the ambiguity stops mattering: on a timeout you simply retry, and whether the first attempt landed becomes irrelevant.
- The client generates an idempotency key — a UUID — and sends it with the request. It must be generated by the client and reused across retries; a key the server generates cannot help, because the client never saw it.
- The server stores the key with the result, atomically with the work itself. This is the part implementations get wrong: if the key is recorded in a different transaction from the effect, a crash between them leaves you with either a duplicate or a lost result.
- A repeat returns the stored result rather than redoing the work.
- Concurrent duplicates need a lock or a unique constraint, because two copies of the same request can arrive simultaneously. A unique index on the key is the simplest correct implementation and lets the database arbitrate.
- Keys expire, typically after 24 hours, and you must decide what a repeat after expiry means.
Some operations are naturally idempotent — setting a value, deleting by ID. Some can be made so — UPDATE ... WHERE version = n. Some cannot be, and those genuinely need the key.
Retries, and how they cause the outage they were meant to prevent
The mechanism, which is worth being able to draw. A downstream service slows down. Every caller times out and retries. The retries are additional load on a service that was already struggling, so it slows further, so more callers time out, so more retries. A service that was at 90% capacity is now at 300% and will not recover on its own even after the original cause is removed.
This is a metastable failure, and the defining property is that removing the trigger does not fix it — the retries are now the cause. It is why some outages require a deliberate load shed to break out of, and why ‘we restarted everything and it came back’ is a common incident narrative.
Exponential backoff with jitter is mandatory, not optional. Without jitter, every client backs off by the same amount and returns in a synchronised wave. Full jitter — a random wait between zero and the current cap — is the standard and it is measurably better than the alternatives.
Budget your retries. Cap them as a fraction of total requests rather than per-call, so a broad failure cannot multiply your traffic. A retry budget of 10% means the worst case is 1.1× load rather than 3×.
Never retry at every layer. Three retries at the client, three at the gateway and three at the service is twenty-seven attempts for one user action. Decide which single layer owns retrying and make the others pass failures through.
Only retry what is safe. A timeout on a non-idempotent write is exactly the case where retrying may double-charge someone.
Circuit breakers, timeouts and bulkheads
- Timeouts must exist everywhere, and they must be shorter as you go deeper. If your caller's timeout is 5 seconds, your call to a dependency cannot be 10 — you will be doing work nobody is waiting for. Budget the deadline and propagate it, which is what gRPC deadlines and context cancellation are for. The default timeout in most HTTP clients is infinite, and that is the cause of more hung services than any other single default.
- Circuit breakers stop calling a dependency that is clearly failing. Closed, then open after a failure threshold, then half-open to test recovery with a trickle. The value is not for you — it is that you stop adding load to something that is already failing, which is what lets it recover.
- Bulkheads give each dependency its own connection pool or thread pool, so one slow dependency cannot consume all your capacity. Without them, a single slow downstream takes down an entire service that only used it for one non-critical feature.
- Load shedding is the one people leave out and it is what actually saves you: when overloaded, reject requests fast rather than queueing them. A fast rejection lets the caller retry elsewhere or degrade; a slow queue converts overload into total failure. Serving 70% of traffic well beats serving 100% of it too slowly to be useful.
Exactly-once, and why it does not exist
You will see ‘exactly-once delivery’ claimed by messaging systems. In the strict sense it is impossible: a network can always lose the acknowledgement, so the sender cannot know whether to resend.
What is achievable is at-least-once delivery with idempotent processing, which produces exactly-once effects. When a system claims exactly-once, it almost always means it provides deduplication within a window, or transactional writes to its own storage — both genuinely useful, and neither covering an effect in a system it does not control.
The practical consequence for a consumer: assume every message may be delivered more than once and make processing idempotent. Being able to say that clearly — including why the guarantee cannot extend past the system's boundary — is a strong senior signal.
Consistency, in the version that matters for design
| Model | What it promises | Where you meet it |
|---|---|---|
| Strong / linearisable | A read sees the latest write, always | A single database, Spanner, etcd. Costs latency and availability during partitions |
| Read-your-writes | You see your own writes; others may lag | What users actually notice. Usually achievable by routing a user's reads to the primary briefly after a write |
| Monotonic reads | You never see time go backwards | Prevents the maddening bug where a refresh shows older data than the previous page load |
| Eventual | Replicas converge, eventually | Most caches, most cross-region replication, DNS |
The design point: you rarely need strong consistency globally, and you almost always need read-your-writes for the acting user. Getting that distinction right removes most of the user-visible weirdness of a replicated system without paying the cost of global linearisability — and it is a much better answer in a design interview than either ‘use strong consistency’ or ‘eventual is fine’.
“Just add a queue” — what it actually changes
A queue converts a synchronous dependency into an asynchronous one. That is genuinely valuable: the caller stops waiting, spikes get absorbed, and the consumer can fail without failing the producer.
What it does not do is remove the problem. It relocates it, and the new problems are real: the work is now eventually consistent, so the user's expectation must change. The queue can grow without bound, so you need a dead-letter queue and an alert on depth. Ordering is usually only guaranteed within a partition, so a global ordering assumption will break. And consumers must be idempotent, because redelivery is normal.
The honest sentence in a design review: “A queue is right here because the caller genuinely does not need the result, and it costs us eventual consistency plus a dead-letter path we have to operate.” That is a decision. “Add a queue” is a reflex.
Check yourself