Handbook · Chapter 3
Design under constraint
The decisive round for SRE, AI Platform, Infrastructure — and the reason strong engineers fail it.
Every family on this site has one round where you design a system out loud. The failure mode is identical across all of them: candidates describe an architecture without ever producing a number. Google's version has a name for the cure — NALSD, non-abstract large system design — and the non-abstract part is the whole exam. You are expected to arrive at machine counts, disk throughput and failure-domain layout, not a diagram of boxes.
The four habits that pass this round
- State the assumption, then use it. “I'll assume a 3× peak-to-average factor — tell me if that's wrong for your traffic” is a strong move. It shows you know the number matters and invites the interviewer to correct you, which turns the round into collaboration.
- Round aggressively and say you are rounding. 86,400 seconds a day is 100,000. Nobody wants long division; they want to see whether you know which quantities dominate.
- Name the binding constraint. Almost every system is limited by one thing — memory, IOPS, network, GPU memory, a rate limit. Finding it early is most of the answer, and it is the thing junior candidates never do.
- Design for the failure you were not asked about. The requirement says serve the load; the job is to serve it while a node, a zone, or a dependency is gone.
The AI-family variant
Same ladder, different arithmetic
For AI Platform and Research roles the estimation is about memory rather than machines, and it gets asked directly and answered badly. Have this decomposition ready:
| Component | Rough size | What moves it |
|---|---|---|
| Model weights | params × bytes per param | fp16 is 2 bytes; int8 quantisation halves it |
| Optimiser state (training) | roughly 2–3× the weights | Adam keeps two moments per parameter |
| Activations (training) | scales with batch × depth | gradient checkpointing trades compute for memory |
| KV cache (inference) | scales with batch × context length | the term people forget, and it dominates at long context |
The interview question is usually a variant of “estimate the memory to serve a 70B model at batch 32 with 8k context”. What is being tested is not the exact figure; it is whether you reach for weights and KV cache, and whether you notice that the cache term grows with both batch and context.
The drill
- Write five capacity questions on cards — a URL shortener, a chat backend, an inference endpoint, a log pipeline, a CI fleet. Answer one per day, out loud, on a fifteen-minute timer, ending each with a machine count and a stated assumption list.
- Then do the AI variant: pick three model sizes and three context lengths and compute serving memory for each until the decomposition is automatic.
- Record one. The recording is where you discover you spent nine minutes describing components and never produced a number.
- “Design a service for 50k QPS at 99.9% availability.”They want machine counts and failure domains, not a component diagram.
- “p50 is fine, p99 is terrible. Where do you look?”Queueing, batching, preemption, cold starts — and see the next chapter on why chains make this worse.
- “What is the binding constraint in what you just designed?”If you cannot name one, you have not designed it yet.