This is the chapter that turns LLM serving from a list of techniques into a single problem. There are two numbers to be able to derive — the hardware's ridge point and the model's KV cache per token — and between them they explain continuous batching, paged attention, quantisation, speculative decoding, prefix caching and disaggregated serving. If you can do this arithmetic out loud you will be ahead of most candidates for any inference or AI platform role.

Prefill processes the whole prompt at once. Every token can be computed in parallel, so the GPU does an enormous amount of arithmetic per byte of weights it reads. Its arithmetic intensity is roughly the sequence length. It is compute-bound.

Decode generates one token at a time. For each token it must re-read the entire weight matrix, plus the growing KV cache, to produce a single output. At batch size one, the arithmetic intensity is about one operation per byte. It is memory-bandwidth-bound, severely.

The formula is worth carrying: 2 × n_layers × n_kv_heads × head_dim × bytes_per_element per token. The 2 is the key and the value. The term people get wrong is n_kv_heads — every modern model uses grouped-query or multi-query attention, where many query heads share one key/value head, so the cache is four to eight times smaller than a multi-head calculation suggests.

Work it for a 70B-class model with 80 layers, 8 KV heads and a head dimension of 128, in FP16: 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB per token. A single 8,000-token request therefore holds about 2.5 GiB of cache.

Now the consequence. An 80 GB accelerator cannot hold that model at BF16 at all. At FP8 the weights take 70 GB, leaving roughly 10 GB — which at 320 KiB per token is about 32,000 tokens of KV in total, shared across every concurrent request. Eight users at 4K context and you are full.

This is why FP8 KV-cache quantisation is one of the highest-leverage settings in serving: it halves the cache and therefore roughly doubles concurrency, at unchanged weights. And it is why architectural work on sparse and compressed attention matters commercially rather than academically — recent sparse-attention designs report roughly a tenth of the KV footprint at million-token context, which is the difference between long context being a research demo and a product.

Static batching pads every sequence in a batch to the longest one and holds the whole batch until the slowest finishes. Since output lengths vary enormously, most of the GPU's time is spent generating padding for sequences that finished long ago.

Continuous batching — also called in-flight or iteration-level batching, and originating in the Orca paper — decides batch membership at every decode step. A finished sequence is evicted immediately and a queued one admitted in its place.

The measured effect is large. A widely-cited benchmark on a 13B model reported up to 23× throughput over naive static batching, and — more instructive — showed static batching's throughput collapsing as output-length variance grew while continuous batching stayed flat. Treat the headline multiple with care: it compares against unoptimised serving rather than against a tuned static batcher. The shape of the result, that the advantage grows with variance, is the durable finding.

Chunked prefill is the companion technique. A long prompt is split into token ranges scheduled across several iterations, so one large prefill does not stall every in-flight decode. It trades a little time-to-first-token for much better inter-token latency at the tail.

Before it, serving systems allocated one contiguous KV buffer per request, sized to the maximum possible sequence length. Since most requests are far shorter, 60 to 80% of KV memory was wasted on reservation and fragmentation.

PagedAttention borrows operating-system virtual memory: fixed-size KV blocks, a per-request block table, and non-contiguous physical placement. Reported waste falls below 4%, and throughput improves two to four times at equal latency. Copy-on-write block sharing also makes parallel sampling and beam search nearly free, because the shared prefix is shared physically rather than copied.

The clean way to describe it in an interview: it is paging, applied to the KV cache, and it works for the same reason paging worked for RAM — it decouples the address space a request thinks it has from the memory it actually occupies.

A small, cheap drafter proposes k tokens. The target model verifies all k in a single forward pass — which costs almost the same as generating one token, because decode is bandwidth-bound and the weights had to be read anyway. You keep the longest accepted prefix. Speedup is roughly the number of tokens accepted per verification step.

Reported figures at batch 1 on a 70B model put modern draft-head methods at 3.0 to 3.4× decode speedup, with acceptance rates around 0.80 to 0.88 on code and instruction-following.

If prefill is compute-bound and decode is bandwidth-bound, running them on the same GPU pool means one of them is always misallocated. Disaggregation runs them on separate pools and ships the KV cache between them over a fast interconnect.

This is now the direction of the industry rather than a research idea — it is supported across the major serving engines and deployed at several large operators. The most instructive production example pools spare cluster CPU memory and SSD into a shared KV store, so requests with overlapping prefixes skip prefill entirely; that reframes the KV cache from a per-request cost into a cluster-wide asset.

A caution on the headline numbers: vendor claims of large multiples usually combine disaggregation with other changes such as wide expert parallelism, so they should not be attributed to disaggregation alone. The mechanism is sound; the specific multiple is marketing.