LLM Systems Engineering · Chapter 3 of 20
Serving stacks, and prompt caching
Five engines and what each is actually for, plus the caching mechanism where most production money is quietly lost.
Choosing a serving engine looks like a benchmark question and is mostly a workload question. And prompt caching — which nearly every application could use and most use badly — is where the largest easy cost savings live.
The five engines
| Engine | What it actually is | Pick it when |
|---|---|---|
| vLLM | The default open production engine. PagedAttention plus continuous batching, broadest model and hardware coverage | The default. Multi-cloud, non-NVIDIA hardware, models that change often |
| SGLang | A vLLM-class engine whose differentiator is a radix tree over the KV cache (the running notes kept per conversation), giving automatic cross-request prefix reuse | Traffic with a shared prefix — chatbots on one system prompt, RAG, agents replaying long histories |
| TensorRT-LLM | A compiler, not just a runtime. Builds a fused hardware-specific engine ahead of time | A stable model on NVIDIA hardware where peak throughput justifies a build step and the lock-in |
| llama.cpp | Single-binary C/C++ inference with GGUF, CPU/Metal/partial offload | Local, edge, Apple Silicon, CPU-only, no-Python environments |
| Ollama | A packaging and UX layer over llama.cpp | Prototyping and single-user local development. Not a server |
The honest reading of the engine benchmarks
Published comparisons put a radix-tree engine roughly 29% above vLLM on prefix-heavy traffic and within 1 to 4% on unique prompts. Essentially all of the advantage is prefix reuse.
So the advantage is a property of your workload, not of the engine. If your traffic shares prefixes, measure it. If every request is unique, choose on model support and ecosystem instead.
Compiler-based serving is reported 15 to 30% above vLLM at sustained high load, at the cost of a compile step per model and configuration measured in tens of minutes.
And the non-server case is worth stating plainly: a single-user local runtime has no batching at all, so at five concurrent users its tail latency is several times its single-user baseline, while a production engine holds 85 to 92% GPU utilisation at a hundred. People deploy the former to production regularly.
Caveat on all of the above: these projects leapfrog each other release to release, and any specific number has a short shelf life. Teach the mechanism — prefix reuse, compilation, batching — and re-measure the numbers.
Prompt caching: one idea, three vendor skins
Every major provider now caches a prefix of your prompt and charges dramatically less to re-read it — typically about a tenth of the normal input rate. The implementations differ in whether it is automatic, how long it lives, and what invalidates it. The design rule does not differ at all.
| Anthropic | OpenAI | ||
|---|---|---|---|
| Opt in? | Explicit — cache_control breakpoints | Automatic, on by default | Implicit on by default; explicit API also available |
| Read discount | ~0.1× input | ~0.1× input | ~90% off (75% on older models) |
| Write cost | 1.25× for 5-minute TTL, 2× for 1-hour | — | Implicit is free; explicit charges storage per hour |
| TTL | 5 minutes or 1 hour | ~30 minutes on current models | ~60 minutes default for explicit |
| Minimum prefix | Model-dependent and not monotonic — 512 to 4,096 tokens depending on the model | 1,024 tokens on current models, 2,048 on older | 2,048 rising to 4,096 on newer models |
| Breakpoints | Up to 4 explicit | n/a — automatic | n/a for implicit |
The break-even arithmetic is worth doing once. With a write at 1.25× and a read at 0.1×, a five-minute cache pays for itself after a single reuse: 1.25 + 0.1 = 1.35 against 2.0 for two uncached passes. A one-hour cache at 2× write pays off after two reuses. So for anything conversational, caching is essentially always correct.
The traps, all of which fail silently.
Below the minimum length there is no error — you simply get no cache, and the usage field reports zero cached tokens. Check the field rather than assuming.
The minimum is not monotonic across a vendor's model tiers. A newer, larger model can have a lower minimum than an older, smaller one. Porting a caching design between models without re-checking is a real and invisible regression.
Invalidation is tiered by position. Because it is a prefix cache over the rendered order — tools, then system, then messages — changing your tool definitions invalidates everything, while changing the system prompt leaves tools cached. This is why adding one tool to an agent can double your bill overnight.
Non-text settings can invalidate. Reasoning effort, output format and compaction settings are invalidators on some providers. Changing effort mid-session drops your prefix cache even though the prompt bytes are identical.
Concurrent identical requests all miss. The cache entry only becomes readable once the first response starts streaming, so firing N identical requests in parallel pays full price N times. Warm the cache with one request first if you are fanning out.
And the classic: an unsorted json.dumps(), a datetime.now(), a per-user tool list, or a session ID at the front of the prefix. Any of these destroys the cache regardless of how carefully you placed your breakpoints.
The engineering practice that follows: treat the prompt prefix as a build artefact. Construct it deterministically, sort every serialised structure, put anything per-request at the end, and assert on the cached-token count in your integration tests. A cache hit rate is a metric worth a dashboard, because a regression in it is invisible in every other signal and shows up only on the invoice.
Check yourself