LLM Systems Engineering · Chapter 2 of 20
Quantisation, with the evidence
Which formats are actually lossless, why the kernel matters more than the algorithm, and why perplexity hides the damage.
Quantisation is the most commonly recommended and least carefully evaluated optimisation in LLM serving. There is one large, rigorous study to anchor on, and a set of results that contradict the folklore in useful ways.
The formats, and the notation to use
| Format | Bits | What matters about it |
|---|---|---|
| FP32 | 32 | Baseline. Nobody serves at this |
| FP16 | 16 | 10-bit mantissa, 5-bit exponent. Narrow range — overflows during training, fine for inference |
| BF16 | 16 | Same exponent range as FP32, fewer mantissa bits. The training default |
| FP8 (E4M3 / E5M2) | 8 | Native tensor-core support on Hopper and later. The current sweet spot |
| INT8 | 8 | Needs per-channel or per-token scales to survive activation outliers |
| INT4 | 4 | Weight-only in practice, with group-wise scales |
| MXFP4 | 4 | Block size 32 with a shared exponent — about 4.25 effective bits |
| NVFP4 | 4 | Block size 16 with a finer scale — about 4.5 effective bits, better per-block accuracy |
The notation engineers actually need is W-A: W4A16 means 4-bit weights and 16-bit activations, which is weight-only — it reduces memory and speeds up decode, and does nothing for prefill compute. W8A8 quantises both, so it helps compute too. Confusing these is the source of most disappointed quantisation results.
What actually degrades — the one rigorous study
A 2024 evaluation running over 500,000 evaluations across a full model family produced four findings that should govern your defaults:
- FP8 (W8A8-FP) is effectively lossless at all model scales. If your hardware supports it, this is close to a free optimisation.
- Well-tuned INT8 (W8A8-INT) loses 1 to 3%. Real but usually acceptable.
- INT4 weight-only (W4A16) rivals 8-bit — more competitive than the folklore suggests.
- And the deployment rule, which is the most useful sentence in the paper: W4A16 wins on cost-efficiency for synchronous, low-concurrency serving and mid-tier GPUs; W8A8 wins for asynchronous high-throughput serving of larger models on high-end GPUs.
That last line follows directly from the previous chapter. Low-concurrency serving is bandwidth-bound, so shrinking the weights is what helps. High-concurrency serving has moved toward compute-bound, so quantising the activations to speed up the arithmetic is what helps. The roofline predicts the empirical result.
Three lessons from the 4-bit benchmarks that generalise
1. The kernel matters more than the algorithm. The same quantised weights ran 2.6× and 10.9× faster with an optimised kernel than with the default one — and with a bad kernel, a quantised model can be slower than FP16. If your quantisation did not speed anything up, check the kernel before you blame the method.
2. Perplexity hides task damage. In one comparison, two methods differed by about 1% in perplexity and by six percentage points on a code benchmark. Perplexity is a poor proxy for code and reasoning quality, and it is what almost every quantisation report leads with.
3. Format is not method. GGUF is a container, AWQ and GPTQ are algorithms, and a bits-and-bytes runtime is a runtime. These get conflated constantly, and the confusion produces meaningless comparisons.
GPTQ and AWQ, mechanically
- GPTQ quantises layer by layer with second-order error compensation: quantise one column, then redistribute its error into the remaining columns. It requires a calibration set and is sensitive to calibration-data mismatch.
- AWQ — activation-aware weight quantisation — observes that roughly 1% of weight channels, identified by activation magnitude rather than weight magnitude, cause most of the damage, and scales those up before quantising. It generalises better outside its calibration domain, which is exactly what the code-benchmark gap above shows.
- GGUF is the container used by CPU-and-Metal-oriented runtimes, with k-quant block schemes. It is the right tool for local and edge inference and the wrong one on a datacentre GPU — measured time-to-first-token in one comparison was nearly a second, against about 13 milliseconds for a GPU-native path.
The memory arithmetic
Serving memory is weights, plus quantisation scale overhead, plus activations and workspace, plus the KV cache (the running notes kept per conversation) — and only the last term scales with load.
| Precision | 70B weights | What it fits on |
|---|---|---|
| BF16 | 140 GB | Two 80 GB accelerators |
| FP8 | 70 GB | One 80 GB accelerator — barely, with little KV room |
| INT4 / W4A16 | ~37 GB | A single 40–48 GB card |
| NVFP4 | ~39 GB | A single Blackwell-class card |
| MXFP4 | ~37 GB | Same |
An open question worth being honest about. The reported failure mode for 4-bit floating-point formats is reasoning-heavy tasks — quantisation noise appears to compound through long chains of thought, so neither MXFP4 nor NVFP4 is a safe drop-in replacement for FP8 on mathematics, multi-step logic or code.
This is consistent with the perplexity-hides-damage finding, and there is no large-scale rigorous study for FP4 equivalent to the 500,000-evaluation work on 8-bit and 4-bit integer formats. Treat FP4 as promising and unproven, and measure it on your own task rather than on perplexity.
That is also the general rule: quantisation quality is task-specific, so evaluate on your task. A format that is lossless for summarisation may cost you several points on code generation, and the aggregate benchmark will not tell you.
Check yourself