LLM Systems Engineering · Chapter 7 of 20
Fine-tuning, and what actually won
LoRA mechanics and the rank trade-off, the evidence on when tuning beats retrieval, and how reinforcement learning modularised into three methods with different jobs.
Fine-tuning is the most requested and least often correct answer in applied AI. This chapter is the mechanics, the evidence on when it helps, and the current state of post-training — which changed substantially and is usually taught a generation behind.
LoRA, mechanically
Freeze the original weights and learn a low-rank update: W = W₀ + (α/r)·BA, where the two factor matrices have inner dimension r, much smaller than the layer's dimensions. B initialises to zero, so the adapted model starts identical to the base.
At inference you can either merge the update into the weights — zero latency cost — or keep adapters separate and swap them per request, which is how multi-tenant adapter serving works.
The rank and alpha are not independent. The scaling factor is α/r, so the widely-used heuristic α = 2r keeps the effective update magnitude roughly constant as you sweep rank. Changing r with α fixed changes both capacity and effective learning rate at once, which is why naive rank sweeps produce confusing results that people then attribute to the rank.
What the evidence says about LoRA versus full fine-tuning
A careful 2024 study produced findings that should govern how you think about this:
- LoRA learns less and forgets less. Relative to full fine-tuning it acquires less of the new task and preserves more of the base model's general capability — a genuine regularisation effect, and it beats classical regularisers like dropout and weight decay on that frontier.
- Forgetting is inversely related to rank. Lower rank preserves base knowledge better; raising rank shifts you toward more learning and more forgetting. That gives you a principled knob rather than a folk one.
- The gap to full fine-tuning is larger for continued pretraining — teaching genuinely new domain knowledge — than for instruction tuning, which teaches behaviour. LoRA is closest to full fine-tuning exactly where most people use it.
- And the mechanistic explanation: singular-value analysis shows the rank of full fine-tuning's weight perturbations is ten to a hundred times higher than typical LoRA ranks. That is why LoRA cannot absorb substantial new knowledge — it structurally does not have the capacity, and no amount of training data changes that.
QLoRA adds a 4-bit quantised frozen base, double quantisation of the quantisation constants, and paged optimiser states, with the adapters in 16-bit. Gradients flow through the frozen 4-bit weights to the adapters. It is reported to match 16-bit fine-tuning quality while making a 65B model tunable on a single 48 GB card, which is what put fine-tuning within reach of ordinary teams.
A practical starting point: rank 16, alpha 32, targeting the attention projections plus the MLP up and down projections for behavioural tasks. Drop the rank if you see degradation on held-out general benchmarks — and measure that, because forgetting is invisible if you only evaluate on your target task.
Fine-tuning is for form, not facts
This is the rule that resolves most fine-tuning arguments. Use it to shape behaviour, style, structured output and refusal patterns — not to inject knowledge that changes weekly.
Where it genuinely wins: fixed output schemas where prompting is unreliable, and a few thousand examples collapse variance that no prompt can. Long, stable system prompts, where folding a 5,000-token preamble into weights removes it from every request. And above all, distilling a frontier model into a small open-weight model for one narrow task, which is the highest-return case in 2026 because it can cut inference cost by an order of magnitude at equal task quality.
The measured comparison worth quoting, from a medical-reasoning study: base model 61.0% → retrieval 89.0% → fine-tuning plus retrieval 90.1%. Note the shape — retrieval did essentially all the work and fine-tuning added about a point on top. Independent work agrees that fine-tuning alone does not match retrieval for less-common knowledge, and that the combination is best.
So the sequence is: prompt → retrieve → fine-tune → distil. And the highest-return fine-tune is a thin adapter paired with retrieval, never replacing it.
Post-training: the stack modularised
The single-recipe era — pretrain, then reinforcement learning from human feedback with a learned reward model — is over. Three methods now do three different jobs:
| Method | Mechanism | Critic? | Where it won |
|---|---|---|---|
| PPO / RLHF | Reward model trained on human preferences, then policy optimisation against it | Yes — roughly 2× memory | Legacy, and still the reference for general helpfulness and harmlessness |
| DPO (and variants) | A closed-form reparameterisation: optimise the policy directly on preference pairs, no reward model and no rollouts | No | General alignment and style. Simple, offline, stable |
| GRPO | Sample a group of 8 to 64 responses per prompt; the advantage is each response's reward normalised within the group. Eliminates the critic entirely | No | Reasoning. The method behind the prominent open reasoning models |
| RLVR | Reward comes from a verifier — a unit test, a proof checker, a maths verifier — not from a human or a learned model | No | The structural shift |
That last row is the important one. The field moved from human-labelled rewards to automated verification. For mathematics, code and structured reasoning you do not need a human judge — a test suite is a better reward: cheaper, more consistent, and much harder to hack than a learned reward model. Verifiable reward is what produced emergent self-checking behaviour in reasoning models trained with no human-labelled reasoning traces at all.
The refinements on top of GRPO are worth knowing by shape rather than by name: preventing entropy collapse by clipping asymmetrically, dynamic sampling to avoid wasting rollouts on prompts where every sample gets the same reward, token-level rather than sequence-level loss, and shaping the reward for overlong outputs. These address the practical instabilities of long-horizon reinforcement learning, and one such recipe reported reaching a strong mathematics score with half the training steps of the baseline.
Distillation
Two kinds, and they get conflated. Response-level distillation trains the student on the teacher's outputs — it works through an API and is what ‘distillation’ usually means in practice. Logit-level distillation matches the teacher's full output distribution, needs weight access, and is far more sample-efficient.
Reasoning distillation is the 2026 variant that matters: transferring reasoning traces from a large reinforcement-trained reasoner into a much smaller student. Reported outcomes span five to twenty times lower cost per token, with teams citing 75 to 80% inference cost reduction at comparable quality.
But there is an honest caveat in those numbers that is worth carrying: distilled small models have been reported retaining 70 to 85% of the teacher's broad-knowledge benchmark performance and only 50 to 70% of its hard-reasoning performance. Distillation transfers breadth much better than it transfers hard reasoning — so distil for throughput on well-defined tasks, and keep the frontier model for the genuinely difficult tail. Routing between them is usually a better architecture than choosing one.
Check yourself