AI from First Principles · Chapter 8 of 8
Fine-tuning, in practice
When it is actually the right answer — which is less often than teams assume, and for different reasons than they give.
Chapter 2 established that fine-tuning changes the weights and therefore creates a deletion problem. This chapter is about the prior question: whether it is the right tool at all. Teams reach for it because it sounds like the serious option, and in most cases it is the expensive way to get a worse result.
The decision, stated simply
| If the requirement is… | The right tool is… | Why |
|---|---|---|
| Know our current facts | Retrieval | Facts change; weights do not. Fine-tuned facts go stale and cannot be corrected without retraining |
| Follow our format or house style | Fine-tuning, or a good prompt | Behaviour and shape are exactly what fine-tuning teaches well |
| Handle our jargon and label taxonomy | Fine-tuning | A narrow classification or extraction task with consistent labels is the classic good fit |
| Be cheaper or faster | Fine-tuning a small model | A tuned small model can match a large one on a narrow task at a fraction of the cost |
| Be more accurate in general | Usually neither | This requirement is not specific enough to act on, and is a sign the evaluation does not exist yet |
What good fine-tuning actually needs
- A narrow, well-defined task. Fine-tuning excels at one thing done consistently and degrades when asked to cover a broad surface.
- Consistent labels. Several hundred to a few thousand examples that agree with each other. Inconsistent labelling is the most common cause of a tuned model that is worse than the base — the model faithfully learns your disagreement.
- An evaluation that predates the tuning. Otherwise you cannot tell whether it helped, and the temptation to declare victory is considerable given the cost.
- A plan for the next model. The base model will be superseded within a year, and your tuning does not transfer. Budget for redoing it, or accept being pinned to an ageing base.
The techniques, briefly
Full fine-tuning updates every weight and needs serious infrastructure. LoRA and other parameter-efficient methods train a small number of additional parameters and leave the base frozen — far cheaper, usually almost as good for narrow tasks, and much easier to govern because the adapter is a small artefact you can version, review and remove. For most enterprise cases, if fine-tuning is right at all, LoRA is the form of it that is right.
The governance consequence of choosing the adapter
A LoRA adapter is a separate, small, versioned file layered on a frozen base. That makes it reviewable, attributable to a training set, and removable — which addresses much of the deletion problem in chapter 2, because removing the adapter removes the learned behaviour even though the training data cannot be extracted from it.
It is worth saying this in a governance conversation, because it turns ‘we cannot un-learn it’ into ‘we can withdraw the adapter and retrain it without that data’, which is a materially better position.
The order to try things in
- Prompt it properly first, with examples in the prompt. Costs an afternoon.
- Add retrieval if the gap is knowledge. Costs days.
- Fine-tune a small model if the gap is behaviour, format or cost. Costs weeks.
- Full fine-tuning of a large model: almost never, outside a lab.
- And measure at every step against the same evaluation, or you will not know which of them helped.