AI from First Principles · Chapter 10 of 10
Temperature, top-p, and how the next word is chosen
The one place randomness enters a language model, what each knob actually does, and why the current answer has changed.
Chapter 1 covered what a model is. This is the step at the very end that turns its output into text — the only place randomness enters the system, and the reason the same prompt can give you two different answers.
Greedy decoding, and why nobody uses it alone
The simplest rule: always take the highest-probability token. It is deterministic in principle, and it produces noticeably poor text — it loops, repeats phrases, and collapses into the same safe constructions. The reason is worth understanding: at every step it takes the locally most likely token, and a sequence of locally optimal choices is not a good sentence. Human writing is not the most probable continuation at every position.
Temperature
Temperature reshapes the distribution before anything is picked from it. The model's raw scores are divided by the temperature before being turned into probabilities, which has a simple effect:
- Below 1 — the peaks get peakier. Likely tokens become more likely, unlikely ones nearly vanish. More focused, more repetitive.
- Exactly 1 — the model's own distribution, untouched.
- Above 1 — the curve flattens. Unlikely tokens get a real chance. More varied, and past about 1.5 it starts producing genuine nonsense.
- At 0 — mathematically undefined (you would divide by zero), so every implementation special-cases it to mean greedy. Which is why ‘temperature 0’ and ‘greedy’ are used interchangeably.
The intuition that helps: temperature does not make the model smarter or more creative. It changes how willing the sampler is to take the model's second and third guesses. ‘More creative’ and ‘more likely to be wrong’ are the same dial.
Top-k and top-p
Both truncate the distribution before sampling, and the difference between them is the whole reason one won.
Top-k keeps the k most likely tokens and discards the rest. Simple, and blind to how confident the model is. When the model is certain, k=50 drags in 49 bad options. When it is genuinely uncertain across hundreds of reasonable words, k=50 cuts off good ones.
Top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p. That makes it adaptive. At p=0.95, a confident prediction keeps two or three candidates and an open-ended one keeps forty. It widens exactly when the model is unsure and narrows when it is not, which is what you wanted top-k to do and it could not.
Stacking them: top-k and top-p filter first, temperature reshapes what survives. Setting all three at once is common and usually unnecessary — temperature plus top-p is the pair that carries almost all the benefit.
The repetition controls
- Frequency penalty — reduces a token's probability in proportion to how often it has already appeared.
- Presence penalty — a flat reduction once it has appeared at all, pushing toward new subject matter.
- Repetition penalty — a multiplicative version of the same idea.
- Use them sparingly. They are blunt: they will happily penalise a technical term, a product name or a variable that legitimately needs repeating, and the result reads like someone avoiding a word.
What to actually set
| Task | Temperature | Notes |
|---|---|---|
| Classification, extraction, structured output | 0 | You want the same input to give the same label. Variation here is pure downside. |
| Code generation | 0 to 0.2 | There is usually one right answer and creativity is not a virtue. |
| Factual question answering, retrieval | 0 to 0.3 | Higher settings do not make it more accurate — they make it more willing to guess. |
| Summarisation | 0.3 to 0.5 | A little variation reads better without inventing anything. |
| Conversation, drafting | 0.7 to 1.0 | The default range in most chat interfaces. |
| Brainstorming, deliberate variety | 1.0 and up | And generate several, rather than turning it up further. |
Raising temperature does not make a model more accurate, and it is a surprisingly common belief that it does.
If the model is wrong at temperature 0, it is wrong because the highest-probability answer is wrong. Turning up the temperature makes it sample further down a distribution whose top was already incorrect.
Wrong answers are a knowledge or context problem — see retrieval. Sampling parameters cannot fix them.
Temperature 0 is not reproducible
The practical consequence: do not write a test that asserts an exact output string from a model, even at temperature 0. It will pass for weeks and then fail for reasons that have nothing to do with your code. Assert on structure, on a schema, on a property that must hold, or use a model-graded check with a tolerance. Anything else is a flaky test you will eventually delete in frustration.
And the part that has changed — check before you tune
On current reasoning models, these parameters are often simply gone
As models moved to built-in reasoning, several providers removed sampling controls altogether rather than deprecating them.
On Anthropic's current models — Opus 5, Sonnet 5, Opus 4.8 and 4.7, Fable 5 — sending temperature, top_p or top_k returns a 400 error. Not ignored. Rejected.
What replaces them is a different kind of control: an effort setting (how hard to think, from low to max) and adaptive thinking. You are no longer tuning how the next token is picked; you are tuning how much reasoning happens before the answer.
So this chapter is doing two jobs. The mechanics are permanently worth understanding — they explain why models vary, why greedy text is poor, and why turning up temperature does not buy accuracy, and they still apply to every open-weight model you self-host. But if you are writing against a current frontier API, check whether the parameter exists before you spend an afternoon tuning it.
On a model you run yourself, all of these controls are fully available and you set them per request — which is one more concrete difference between using an API and hosting your own.
Check yourself