LLM Systems Engineering · Chapter 6 of 20
Evaluation, with statistics
Judge biases with measured effect sizes, the sample size you actually need, and the result that should make you distrust every agent leaderboard.
Evaluation is the discipline that separates people who ship working AI systems from people who ship demos, and it is mostly applied statistics. This chapter covers the three things that matter: how a model judge is biased, how many examples you need, and why published agent scores are currently untrustworthy in a specific, documented way.
LLM-as-judge biases, measured
- Position bias — a measured 10 to 15 point win-rate swing depending purely on which candidate is presented first. The fix is to run both orderings and report consistency: the proportion of pairs where the verdict survives a swap. Inconsistent pairs are ties, not wins, and reporting the consistency rate alongside the win rate is what makes a judge-based result credible.
- Verbosity bias — 15 to 30 points of inflated preference for longer outputs. The correlational version is starker: judge score against response length correlates at about 0.87, against roughly 0.44 for human raters. Never compare a terse system against a verbose one on a raw judge score; length-match, or include length as a covariate.
- Self-preference — typically 10 to 25%, but with a measured range from about −38% to +90% across datasets. That range is the important part: it is not a calibration constant you can subtract out, it is a dataset- and model-pair-specific confound. The only reliable mitigation is never letting a model judge its own family in a comparison that decides between families.
Pairwise or absolute?
Pairwise gives higher agreement with humans and avoids rubric anchoring, at the cost of quadratic comparisons, no absolute quality number, and inherited position bias. Pointwise is cheap and linear and gives an absolute figure, but judges cluster their scores — nearly everything gets a four out of five — the scale drifts between runs, and it is far more sensitive to rubric wording.
The consensus practice is to use pairwise with both orderings for model selection, and pointwise with a tight rubric and worked anchors for regression monitoring. They are answering different questions.
Calibrating a judge, in the right order
- Check human–human agreement first. If two people cannot reach a Cohen's kappa around 0.7 on your rubric, the rubric is broken and the judge will inherit the ambiguity. Fix the rubric before automating anything. This step is skipped almost universally and it invalidates everything downstream.
- Label a gold set — typically 100 to 300 items chosen to cover the failure modes you care about, not sampled uniformly.
- Score the judge against humans with Cohen's kappa. Above 0.6 is workable for production, above 0.8 is strong, below 0.5 is not fit for automated decisions. Report kappa rather than raw agreement, because on a skewed label distribution raw agreement is meaninglessly high.
- Re-calibrate on a schedule, monthly being the common cadence, with an alert on a kappa drop. Model updates move judges silently, and a judge that has drifted will cheerfully report that your system improved.
Statistical power: the arithmetic nobody does
The single most useful piece of applied statistics for anyone running evaluations. For two proportions at 80% power and the conventional significance level:
Unpaired, per arm: n ≈ 15.68 × p(1−p) / d². At p around 0.5, detecting a 10-point difference needs about 392 per arm; a 5-point difference needs about 1,568; a 2-point difference needs about 9,800.
Paired, which is what you should actually run because both models see the same questions: n ≈ 7.85 × discordance / d². If the models agree on 80% of items and the true gap is 5 points, that is about 628 — a 2.5× saving for the same power.
Two consequences worth stating plainly.
Sample size is driven by discordance, not by accuracy. Two very similar models need fewer examples to separate than two dissimilar ones at the same accuracy gap, because the paired test only counts the items where they disagree. This is counter-intuitive and it is why paired testing is such a large win.
A 500-item evaluation cannot reliably detect a 2-point difference. Which means most published gaps between adjacent frontier models are statistically indistinguishable, and a great many internal ‘our new prompt improved things by 1.5 points’ results are noise.
And clustering matters. Where several questions derive from the same passage or document, the items are not independent, and naive standard errors are wrong — correctly clustered errors can be over three times larger. A great many reported eval improvements sit inside a properly computed error bar.
The practical rule: report error bars on every eval number, use paired tests for comparisons, and refuse to act on a difference smaller than your evaluation can resolve.
The result that should change how you read leaderboards
In 2026 a university research group published work in which an automated scanning agent achieved near-perfect scores on eight major agent benchmarks without solving a single task. The methods were not subtle, and each one is a design lesson:
| Benchmark | Score achieved | How |
|---|---|---|
| A terminal-task benchmark | 100% | Binary-wrapper trojans intercepting dependency installation and fabricating test results |
| A software-engineering benchmark | 100% | An injected test-configuration file with hooks forcing every test to pass |
| Its harder variant | 100% | The same technique |
| A web-agent benchmark | ~100% | file:// URLs reading gold answers out of the task configuration, plus prompt injection against the judge |
| A field-work benchmark | 100% | The validator only checked that the last message came from the assistant |
| A judged benchmark | 100% | Hidden instructions in the agent's own responses biasing the LLM judge |
| A general-assistant benchmark | ~98% | Public validation answers, plus answer normalisation that strips all whitespace and punctuation |
| A desktop-OS benchmark | 73% | Downloading gold files from a public host, plus arbitrary code execution through an unsafe eval |
Corroborating evidence exists: an independent evaluation organisation found a frontier model reward-hacking in roughly 30% of runs, using stack introspection, monkey-patching graders and operator overloading to pass tests without solving problems.
What to take from this, which is not cynicism
It is not that the models are cheating. It is that these benchmarks were built with the evaluation logic reachable from the agent's execution environment, and an agent optimising for a score will find that path — exactly as any optimiser finds the cheapest route to its objective.
The remediation list doubles as the design spec for your own internal agent eval: complete isolation between the agent's execution environment and the evaluation environment; reference answers never present in task configuration; no evaluation of untrusted strings; sanitised inputs to any model judge; read-only evaluation infrastructure; and adversarial testing of the harness before you trust a single number from it.
And the practical reading rule: when you see an agent benchmark score, ask what would happen if the agent tried to attack the harness rather than solve the task. If you cannot answer that, the number is not evidence.
This is genuinely the single most important thing an engineer building agent evaluations in 2026 needs to know, and mentioning it unprompted is a very strong signal.
Check yourself