Handbook · Chapter 2
The control
One habit separates strong candidates from plausible ones more reliably than any technical depth.
A number without a baseline is not a result. This is the single most portable idea in this handbook, it takes an afternoon to internalise, and it is the thing interviewers at every archetype are quietly listening for.
Why it works as an interview signal
Interviewers cannot verify your past work. What they can do is watch how you talk about numbers. A candidate who volunteers the baseline unprompted is demonstrating, in one sentence, that they have shipped something and been wrong about it before. That is very hard to fake and it is why the habit carries so much weight.
Match the control to the treatment
The mistake that survives review
A bad control is worse than none, because it manufactures confidence. The rule: the control must differ from the treatment in exactly one respect — the thing you are claiming.
| Claim | A weak control | The control that actually tests it |
|---|---|---|
| “Our agent resolves tickets well” | No comparison at all | The keyword rules it replaced, on the same tickets |
| “This retrieval strategy is better” | A different eval set | The old strategy, same queries, same judge |
| “The new build cache helped” | Build times last month | Same commits, cache on versus off |
| “Our SLO improved” | Last quarter's number | Same traffic mix — a quiet quarter is not an improvement |
| “This feature drove retention” | Users who adopted it | Matched users who did not — adopters were already engaged |
The subtle one: when the entry condition is itself a selector
If your treatment only triggers under some condition — a dip, an error, a slow request — then your control must be drawn from the same condition. Comparing “requests we retried” against “all requests” measures the difficulty of the requests, not the value of retrying.
This is the failure mode that survives peer review most often, because both numbers are real and the comparison looks reasonable until you ask what selected each group.
Power, in one paragraph
You do not need statistics in an interview. You need to know roughly when a difference is too small to believe. A workable rule of thumb for pass/fail rates near 50%: the margin of error is close to 1/√n. At n=100 that is about ten points; at n=400, five points; at n=1,600, two and a half. So on a 200-case eval set, a three-point improvement is noise, and saying so out loud is worth more than the improvement would have been.
Where each archetype hits this
| Archetype | How the question arrives |
|---|---|
| Forward Deployed / Applied AI | “How would you know this deployment succeeded?” |
| Agent Engineer | “You changed a prompt and the demo looks better. Do you ship it?” |
| ML Engineer | “Offline AUC improved, the A/B was flat.” |
| Research | “Your eval improved by two points. Convince me it is real.” |
| Evaluation | The entire job. |
| SRE | “Reliability improved — or was last quarter just quiet?” |
| DevEx | “Justify your team's headcount.” |