Lakshya

Handbook · Chapter 1

Measuring what resists measurement

Named in 59% of AI postings and 42% of platform postings. Almost nobody studies it deliberately.

Four disciplines on this site are assessed on the same skill under four different names. An AI engineer is asked about evals. An SRE is asked about SLOs. A DevEx engineer is asked how they'd measure productivity without gaming it. A research engineer is asked to convince me that two-point improvement is real. These are one problem: deciding whether a system that behaves differently each time you look at it got better or worse, with enough rigour that a team can act on the answer.

THE STAGE 1 · Fix a reference2 · Run the system3 · Compare4 · Decide reference is revised, and the loop runs again SAME LOOP, THREE VOCABULARIES AI — evals Platform — SRE DevEx Research golden setinference runscore vs baselineship or blockSLOproduction trafficSLI vs objectivepage or holdbaseline build timeCI rundelta vs baselineadopt or revertheld-out settraining runablation vs controlkeep or discard
The loop is identical; only the nouns change. Read across a row for one discipline's vocabulary, or down a column to see four fields describing the same stage. Candidates who learn the loop once can answer this question in any of the four interviews; candidates who memorise one field's vocabulary can answer it in exactly one.

Stage 1

Fix a reference — and the reference is the hard part

Every failure of measurement traces back to a bad reference, not a bad metric. The reference is the thing you compare against: a golden set, an SLO, a baseline build time, a held-out split.

  • Golden sets go stale silently. Yours was built from last quarter's traffic; the product shipped a new surface; your evals still say everything is fine. Date your reference set and re-sample it on a schedule.
  • SLOs get chosen for achievability, not for user pain. If your SLO is 99.9% because that is what you currently hit, it measures nothing. The test: would a user notice the difference between meeting and missing it?
  • Fifty well-chosen examples beat five thousand scraped ones. Coverage of the failure modes you care about matters more than volume, and a set you can read in an afternoon is one you will actually maintain.
  • The reference must predate the change. Choosing it afterwards is how you get a result that confirms whatever you did.

Stage 2

Run — non-determinism is the whole difficulty

A deterministic system gives you one number. An LLM gives you a distribution, and so does production traffic, and so does a CI runner on a noisy neighbour. Two consequences most candidates miss:

  • One run is not a measurement. Temperature above zero means you need several samples per item to say anything. At temperature zero you have removed variance you will still meet in production.
  • Your sample must resemble what you are deciding about. Evaluating on easy cases and shipping to hard ones is the most common way a green eval precedes an outage.

Stage 3

Compare — this is where a control belongs

The comparison step is where almost every weak answer collapses, because the candidate compares the new thing to nothing at all. This matters enough that it has its own chapter: the control →

The second failure here is statistical. On a 200-item eval set, a three-point difference is usually noise. You do not need to run a formal test in an interview, but you do need to say the words — “that is inside the noise for a set this size” — because saying them is the signal being screened for.

LLM-as-judge, and the four biases you must be able to name

Using a model to grade a model is standard now, and interviewers expect you to know how it misleads: position bias (the first option wins more often — mitigate by swapping order and averaging), verbosity bias (longer answers score higher regardless of quality), self-preference (a model rates its own family's output above others), and score compression (everything lands between 7 and 9 on a 10-point scale, so the scale carries almost no information).

The mitigations are worth as much as the biases: prefer pairwise comparison over absolute scoring, calibrate the judge against human labels on a subset, and report agreement rate with humans rather than the raw score.

Stage 4

Decide — the policy has to exist before you need it

A measurement that does not change a decision is theatre. The decision rule must be agreed before the number arrives, because afterwards it gets negotiated by whoever is under the most pressure. This is exactly what an error-budget policy is, and exactly what an eval-gated release is — the same idea in two vocabularies.

The drill

Two evenings, and it transfers to four interviews

  • Evening one. Take something you have already built. Write fifty reference cases by hand, covering the failure modes you have actually seen. Score your current version. That number is your baseline.
  • Evening two. Change one thing — a prompt, a threshold, a retrieval parameter. Re-run. Write down the delta, whether it is inside the noise, and what decision you would make. Then break the system three ways and check whether your harness catches it.
  • Then write one paragraph describing the whole thing, ending with a number and a decision. That paragraph is your answer in every interview on this site.
  • “How do you evaluate something with no single correct answer?”Rubrics, pairwise preference, task-completion proxies — and honesty about what each one misses.
  • “Your judge agrees with humans 85% of the time. Ship on it?”Depends entirely where the 15% falls. Strong answers ask about the error distribution before answering.
  • “How would you measure developer productivity without gaming it?”The DevEx phrasing of the same question. Pair a quantitative proxy with qualitative survey data, and say out loud that every proxy is gameable.
  • “You improved the score and users complained more.”The reference measured the wrong thing. Being able to say that plainly is the test.
← All chaptersThe control →