Product & Delivery · Chapter 12 of 13
Experimentation: the pitfalls you are expected to know
Win rates traced to their origin, the split that is not close enough, why novelty is the wrong first answer, and the identity that explains the most famous metric failure.
This is the most technical chapter in the book and the cheapest place to out-perform, because most preparation material repeats a small set of figures without knowing where they came from. Everything below is from published primary sources — conference papers and talks by the people who built the largest experimentation platforms in the world.
Win rates, and what they are actually measuring
“About a third of experiments win” is roughly right and answers a narrower question than people think. That figure — a third positive, a third flat, a third negative — is measured against the metric the feature was designed to move.
Against a company's actual north-star metric, wins are vanishingly rare. For a major search engine, improving sessions per user happened in roughly one experiment in five thousand — and the platform's own architect notes that most apparent successes at that level are false positives that do not replicate.
The “only 10% of experiments at a large search company succeed” figure is a misquote, and its provenance is traceable. It comes from a book about experimentation, and the underlying phrase was about experiments ‘leading to business changes’ — a launch rate, not a win rate.
Other widely-quoted numbers — 75% of ideas have no impact, 80% of the time we are wrong about what customers want — trace to a vendor's own book and to an informal assertion in a primer respectively. Both may well be true. Neither is a measurement.
The interview move: when asked for a win rate, ask against which metric. That single clarification separates people who have run experiments from people who have read about them.
Sample ratio mismatch: the pitfall almost nobody mentions
You designed a 50/50 split. The actual counts came out 821,588 against 815,482 — 50.2%. That looks like noise and it is not: the p-value on that imbalance is about two in a million. Something is wrong with the assignment or the analysis population, and every metric in that experiment is now untrustworthy.
This is not exotic. Roughly 6% of experiments at Microsoft show a sample ratio mismatch, ranging from about 3% on one product to 13–14% on another, measured across more than twenty-five products at four companies.
The example that shows why an SRM is a finding, not just a fault
A news portal increased its front-page carousel from twelve cards to sixteen. The engagement metric said the change was worse. The team nearly killed it.
The split had come out 49.8% rather than 50.0% — small enough to ignore, and it was the whole story.
The root cause: engagement rose so much that the heaviest users were reclassified as bots and filtered out of the analysis. The feature's biggest fans were removed from the data because the feature worked.
After the filter was fixed, the treatment was much better.
Two lessons. First, an SRM is a signal to stop analysing and start investigating, not a rounding error. Second, root-causing one can take weeks — during which the experiment owners are simply stuck. That is a real cost and it is why the check belongs in the platform rather than in someone's discipline.
One more piece of context that reframes bot filtering entirely: over half the traffic on a major search engine is identified as non-human. The filter is not a detail.
Multiple comparisons: two numbers to memorise
Assume a feature does nothing at all. Run several variants and iterate:
- Five treatments takes a 2.5% false-positive rate to about 12%.
- Six iterations of a five-treatment experiment gives you better than a 50% chance of a statistically significant result from a feature with no effect whatsoever.
- The defences used in practice: a much stricter threshold when slicing and dicing — one platform pushes experimenters to a p-value below one in ten thousand for exploratory segment analysis — and a mandatory replication run with higher power before anything ships.
- Say the replication step out loud in an interview. ‘It is significant, so ship it’ after six iterations is the answer this section exists to prevent.
Peeking, and why the dashboard invalidates the statistics
A conventional p-value assumes the sample size was fixed in advance. If you watch the dashboard and stop when it goes green, you have chosen your stopping point using the data, and the guarantee is gone.
There are two real fixes and one non-fix. The non-fix is telling people not to look; they will look. The real fixes are always-valid inference — sequential methods designed so that continuous monitoring is legitimate — and pre-specified spending boundaries, where the early-day thresholds are brutally strict. On a seven-day experiment at one platform, the day-one cutoff is around five in a hundred million, relaxing to about 0.04 on the final day.
Novelty effects, and why they are the wrong first answer
Every preparation guide teaches novelty effect as the default explanation for a decaying treatment effect. The people who built these platforms say it is uncommon.
Their published position: novelty and primacy effects are rare in practice, and the cases they do observe are recommendation systems — where diversity itself causes a short-term effect — or treatments that consume a finite resource.
Before reaching for novelty, check: a sample ratio mismatch, whether the analysis population changed mid-experiment, and whether the decay is in the metric or only in the complaints.
That last distinction is worth a story. When a portal changed search results to open in a new tab, twenty percent of all user feedback on day one was about that change, mostly negative. Week two it was 4%; weeks three and four, 2%. Meanwhile the metric improvements held throughout. That is novelty in the complaints, not in the effect — and a team that reads the feedback volume as the signal kills a good change.
The practical rule they give: run for two weeks and look for decay, rather than assuming it.
Power, outliers and the checks that validate the platform itself
- Sample size scales with variance over effect size squared. Concretely: a commerce site detecting a 2% revenue change might need 100,000 users per arm. Detecting a 0.1% change — worth millions a year at scale — needs four hundred times that. Small effects at large companies are expensive to measure, and that is why variance reduction is a real discipline rather than a nicety.
- The variance-reduction levers, in the order one platform applies them: analyse only the users who were actually exposed; prefer a lower-variance metric (a boolean over a revenue figure; trimmed revenue over raw); use pre-experiment data to remove known variation; and reject randomisation seeds that fail a pre-period null test.
- Outliers, with a lovely worked example: on a bookseller with 100,000 users converting at 2% and spending $30 on average, a 2% lift is about $1,200 — and a single $2,500 order is enough to skew the result. The investigation found the cause was libraries, which order rarely and enormously. The fix is capping per-user values at the 99th percentile.
- Null tests validate the platform, not the feature. Run an experiment where both arms are identical; a correct system should report significance about 5% of the time. Run a thousand of them and the p-values should be uniformly distributed. One platform found they were not, for a set of metrics, and had to fix the platform — a defect no individual experiment would ever have revealed.
Twyman's Law, and the bias it is there to counteract
“Any figure that looks interesting or different is usually wrong.” It has been called the most important single law in the whole of data analysis, and it is aimed at a specific organisational failure.
The failure is asymmetric scrutiny: teams accept good results and investigate bad ones. Every surprising positive gets shipped; every surprising negative gets a root-cause investigation. Over time this systematically fills your launch history with measurement errors that happened to point the right way.
The fix is a rule rather than a virtue: investigate anything surprising, in either direction, with the same effort. Write it into how results are reviewed, because it will not survive as good intentions.
And a calibration figure worth carrying into any argument about observational data: in one review, roughly 90% of large randomised experiments produced results that stood up to replication, against about 20% of non-randomised studies. In another, of twelve beneficial-treatment claims from observational medical studies, none replicated in randomised trials and five were significant in the opposite direction.
Check yourself