Data Platform & Analytics · Chapter 10 of 10
Statistics and experimentation, without the degree
Named in 44% of data postings in the reader sample. The part of the field that separates a data scientist from an analyst.
Statistics language appears in nearly half of the data postings supplied by a reader, and it is the clearest boundary between the analytics and the science side of this field. You do not need a degree in it; you need to be able to avoid four specific mistakes and to say honestly when a result is not a result.
The four mistakes that matter
| Mistake | What it looks like | The correction |
|---|---|---|
| Peeking | Checking the test daily and stopping when it goes significant | Fix the sample size in advance, or use a sequential method designed for it |
| Multiple comparisons | Twenty metrics, one significant, reported as the finding | One primary metric decided beforehand; everything else is exploratory and labelled so |
| Underpowered tests | Flat result reported as ‘no effect’ | Compute the minimum detectable effect first — flat may mean the test could never have seen it |
| Simpson's paradox | The effect reverses when you split by segment | Check segments before reporting, especially when traffic mix changed |
Power, in the form you actually need it
Before any test, answer: what is the smallest effect worth detecting, and how long must this run to detect it? If the answer is longer than anyone will wait, the honest conclusion is that this experiment cannot settle the question and you should say so rather than run it and report whatever comes out. As a rough guide for rates near 50%, the margin of error is about 1/√n — ten points at n=100, five at n=400, two and a half at n=1,600.
Causation, when you cannot randomise
A large share of real questions cannot be A/B tested — you cannot randomise which customers get a price rise or which regions get a policy. The methods that exist for this are worth recognising by name even if you do not implement them: difference-in-differences (compare change over time against a control group), regression discontinuity (exploit a threshold), instrumental variables, and synthetic control (construct a comparator from a weighted blend of untreated units). Each rests on an assumption that can be stated and challenged, which is the point — a causal claim with a named assumption is arguable, and one without is just a correlation with confidence.
The sentence that marks a real practitioner
“That is inside the noise for a sample this size.”
Saying it costs you a finding and buys you every future finding, because the person who reports only real results is the person whose results get believed. The counterpart sentence is equally valuable: “this test could not have detected an effect that small, so flat does not mean no effect.”