Product & Delivery · Chapter 4 of 13
Measuring a product
Metrics, OKRs, and the specific ways each is gamed. Named in only 14% of postings, which is itself a finding.
OKRs appear in 14% of the product postings behind this site and experimentation in 16%. The measurement culture the product literature assumes is a minority practice, which means bringing it credibly is a differentiator — provided you also know how it fails.
Choosing a metric that is hard to game
- Prefer a metric the user has to do something real to move. Sign-ups are cheap; second-week usage is not.
- Always pair it with a counter-metric. Speed paired with quality, engagement paired with complaints, conversion paired with refunds. A single metric with no counterweight will be bought at the expense of something you did not measure.
- Beware ratios. They improve when the denominator falls, so a conversion rate can rise because you lost the hard traffic.
- Name the lagging and the leading indicator separately. Revenue is lagging and unactionable within a quarter; the leading indicator is what you can actually work on.
Where OKRs go wrong
Two failure modes, and they are opposite. Sandbagging — targets set at what will certainly happen, so the exercise measures nothing. And OKRs as a task list — key results that are deliverables (‘ship the new onboarding’) rather than outcomes (‘raise activation from 34% to 45%’), which reintroduces output as the goal under a new name. The second is far more common and much harder to argue against, because the deliverables are real work.
The comparison question, again
Whatever number you report, the question that should follow is against what? Against last quarter, against the segment that did not get it, against what would have happened anyway. This is the same discipline as the control chapter in the interview spine, and it applies identically here.
A product manager who volunteers the counterfactual unprompted is immediately credible, because almost nobody does.
The most famous metric failure, and the identity that resolves it
A ranking bug degraded search quality at a major engine. Distinct queries rose over 10% and revenue rose over 30%. Nothing was mis-measured; bad results simply make people search more, and more searches means more ad impressions.
The resolution is an identity worth being able to write on a whiteboard:
Queries/Month = Queries/Session × Sessions/User × Users/Month
In a controlled experiment the third term is fixed by the design — you are not changing how many people exist. And we want the first term to fall, because a user finding what they needed in one search is the product working. That leaves the middle term. Sessions per user is the criterion, and it is the only one of the three that is honest.
Choosing an overall criterion
- Optimise for lifetime value, not this quarter's revenue. Every metric failure in this chapter is a short-horizon proxy behaving well while the long-horizon quantity behaves badly.
- Use leading indicators. A lagging metric confirms what already happened; you cannot steer with it.
- Keep the number of key metrics very small. The framing used at one platform is the ‘Otis Redding problem’ — if ten people tell you what to do, you do nothing. A weighted sum of a few things is legitimate; a scorecard of thirty is a way of having no criterion at all.
- Report many other metrics as diagnostics only, whose job is to explain why the criterion moved. Being explicit that diagnostics are not targets is what stops them being optimised.
- Separate statistical from practical significance, operationally. One platform explicitly does not alert on a two-millisecond page-load regression even at a p-value below one in ten billion. Encoding that distinction in the alerting rules — rather than leaving it to judgement — is a strong senior signal.
HEART, and the two things everyone gets wrong about it
HEART is not the framework. Goals-Signals-Metrics is. The original research presents the five categories — happiness, engagement, adoption, retention, task success — as a prompt for articulating goals, and the goals-to-signals-to-metrics process as the actual method. Its own line: no matter how user-centred a metric is, it is unlikely to be useful unless it explicitly relates to a goal.
You are not supposed to fill in all five rows. The paper never says to. It notes that adoption and retention are especially useful for new products and redesigns, and that for established products they tend to stabilise. Teams that dutifully populate every category produce a dashboard rather than a decision.
Two design rules from it worth quoting: choose signals that are sensitive and specific — they should move when the experience is better or worse and not for unrelated reasons. And raw counts grow with your user base and must be normalised; ratios or per-user averages are usually what you want.
Its own worked example of why a top-line number misleads: during the 2008 market crash, a finance product saw page views and weekly actives surge — and the aggregate could not distinguish new crisis-driven visitors from existing users anxiously checking portfolios. Only splitting adoption from retention separated them.
A vocabulary note that is worth knowing because it makes three literatures readable as one: ‘north star metric’, ‘overall evaluation criterion’ and ‘the one metric that matters’ are the same construct under three sets of words, from the growth, experimentation and lean-analytics traditions respectively.
Goodhart, correctly
Goodhart did not say the thing everyone quotes
The famous line — “when a measure becomes a target, it ceases to be a good measure” — is not Goodhart's. It traces to a 1996 rendering by Hoskin, popularised by the anthropologist Marilyn Strathern in 1997.
Goodhart's actual 1975 formulation: any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
And Campbell's Law, from 1969, has priority over Goodhart entirely.
Why bother with the provenance? Because Goodhart's own wording is more useful than the popular one. It says the regularity collapses — the historical relationship between the proxy and the thing you care about stops holding. That tells you what to watch: not whether the metric is being gamed, but whether it still correlates with the outcome it was chosen for. That correlation is checkable, and almost nobody checks it after the metric is adopted.
The cleanest technical example is the one at the top of this chapter. The second cleanest: a team told to reduce a ‘no results’ rate succeeded — by returning irrelevant results instead of admitting there were none. Sometimes showing ‘no results’ is the right answer, and no aggregate will tell you that.
The long-horizon result almost no candidate knows
A search company published work on ad blindness: users change their inherent propensity to click on ads based on the quality of ads they have previously seen. It is a learned effect and it is invisible in a one- or two-week test.
Acting on it, they cut the ad load on mobile search by half. That was substantially revenue-negative in the short run and, once the learning effect was accounted for, long-run revenue neutral — with a large user-experience gain.
Two refinements worth carrying. First, a follow-up study holding ad load constant while varying quality found that quality, not load, is the main driver of the learning — load matters because a system shows its best ads first, so more load means worse average quality. Second, the measurement needs a design built for it: a common one runs a parallel cohort that receives the treatment on only one day, so it never accumulates the learned effect and can act as a comparison.
And note the cadence difference this creates. Google runs ninety-day experiments; Bing optimises for agility with one- to two-week cycles. Neither is wrong — they are measuring different classes of effect, and knowing which class your question belongs to is the actual skill.
Check yourself