Product Design & UX · Chapter 3 of 5
Research at the sample sizes you actually get
Expected in 83% of postings, and almost never with the sample size the textbooks assume.
Research in most product contexts means five to eight conversations, not a statistically powered study. That is not a compromise — qualitative research at that size reliably surfaces usability problems, and treating it as underpowered quantitative research is the mistake.
- Ask about the last time, not the general case. ‘Walk me through the last time you did this’ produces facts; ‘would you use a feature that…’ produces politeness.
- Watch rather than ask where you can. What people do and what they report differ, consistently and predictably.
- Separate usability from desirability. Five users will tell you whether they can operate it. They cannot tell you whether the market wants it.
- Recruit for the segment that matters, not for who is available. Five of the wrong users is worse than none, because it produces confident wrong conclusions.
- Know when you need numbers instead. Preference, magnitude and prevalence questions need instrumentation or a survey, not interviews.
The methods, and what each can actually tell you
| Method | Answers | Cannot answer |
|---|---|---|
| Usability testing (5–8 people) | Can people use this? The single highest-value method, and the cheapest | Whether they want it. People will complete a task perfectly and never return |
| Interviews | What people currently do, and what is hard about it | What they would do. Stated future behaviour is close to worthless — ask about last time, not next time |
| Surveys | How widespread something you already understand is | Anything you have not already characterised. Surveys measure the answers you thought to offer |
| Analytics | What happened, at scale | Why. Analytics tells you where people drop off, never what they were trying to do |
| A/B testing | Which of two options performs better on a chosen metric | Whether either is any good, or what a third option would have done |
The pairing that resolves most research arguments: analytics tells you where, qualitative work tells you why, and neither substitutes for the other. A team with only analytics knows exactly which step loses people and has no idea what to change; a team with only interviews has a rich account of a problem that may affect eleven users.
The most useful single rule in research: ask about the past, never the future.
‘Would you use a feature that did X?’ produces a polite yes with no predictive value whatsoever. People are not lying — they are genuinely bad at forecasting their own behaviour, and they are being agreeable to someone who clearly wants a yes.
‘Tell me about the last time you needed to do X. Walk me through what you actually did.’ produces evidence, because it is a memory rather than a prediction — and the workarounds people describe are usually the most valuable thing in the whole study.
The corollary for feature requests: a user asking for a specific feature is reporting a solution they have designed. Ask what they were doing when they wanted it. The problem underneath usually admits a better answer, and occasionally reveals that the feature would not have helped.
And on sample size: five to eight participants finds most usability problems, which is why usability testing is so cheap relative to its value. Five participants tells you nothing about prevalence — that is what the analytics are for.
Check yourself