Product & Delivery · Chapter 8 of 8
Scoping a feature whose core is non-deterministic
AI appears in 92% of product postings. This is the one skill that genuinely differs from ordinary product work.
Most of what makes AI product management hard is not the technology — it is that you cannot specify the behaviour, only the acceptable distribution of behaviours. That changes scoping, acceptance, launch and pricing simultaneously.
| Ordinary feature | AI feature | |
|---|---|---|
| Specification | This button does this | This works acceptably across this distribution of inputs |
| Acceptance | Matches the spec | Clears an evaluation threshold you had to define |
| Launch gate | Feature complete | Quality bar met — and you decided what quality means |
| Cost | Roughly fixed | Variable, with a tail that can invert margin |
| A defect | Reproducible | A confident wrong answer, often not reproducible |
| Roadmap risk | Can we build it | Can the model do it well enough — frequently unknown until you try |
The question that decides the feature
Not ‘how accurate is it’, which is unanswerable in isolation, but what is the cost of being wrong, and who finds out? A drafting assistant a human reviews before sending is useful at 85%. An auto-approval at 85% is a liability. Same number, opposite decision.
- Where do the errors fall? Uniform errors are survivable; errors concentrated on your highest-value cases or one customer segment are not, and the aggregate hides it.
- Can the user tell? A visibly wrong answer is far safer than a plausible one. How the output is presented is a product decision with safety consequences.
- What is the human baseline? If the process it replaces is right 80% of the time, 85% with review is an improvement — and almost nobody has measured the human baseline.
- What is the escape hatch? Every AI feature needs an obvious route to a human, and its usage rate is one of your best quality signals.
Define the launch gate as an eval threshold, in advance
“We ship when the golden set clears 80% with no regression on the safety cases” is a gate. “We ship when it feels good” is a gate that moves under commercial pressure, and it always moves in the same direction.
For the mechanics of building that evaluation, read Evaluation — how anyone knows it works; for what the model can and cannot do, What a model actually is.