Product & Delivery · Chapter 8 of 13
Scoping a feature whose core is non-deterministic
AI appears in 92% of product postings. This is the one skill that genuinely differs from ordinary product work.
Most of what makes AI product management hard is not the technology — it is that you cannot specify the behaviour, only the acceptable distribution of behaviours. That changes scoping, acceptance, launch and pricing simultaneously.
| Ordinary feature | AI feature | |
|---|---|---|
| Specification | This button does this | This works acceptably across this distribution of inputs |
| Acceptance | Matches the spec | Clears an evaluation threshold you had to define |
| Launch gate | Feature complete | Quality bar met — and you decided what quality means |
| Cost | Roughly fixed | Variable, with a tail that can invert margin |
| A defect | Reproducible | A confident wrong answer, often not reproducible |
| Roadmap risk | Can we build it | Can the model do it well enough — frequently unknown until you try |
The question that decides the feature
Not ‘how accurate is it’, which is unanswerable in isolation, but what is the cost of being wrong, and who finds out? A drafting assistant a human reviews before sending is useful at 85%. An auto-approval at 85% is a liability. Same number, opposite decision.
- Where do the errors fall? Uniform errors are survivable; errors concentrated on your highest-value cases or one customer segment are not, and the aggregate hides it.
- Can the user tell? A visibly wrong answer is far safer than a plausible one. How the output is presented is a product decision with safety consequences.
- What is the human baseline? If the process it replaces is right 80% of the time, 85% with review is an improvement — and almost nobody has measured the human baseline.
- What is the escape hatch? Every AI feature needs an obvious route to a human, and its usage rate is one of your best quality signals.
Define the launch gate as an eval threshold, in advance
“We ship when the golden set clears 80% with no regression on the safety cases” is a gate. “We ship when it feels good” is a gate that moves under commercial pressure, and it always moves in the same direction.
For the mechanics of building that evaluation, read Evaluation — how anyone knows it works; for what the model can and cannot do, What a model actually is.
Has AI made the team faster? The honest state of the evidence
This is now a standard interview question for product and engineering leaders, and it is a trap for anyone who answers from vibes in either direction. The published evidence is genuinely mixed, and the most instructive thing about it is a research team publicly revising its own headline result.
The study everyone quotes, and the update almost nobody has read
July 2025: a randomised trial with sixteen experienced open-source developers on 246 real issues, in repositories averaging over a million lines, found they were 19% slower with AI assistance. The finding that made it famous: they expected a 24% speed-up, and even after experiencing the slowdown still believed AI had sped them up by 20%.
February 2026: the same team published a design change. The follow-up data gives an unreliable signal, because 30–50% of developers were declining to submit tasks they did not want to do without AI — a selection effect that biases the measured task mix.
Their revised estimates: roughly an 18% speed-up for returning developers, with a confidence interval running from −38% to +9% — which crosses zero — and about 4% for new recruits. Their own conclusion is that developers are likely more sped up now than their 2025 estimate suggested.
So anyone citing “AI makes developers 19% slower” today is quoting a number the authors have publicly qualified. Knowing that is a differentiator, and using it well is better still.
Why this is the best available answer to “how would you measure the impact of X?” It is a live, well-documented case containing self-report versus measurement, a selection effect that only appeared in the second round, a confidence interval that crosses zero, and a team revising its own headline in public. That beats any hypothetical.
The measurement gap deserves its own note, because it recurs in every productivity discussion. The same group found people overestimated AI's effect on their own task time by about 40 percentage points, and observed that survey-based estimates of productivity impact tend to be larger than those from field experiments. If your evidence is a survey, say so.
What the delivery research says, which is a different finding
- The large annual delivery-research programme found that AI adoption significantly increases individual productivity, flow and job satisfaction — and negatively affects delivery throughput and stability.
- Those two findings are not in conflict; they are the whole insight. Individuals are faster and the system is not, because removing the coding bottleneck saturates everything downstream of it — review, testing, release.
- The following year's framing is that AI is an amplifier: it magnifies an organisation's existing strengths and weaknesses, and the returns come from the organisational system rather than the tools.
- Which changes the question from ‘should we adopt AI’ to ‘what is our next bottleneck, and have we widened it first?’
- The same programme publishes work on the productivity dip that precedes the gain during a rollout — worth planning for rather than reading as failure.
And a finding from the same research that has nothing to do with AI and outranks most of it: unstable organisational priorities cause meaningful drops in productivity and substantial increases in burnout — and the damage is highly resistant to mitigation, persisting even with strong leadership and good documentation. That is the strongest available empirical argument for outcome-based roadmaps, and it is a much better argument than ‘dates are bad’: the roadmap is a priority-stability instrument.
Where the role is going
The most credible practitioner argument for what changes: because the cost of delivery has dropped so far, building the thing is no longer the bottleneck — the bottleneck is discovering something worth building. The useful framing that goes with it is the distinction between building to learn (prototypes, many per week, disposable) and building to earn (production software).
On “the PM layer got cut”: I could find no data supporting it. No source with a stated methodology measures product-management headcount, postings or PM-to-engineer ratios across this period. The honest move is to decline to quote a number.
The defensible reframe: if product management means being responsible for the value and viability of what gets built, there is no alternative to it — someone is doing it one way or another. What gets cut are people titled product manager who were in fact running a backlog or a project.
Why that reframe is useful rather than defensive: it tells you exactly what to demonstrate in an interview. Evidence that you owned value and viability decisions — what you decided not to build, which assumption you tested, what the business case was — rather than evidence that you ran a well-groomed backlog.
The blunt version from the same source: if you have Agile-trained product owners sitting where you need product managers, what you actually have are backlog administrators, and leaders eventually work that out.
Check yourself