Lakshya

Microsoft 365 & Copilot in the Enterprise · Chapter 11 of 12

Does it actually work? The evidence, both ways

Time savings are real and replicate. Whether they become productivity is genuinely unproven — and the best study says no.

6 min read1 diagramAll 12 chapters

This is the chapter that should decide your business case, and it is the one nobody writes honestly because the vendor material and the backlash are both louder than the studies. The evidence is more interesting than either.

What replicates: task-level time savings

Microsoft's own randomised controlled trial — genuinely randomised, 6,000-plus workers across 56 firms in 11 industries, published April 2025 — found email reading down 7% across everyone assigned a licence, or 18% among regular users, about 31 minutes a week; document completion 12% faster; and no significant reduction in meeting time.

Four independent UK government evaluations found the same order of magnitude, converging around 12–30 minutes a day on narrow tasks: email, drafting, summarising, finding things. That much is real, replicated, and not seriously in dispute.

What does not replicate: that the saved time becomes output

FOUR UK GOVERNMENT EVALUATIONS — ORDERED BY HOW WELL THEY WERE DESIGNED GDS20,000 users, self-report, NO CONTROL GROUP~26 min/day saved · 82% would not give it up — but nothing to compare againstHMRC3,000 randomised, self-report discounted ~20% for bias~60 min/week · and 46% of NON-USERS cited security as their reasonDWPTreatment 1,716 + a real comparison group of 2,53519 min/day · but flags that AI-keen volunteers self-selected inDBTReal control group + 11 DIRECTLY OBSERVED task sessionsSee below — the only study that watched people work THE BEST-DESIGNED STUDY IS ALSO THE MOST NEGATIVE. That ordering is the finding. DBT: “did not find evidence that time savings have led to improved productivity.” Observed users produced LOWER-quality Excel andPowerPoint output. 43% of users failed to spot a hallucination when one was present.
Read this top to bottom. The studies get more rigorous as you go down, and the findings get worse. That correlation is the single most important thing on this page.

The two findings that a business case has to survive

Microsoft's own companion study — 66 firms, 7,137 workers — found the treated group saved about two hours a week on email and showed no shifts in quantity or composition of tasks. An organisational-level null sitting directly beside Microsoft's own individual-level time savings.

The UK Department for Business and Trade is the only evaluation with both a real control group and directly observed output rather than self-report. It did not find evidence that time savings have led to improved productivity, and in observed sessions Copilot users produced lower-quality Excel and PowerPoint output than non-users. 43% failed to identify a hallucination when one was present.

Neither is a fringe critic. One is Microsoft Research; the other is the best-designed study in the field.

Why self-reported time savings and measured productivity can diverge so far is worth thinking through, because it generalises well beyond this product. Saved time only becomes output if it is reallocated to something valuable — and an hour recovered in fifteen scattered four-minute fragments across a day is not an hour of work. The Excel finding suggests a second mechanism: assistance that is fast and usually right teaches people to check less, and a lower checking rate on a tool that is occasionally wrong is a quality loss that the person doing it cannot feel.

Adoption, in Microsoft's own numbers

  • Paid seats went 15 million → 20 million → 30 million-plus across three quarters to July 2026, and organisations deploying to a majority of their workforce rose about 75% quarter on quarter. Real growth.
  • Against a Microsoft 365 commercial base of roughly 450–680 million seats, that is under 5% penetration, three years after launch.
  • Copilot revenue has never been broken out separately. It is folded into Microsoft 365 commercial cloud. For a product this heavily promoted, that is itself information.
  • Gartner reports that only 5% of organisations that completed a pilot moved to scaled deployment.

A statistic to stop repeating

“MIT found 95% of AI projects fail” does not apply to Copilot, and checking the primary text shows why.

The MIT NANDA report is about custom-built internal generative AI. In the same report, deployments done with a vendor succeed at roughly double the rate of internal builds, and general-purpose tools show pilot rates above 80%.

Microsoft 365 Copilot is named exactly once, in a neutral sentence, never as a failure example.

Widely-read coverage of it also misreports its own sample size. If you are going to cite the 95%, cite it for what it measured — and notice that it mildly supports buying a vendor tool over building your own.

Where that leaves a recommendation

Stated plainly, and you can quote this in a steering meeting: the task-level time savings are real and replicate across Microsoft's own trials and independent government evaluations. The load-bearing claim — that saved time compounds into more or better output — is not established, and the single best-designed study found the opposite on quality. Commercial traction lags purchasing substantially.

That is not ‘do not buy it’. It is: buy it for a named task where you can measure the output, not for a general productivity uplift you have no way to detect; expect the oversharing work in chapter 8 to be the real cost; and if the business case rests on recovered hours becoming revenue, say out loud who is going to check whether that happened, and when.

Check yourself

What replicates across studies?Task-level time savings of roughly 12–30 minutes a day on email, drafting, summarising and retrieval — in Microsoft's own RCT and four independent UK evaluations.
What did Microsoft's own companion study find?Two hours a week saved on email but no shifts in the quantity or composition of tasks — an organisational-level null next to its own individual-level savings.
Why is the DBT study the important one?It is the only evaluation with both a real control group and directly observed output. It found no productivity improvement and lower-quality Excel and PowerPoint work.
What is the hallucination finding?43% of users failed to identify a hallucination when one was present.
What is the adoption picture?30 million-plus paid seats and real growth, but under 5% of the commercial base three years in, and only 5% of completed pilots scale.
Why doesn't the MIT 95% statistic apply?It measured custom internal AI builds. In the same report vendor deployments succeed at about double the rate, and Copilot is named once, neutrally.
What is the defensible recommendation?Buy it for a named, measurable task rather than a general uplift; budget for the oversharing work as the real cost; and name who will check whether recovered hours became output.
← Extending with code, and choosing your platformThe rollout, start to end →