In 2025 the research nonprofit METR ran a randomized controlled trial with 16 experienced developers working on their own mature codebases. They completed 246 tasks, each randomly assigned to allow or disallow AI tools. Tasks where AI was allowed took 19% longer on average. Before the study the developers predicted AI would speed them up by 24%, and afterwards they still estimated about a 20% improvement.
Read the caveats before the headline
- The sample was small — 16 developers, all very familiar with large, mature codebases of around a million lines.
- Acceptance of AI suggestions was low — under 44% in the study, so much effort went into reviewing and discarding output.
- The authors are explicit that the results do not imply AI systems aren't useful in other realistic settings. The setting, not the technology, drove much of the effect.
It would be a mistake to conclude that AI doesn't help. It would also be a mistake to ignore the part that generalizes.
The part that generalizes: perception and measurement diverge
The developers were fairly accurate at forecasting task time, yet badly wrong about the effect of the tool on their own speed. Feeling faster is not the same as being faster. Fluent, plausible output creates a sense of progress, while the time spent reviewing, correcting and re-prompting is easy to forget.
If the only evidence for a productivity gain is that people say they feel more productive, you have a hypothesis, not a result.
How to measure it in your own team
- Compare like with like — the same type of task with and without the tool, ideally assigned at random or at least across comparable periods.
- Measure the whole task — from start to accepted, finished output, including review and rework, not just time to first draft.
- Track quality alongside speed — defects, corrections and downstream complaints, since a faster task that creates cleanup isn't faster.
- Segment by task type and experience — gains tend to differ between routine and expert work, so averages mislead.
- Run it long enough — early novelty and learning effects distort the first weeks.
What this means for AI projects
Where AI replaces a clearly defined, repetitive step, the measurement is straightforward and the gains are visible in the process data. Where it assists open-ended expert work, the honest answer may be that it depends — on the person, the task and the context the tool lacks. Design your rollout to find out which is which, and report what you measure instead of what people expect.