The AI Spotlight
AI Coding

Developers Swore AI Made Them 20% Faster. The Stopwatch Said 19% Slower.

A randomized trial with experienced open-source developers found AI tools slowed them down, even as they felt sped up. Here is what the study really shows and how to avoid the trap.

A
The AI Spotlight DeskNov 6, 2026 · 4 min read
Advertisement
AdSense slot · articleTopplaceholder until ca-pub ID is set

If you have used an AI coding assistant, you know the feeling. You describe what you want, code pours onto the screen, and it feels like you just skipped an hour of work. So it was a shock to many developers when a careful study found the opposite.

In July 2025, the research nonprofit METR published the results of a randomized controlled trial on how early-2025 AI tools affect experienced developers working on their own projects. The headline number: when developers were allowed to use AI, they took 19% longer to finish their tasks.

The twist is what they believed. Before starting, the developers predicted AI would make them 24% faster. After finishing, they still estimated it had made them about 20% faster. The stopwatch disagreed.

How the study worked

This was not a lab puzzle or a toy benchmark. According to METR’s write-up and paper:

Advertisement
AdSense slot · articleMidplaceholder until ca-pub ID is set
  • 16 experienced developers took part, each a long-time contributor to a large, popular open-source repository.
  • They worked on 246 real issues from their own projects, the kind of bug fixes, features and refactors they would do anyway.
  • Each issue was randomly assigned to either allow AI tools or forbid them.
  • When AI was allowed, developers mostly used Cursor with Claude 3.5 and 3.7 Sonnet, which were frontier models at the time.
  • Developers recorded their screens and self-reported how long each task took.

Randomizing at the level of individual tasks is what makes the result hard to dismiss. The same people, on the same codebases, did similar work with and without AI.

Why would AI slow experts down?

METR investigated a range of possible factors. Several themes stand out, and they will sound familiar to anyone who codes with AI daily.

Experts know their codebase better than the model does. These developers had years of context on huge, mature projects. The AI did not. Explaining the context, or fixing code that ignored it, ate up the time saved by generation.

Reviewing is real work. Developers accepted fewer than half of the AI’s suggestions, and many accepted changes still needed cleanup. Reading, testing and correcting plausible-looking code is slower than it feels.

Waiting and prompting add up. Screen recordings showed time spent writing prompts, waiting on generations and idling. None of that feels like work, which is part of why the perceived speed-up was so far off.

High quality bars. Mature open-source projects have strict standards for style, tests and documentation. AI output that would pass in a weekend project often does not pass review there.

This does not mean AI coding is useless

It is important to read the study for what it is. METR was explicit that the result applies to a specific setting: very experienced developers, large and mature codebases they knew well, and the tools of early 2025. It did not show that AI slows down all developers, or that it cannot speed up anyone.

Other research points the other way in different settings. A widely cited 2023 experiment by researchers including GitHub found developers using Copilot finished a well-defined task, writing an HTTP server in JavaScript, about 55% faster than a control group. A small, greenfield, clearly specified task is exactly where AI shines.

Put the two together and a picture emerges: AI speeds up work where context is small and the task is well defined, and it can slow down work where deep context and high standards dominate. Developer sentiment echoes the tension. In the 2025 Stack Overflow Developer Survey, a large majority of respondents said they use or plan to use AI tools, yet more said they distrust the accuracy of AI output than trust it.

The real lesson: your gut is a bad stopwatch

The most important finding is not the 19%. It is the gap between feeling and reality. If skilled developers misjudged the effect by roughly 40 percentage points, so can you, your team and your manager.

That has practical consequences:

  1. Measure, don’t vibe-check. Time a few real tasks with and without AI. Track cycle time and review rework, not just lines generated.
  2. Use AI where context is cheap. New prototypes, scripts, tests, boilerplate, unfamiliar languages and documentation lookup are strong use cases.
  3. Feed context deliberately. On big codebases, invest in project instruction files, clear specs and focused prompts, or the model will guess.
  4. Keep diffs small. Ask for one change at a time. Big AI-generated diffs are where review time explodes.
  5. Re-test as tools improve. Models and agentic coding tools have changed a lot since early 2025. A result from then is a baseline, not a verdict.

The bottom line

AI coding tools are powerful, but they are not magic multipliers in every situation. The METR study is a rare, rigorous look at the hardest case, experts on code they know deeply, and it found the tools of the time slowed them down while convincing them otherwise. The fix is not to quit AI. It is to stop trusting the feeling and start measuring the result.

Sources

Up next · 4 min readVibe Coding Went From One Tweet to Word of the Year. Here’s How to Do It Without Getting BurnedAndrej Karpathy coined it in February 2025. By November, Collins Dictionary crowned it. Vibe coding lets anyone build software by describing it, and it comes with very specific traps.Keep reading →

More stories you'll want to read