AI Makes Developers 19% Slower But They Feel 20% Faster: What METR's RCT Reveals

A landmark randomized controlled trial from METR, the nonprofit AI evaluation research organization, has delivered one of the clearest — and most counterintuitive — findings in the AI productivity debate: experienced developers using AI coding tools completed real tasks 19% slower than those working without them. Yet after the study, those same developers estimated that AI had made them 20% faster.

The gap between what developers felt and what the clock recorded is the central finding of Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, authored by Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein.

What the Study Actually Measured

This was not a survey or a benchmark. METR recruited 16 experienced open-source developers and assigned them 246 real tasks — bug fixes, features, and refactors — from codebases they already knew well (averaging five years of prior experience). Each task was randomly assigned to either an "AI allowed" or "AI disallowed" condition.

When AI was permitted, developers used their tools of choice. In practice, most used Cursor Pro with Claude 3.5 or 3.7 Sonnet, including chat, agent mode, and autocomplete.

Tasks averaged roughly two hours each. Developers recorded their screens and logged completion times.

The Perception Gap

Before the study, developers expected AI to reduce completion time by 24%. After completing it, they still estimated a 20% speedup — even though measured task times showed a 19% slowdown.

METR offers several explanations for why perception and measurement diverge:

• AI makes individual moments feel faster (fast drafts, quick suggestions) while adding overhead across the full task
• Review, correction, and integration time may feel like "normal engineering work" rather than AI-related cost
• The visible first step (fast code generation) is memorable; the slower verification step is less so

What This Does — and Doesn't — Mean

METR is explicit about scope limits. The trial studied experienced contributors in mature, high-quality codebases using early-2025 tools. It doesn't apply to junior developers, greenfield projects, enterprise teams, or later AI generations. METR itself labels the result "historical" and notes that a February 2026 follow-up showed some evidence of speedup, though selection effects complicated interpretation.

Key takeaways:

• Perceived productivity gains from AI tools can be real but misleading — measurement matters
• The perception-measurement gap is a warning for any organization making AI ROI decisions based on self-reported speed
• The result is not a verdict on AI's future in software development, but a caution against vibes-based adoption metrics

Read the full article on ScienceBlog.com