Does AI Make Developers Faster? METR and DORA Research
On this page
Learning Objective
Read the evidence on AI coding productivity and quality critically: what controlled studies, large surveys and repository analyses found, why perceived and measured speed differ, and how to measure the impact of AI tools on your own team without fooling yourself.
Concept Explanation
Claims about AI coding productivity range from ten times faster to no effect at all. The evidence is more interesting than either extreme, and knowing it protects you from both hype and dismissal.
The most cited controlled study came from METR in July 2025. In a randomized trial, 16 experienced open-source developers worked on 246 real issues in large repositories they knew well, with AI tools allowed or disallowed per issue. They were 19 percent slower with AI, yet afterwards estimated that AI had made them about 20 percent faster. METR's 2026 follow-up with newer tools found estimates moving toward a speedup but with wide uncertainty, and reported that 30 to 50 percent of developers chose not to submit tasks they did not want to do without AI, which itself makes measurement hard. The robust lesson is the perception gap: developers' sense of speed is not a reliable measure.
Large surveys describe adoption and organizational effects. The 2025 DORA report, based on nearly 5,000 technology professionals, concluded that AI acts as an amplifier: it magnifies the strengths of teams with good practices and the dysfunctions of struggling ones. Teams with strong testing, small batches and clear platforms benefit; teams without them see more instability. The 2026 Stack Overflow survey found 84 percent of developers using or planning to use AI tools, with trust in AI output lagging well behind usage.
Repository analyses show quality signals. GitClear's 2025 study of 211 million changed lines found an eightfold increase in duplicated code blocks during 2024 and, for the first time, more copy-pasted lines than moved, meaning refactored, lines. Duplication and declining refactoring are classic precursors of maintenance cost. Security studies tell a similar story, which Day 21 covers.
Learning effects matter for every team that hires juniors. In a January 2026 randomized study, Anthropic found that developers who learned a new Python library with AI assistance scored 17 percent lower on a quiz about it shortly afterwards than those who coded by hand, while finishing only slightly faster. Developers who used AI to ask questions and explanations retained much more than those who delegated the code.
Taken together, the evidence says AI tools can speed up well-specified work, especially for unfamiliar code and boilerplate, but gains are not automatic, are easily overestimated, and can be offset by review effort, rework, duplication and lost learning. Teams that measure outcomes, not impressions, find out which of these applies to them.
| Source | Method | Key finding |
|---|---|---|
| METR (July 2025, update 2026) | Randomized trial, experienced developers, real issues | 19% slower but felt 20% faster; newer data uncertain |
| DORA 2025 | Survey of nearly 5,000 professionals | AI amplifies existing strengths and weaknesses |
| Stack Overflow 2026 | Developer survey | 84% use or plan to use AI; trust lags usage |
| GitClear 2025 | 211 million changed lines | Duplicated blocks up 8x in 2024; refactoring declining |
| Anthropic (January 2026) | Randomized learning study | AI group scored 17% lower on mastery quiz |
Worked Example: Measuring Impact on Your Own Team
Self-reported speed is unreliable, so measure delivery outcomes before and after adoption. This script pulls merged pull requests from GitHub and reports cycle time and size, two signals that are hard to fool.
#!/usr/bin/env bash
# pr-metrics.sh: cycle time and size of merged PRs in a date range (requires gh and jq).
set -euo pipefail
REPO="${1:?owner/repo}"; FROM="${2:?YYYY-MM-DD}"; TO="${3:?YYYY-MM-DD}"
gh pr list --repo "$REPO" --state merged --limit 1000 \
--search "merged:$FROM..$TO" \
--json number,createdAt,mergedAt,additions,deletions \
| jq -r '
map({
hours: (((.mergedAt | fromdateiso8601) - (.createdAt | fromdateiso8601)) / 3600),
size: (.additions + .deletions)
})
| sort_by(.hours) as $s
| "PRs merged: \(length)",
"Median cycle time h: \($s[(length/2|floor)].hours | floor)",
"Median size (lines): \((sort_by(.size))[(length/2|floor)].size)",
"PRs over 400 lines: \(map(select(.size > 400)) | length)"'Run it for a quarter before adoption and a quarter after, and pair it with change failure rate and the number of reverts from your incident and deploy data. If cycle time falls while size, reverts and failures stay flat, the tools are helping. If size and reverts rise, you are shipping more code, not more value.
Real-World Examples
- METR's July 2025 randomized study of experienced open-source developers found they took 19 percent longer with AI tools while believing they had been about 20 percent faster, and its February 2026 update described how selection effects complicated later measurements.
- Anthropic's January 2026 study of 52 mostly junior developers learning a new library found the AI-assisted group averaged about 50 percent on a follow-up quiz versus about 67 percent for the hand-coding group, with the gap smallest for those who used AI to ask conceptual questions.
Review Questions & Answers
The perception gap: developers believed AI had made them about 20 percent faster even though they were measurably slower, so self-reported speed is not a reliable productivity measure.
AI magnifies what a team already has. Teams with good engineering practices such as strong testing and small batches benefit, while teams with weak practices see their instability and dysfunction increase.
Rising duplication and declining refactoring mean more copies of logic that must be maintained and kept consistent, a classic driver of bugs and long-term maintenance cost.
AI makes producing code cheap, so more lines can mean more duplication and review burden rather than more value. Outcome measures such as cycle time, change failure rate and reverts reflect delivery better.