Catalog My learning Blog
AI-Assisted Coding Course: Agentic Development in 30 Days · Phase 1 · Lesson 5 · 5 min read

Does AI Make Developers Faster? METR and DORA Research

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Measuring Impact on Your Own Team
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Read the evidence on AI coding productivity and quality critically: what controlled studies, large surveys and repository analyses found, why perceived and measured speed differ, and how to measure the impact of AI tools on your own team without fooling yourself.

Concept Explanation

Claims about AI coding productivity range from ten times faster to no effect at all. The evidence is more interesting than either extreme, and knowing it protects you from both hype and dismissal.

The most cited controlled study came from METR in July 2025. In a randomized trial, 16 experienced open-source developers worked on 246 real issues in large repositories they knew well, with AI tools allowed or disallowed per issue. They were 19 percent slower with AI, yet afterwards estimated that AI had made them about 20 percent faster. METR's 2026 follow-up with newer tools found estimates moving toward a speedup but with wide uncertainty, and reported that 30 to 50 percent of developers chose not to submit tasks they did not want to do without AI, which itself makes measurement hard. The robust lesson is the perception gap: developers' sense of speed is not a reliable measure.

Large surveys describe adoption and organizational effects. The 2025 DORA report, based on nearly 5,000 technology professionals, concluded that AI acts as an amplifier: it magnifies the strengths of teams with good practices and the dysfunctions of struggling ones. Teams with strong testing, small batches and clear platforms benefit; teams without them see more instability. The 2026 Stack Overflow survey found 84 percent of developers using or planning to use AI tools, with trust in AI output lagging well behind usage.

Repository analyses show quality signals. GitClear's 2025 study of 211 million changed lines found an eightfold increase in duplicated code blocks during 2024 and, for the first time, more copy-pasted lines than moved, meaning refactored, lines. Duplication and declining refactoring are classic precursors of maintenance cost. Security studies tell a similar story, which Day 21 covers.

Learning effects matter for every team that hires juniors. In a January 2026 randomized study, Anthropic found that developers who learned a new Python library with AI assistance scored 17 percent lower on a quiz about it shortly afterwards than those who coded by hand, while finishing only slightly faster. Developers who used AI to ask questions and explanations retained much more than those who delegated the code.

Taken together, the evidence says AI tools can speed up well-specified work, especially for unfamiliar code and boilerplate, but gains are not automatic, are easily overestimated, and can be offset by review effort, rework, duplication and lost learning. Teams that measure outcomes, not impressions, find out which of these applies to them.

SourceMethodKey finding
METR (July 2025, update 2026)Randomized trial, experienced developers, real issues19% slower but felt 20% faster; newer data uncertain
DORA 2025Survey of nearly 5,000 professionalsAI amplifies existing strengths and weaknesses
Stack Overflow 2026Developer survey84% use or plan to use AI; trust lags usage
GitClear 2025211 million changed linesDuplicated blocks up 8x in 2024; refactoring declining
Anthropic (January 2026)Randomized learning studyAI group scored 17% lower on mastery quiz

Worked Example: Measuring Impact on Your Own Team

Self-reported speed is unreliable, so measure delivery outcomes before and after adoption. This script pulls merged pull requests from GitHub and reports cycle time and size, two signals that are hard to fool.

Code
#!/usr/bin/env bash
# pr-metrics.sh: cycle time and size of merged PRs in a date range (requires gh and jq).
set -euo pipefail
REPO="${1:?owner/repo}"; FROM="${2:?YYYY-MM-DD}"; TO="${3:?YYYY-MM-DD}"

gh pr list --repo "$REPO" --state merged --limit 1000 \
  --search "merged:$FROM..$TO" \
  --json number,createdAt,mergedAt,additions,deletions \
| jq -r '
    map({
      hours: (((.mergedAt | fromdateiso8601) - (.createdAt | fromdateiso8601)) / 3600),
      size: (.additions + .deletions)
    })
    | sort_by(.hours) as $s
    | "PRs merged:          \(length)",
      "Median cycle time h: \($s[(length/2|floor)].hours | floor)",
      "Median size (lines): \((sort_by(.size))[(length/2|floor)].size)",
      "PRs over 400 lines:  \(map(select(.size > 400)) | length)"'

Run it for a quarter before adoption and a quarter after, and pair it with change failure rate and the number of reverts from your incident and deploy data. If cycle time falls while size, reverts and failures stay flat, the tools are helping. If size and reverts rise, you are shipping more code, not more value.

In METR's 2025 randomized trial, experienced developers expected AI to speed them up by about 24 percent, believed afterwards it had sped them up by about 20 percent, and were measured 19 percent slower.
In METR's 2025 randomized trial, experienced developers expected AI to speed them up by about 24 percent, believed afterwards it had sped them up by about 20 percent, and were measured 19 percent slower.

Real-World Examples

  • METR's July 2025 randomized study of experienced open-source developers found they took 19 percent longer with AI tools while believing they had been about 20 percent faster, and its February 2026 update described how selection effects complicated later measurements.
  • Anthropic's January 2026 study of 52 mostly junior developers learning a new library found the AI-assisted group averaged about 50 percent on a follow-up quiz versus about 67 percent for the hand-coding group, with the gap smallest for those who used AI to ask conceptual questions.
Key TakeawaysEvidence on AI coding productivity is mixed and nuanced, and developers' own sense of how much faster they work is unreliable. AI amplifies a team's existing practices, repository data shows rising duplication, and learning suffers when AI does the thinking. Measure your own team on delivery outcomes such as cycle time, change size, change failure rate and reverts, before and after adoption, rather than on impressions or lines of code.

Review Questions & Answers

The perception gap: developers believed AI had made them about 20 percent faster even though they were measurably slower, so self-reported speed is not a reliable productivity measure.

AI magnifies what a team already has. Teams with good engineering practices such as strong testing and small batches benefit, while teams with weak practices see their instability and dysfunction increase.

Rising duplication and declining refactoring mean more copies of logic that must be maintained and kept consistent, a classic driver of bugs and long-term maintenance cost.

AI makes producing code cheap, so more lines can mean more duplication and review burden rather than more value. Outcome measures such as cycle time, change failure rate and reverts reflect delivery better.