Evidence · strength 1 of 5 · searched to 2 Nov 2026

Does AI make software developers faster?

Not enough evidence yet to say whether AI coding assistants make developers faster. Across 3 studies, time per task changed by −42.4% to +19%.

What randomised trials and field experiments found when developers wrote and changed code with AI assistants, and how sure we can be.

The studies How sure are we?

VerdictNot enough evidence yet

No good evidence

ImpactMixed

Across 3 studies, time per task changed by −42.4% to +19%.

Evidence strength1 of 5

We have very low confidence in this estimate.

Cost per user a year$$

Paid plans list about $10–$40 a user a month (checked 1 Nov 2026), so roughly $120–$480 a year before checking time. Heavy agent use costs more.

Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.

What the studies found

Time (headline)

Mixed−42.4% to +19%

Becker et al. (2025): +19% Implementation time per issue (developer-logged time, before and after PR review), regression-adjusted for forecast difficulty (95% CI +2% to +39% (stated in METR's 24 Feb 2026 update; shown graphically in Figure 1 of the paper)). Self-reported.we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down.
Paradis et al. (2025): −21% Time on a standardised enterprise-grade coding task (regression-adjusted, best-fit Model 2) (beta1 = -0.24 on log time, 95% CI [-0.51, 0.03], p = 0.086 (not significant at 0.05); unadjusted t-test on log time p = .038). Not statistically clear.Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21%, although our confidence interval is large.
Paradis et al. (2025): −15.8% Raw mean minutes on task, AI vs no-AI (SE 9.3 (AI) and 8.1 (no-AI) minutes).developers who were exposed to AI features completed the task in 96 min (N = 47, SE = 9.3), compared to 114 min (N = 46, SE = 8.1)
Chatterjee et al. (2024): −42.4% Total time to solve algorithmic Python challenges, Copilot vs Copilot-disabled (randomised halves, groups swapped between weeks) (Wilcoxon signed-rank, one-sided, p = 0.001). Self-reported.The group that had access to GitHub Copilot was able to complete their tasks 42.36% faster than the control group participants. This result is statistically significant.

Output

Moderate gain+26.1% to +36.2%

Cui et al. (2026): +26.1% Weekly completed pull requests (pooled weighted IV, treatment-on-treated) (SE 10.3%).our analysis reveals a 26.08% increase (SE: 10.3%) in completed tasks among developers using the AI tool.
Cui et al. (2026): +13.6% Weekly commits (pooled W-IV) (SE 10.0%). Not statistically clear.the number of weekly commits increases by 13.55% (SE: 10.0%)
Cui et al. (2026): +38.4% Weekly builds (pooled W-IV) (SE 12.55%).the number of weekly builds increases by 38.38% (SE: 12.55%)
He et al. (2026): +28.6% Lines added per repository-month (Borusyak et al. DiD, average post-adoption effect) (log coefficient 0.2514 (SE 0.1063); +28.58% (±13.7%)).Lines added increase by about 28.6% (Table 2). There is no statistically significant effect for the volume of commits.
He et al. (2026): +2.6% Commits per repository-month (average post-adoption effect) (log coefficient 0.0260 (SE 0.0429); not significant). Not statistically clear.Commits 0.0260 (0.0429) +2.63% (±4.40%)
Agarwal et al. (2026): +36.2% Commits per repo-month, agent-first repositories (no prior AI IDE) (log coefficient 0.309 (SE 0.051); p < 0.001).AF repos see large average gains (+36.3% commits; +76.6% lines added), whereas IF repos show minimal changes (+3.1%; -6.3%).
Agarwal et al. (2026): +76.6% Lines added per repo-month, agent-first repositories (log coefficient 0.569 (SE 0.103); p < 0.001).AF repos see large average gains (+36.3% commits; +76.6% lines added)
Agarwal et al. (2026): +3.1% Commits per repo-month, repositories that already used AI IDEs (IDE-first) (log coefficient 0.030 (SE 0.092); not significant). Not statistically clear.whereas IF repos show minimal changes (+3.1%; -6.3%).

Quality

No clear effect

Cui et al. (2026): −5.5% Build success rate (pooled W-IV), the paper's code-quality proxy (SE 3.64; not statistically significant). Not statistically clear.26.08** (10.3) 13.55 (10.0) 38.38*** (12.55) -5.53 (3.64)
He et al. (2026): +30.26% Static analysis warnings (SonarQube), average post-adoption effect (log coefficient 0.2644 (SE 0.0511); significant).static analysis warnings increase significantly by 30.3%
He et al. (2026): +41.64% Code (cognitive) complexity, average post-adoption effect (log coefficient 0.3481 (SE 0.0538); significant with Borusyak and TWFE, not with Callaway-Sant'Anna).Code Complexity 0.3481 (0.0538) +41.64% (±7.62%)
Agarwal et al. (2026): +17.73% Static analysis warnings, agent-first repositories (IDE-first estimate +19.00% is not significant) (log coefficient 0.163 (SE 0.048), p < 0.001; IDE-first 0.174 (SE 0.114), not significant).Static Analysis Warnings 0.163***/0.174 (0.048)/(0.114) 17.73 / 19.00
Agarwal et al. (2026): +39% Cognitive complexity, both groups (agent-first log coefficient 0.299 (SE 0.059), p < 0.001, +34.85%; IDE-first 0.357 (SE 0.114), p < 0.01, +42.87%).static-analysis warnings rise by about 18% and cognitive complexity by roughly 39% (Table 2).
Chatterjee et al. (2024): +12.86% Unit test success ratio of submitted solutions (Wilcoxon, p = 0.060; not significant). Not statistically clear.methods written by Copilot Group participants had 12.86% more success ratio than methods written by Control group users. However, this result is not statistically significant.

A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.

How sure are we?

Starting level3 of 5, from 2 grade A or B studies.
Down one: old models0 of 6 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
Down one: self-reportedThe headline outcome is mostly self-reported rather than measured.
Publication biasnot testable with fewer than 10 studies.
Result1 of 5. We have very low confidence in this estimate.

Behind the average

Short-tenure vs long-tenure developers (Microsoft) (Cui et al. (2026)): Short-tenure output up 27% to 39%; long-tenure up 8% to 13% (noisy, not significant at conventional levels)
Junior vs senior developers (Microsoft) (Cui et al. (2026)): Junior output up 21% to 40%; senior up 7% to 16% (noisy, not significant at conventional levels)
Experienced developers working in their own large projects (the whole sample) (Becker et al. (2025)): Tasks took 19% longer when AI tools were allowed, although the developers expected AI to speed them up.
Developers who code more hours per day (Paradis et al. (2025)): Faster with AI than those who code less; interaction large but not significant
More senior developers (Paradis et al. (2025)): Interaction negative (faster with AI) but not significant (p = 0.706)
First and second month after adoption (He et al. (2026)): Velocity gains are front-loaded: commits +55.4% and lines added +281.3% in month one, +14.5% and +48.4% in month two; no significant gain after that (section 4.1.1)
Repositories already using AI IDEs before agents (IDE-first) (Agarwal et al. (2026)): Only a short bump in commits (+16-28% in months 0-2), then lines added about -61% and commits about -35% by month 6, while complexity still rises (section 4)
Self-rated Python proficiency (Chatterjee et al. (2024)): Time reduction 52.27% for beginners (incl. novices), 41.6% intermediate, 40.48% advanced (incl. experts) (Table 3); section 4.9.2 says beginners benefited most, while section 4.9.1 says 'Expert' programmers were helped most, citing Figure 3

How, where and why it works

Why it works

1 of 4. Studies measure speed and output, not why. One trial's screen recordings show time going into prompting, waiting and reviewing the AI's code.

For whom and where

2 of 4. Two studies split results by experience: gains were larger for newer and more junior developers, and smallest or negative for experts in code they know well.

What it takes

1 of 4. Little is reported on training, review rules or rollout. One trial found the gains appeared over months as people took up the tool.

The full cost

1 of 4. Only one study measures time spent checking and fixing AI output (about 9% of working time). None counts licence or review costs together.

Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).

What to do next

Measure your pilot →

If you are trying it yourself

Time a few typical tasks with and without the assistant before you decide it helps.
Read every change the AI makes before you accept it.
Start with routine work such as tests, boilerplate and documentation.

If you are rolling it out

Run a pilot with a comparison group, not a show of hands.
Measure finished work (merged changes, defects) as well as speed.
Track time spent reviewing AI code separately.
Compare tools on the facts: see AI coding agents compared.

What it takes

Developers need time to learn the tool: in one field experiment, take-up grew over months.
Code review stays essential: one trial found developers spent part of their time checking and cleaning the AI's code.
Agree what code and data may be sent to the tool before you start (see the rollout kit).

What can go wrong

Experienced developers working in code they know well may get slower, not faster, while feeling faster.
Open-source projects that adopted AI coding tools saw more static-analysis warnings and more complex code in two studies.
More code is not better code: one study found more builds but no clear change in build success.
People's own estimates of time saved were far off what was measured in one trial.

The studies

Study	Grade	Design	People	AI model (released)	Company role
The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, Tobias Salz, 2026. Management Science, published online 27 Feb 2026 (journal)	B	field RCT, deployed work	4,867 developers analysed across three firms: Microsoft 1,521 (of 1,746 randomised: 616 individually, 1,130 by team), Accenture 316 (of 320), Anonymous Fortune 100 company 3,030 (of 3,054; staggered randomised invitation dates)	GitHub Copilot code completions (underlying model not named in paper) (Jun 2021)	authors employed by vendor: Jaffe and Peng are at Microsoft (owner of GitHub Copilot)
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity Joel Becker, Nate Rush, Elizabeth (Beth) Barnes, David Rein, 2025. arXiv:2507.09089 [cs.AI], v2 25 Jul 2025 (preprint)	C	RCT, deployed work	16 experienced open-source developers; 246 tasks (136 AI-allowed, 110 AI-disallowed) in their own mature repositories	Cursor Pro with Claude 3.7 Sonnet (thinking and standard) and Claude 3.5 Sonnet (main use); also GPT-4o, Gemini 2.5 Pro, o1 (Feb 2025)	none
How much does AI impact development speed? An enterprise-based randomized controlled trial Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, Satish Chandra, 2025. ICSE-SEIP 2025 (journal)	B	RCT, realistic task	96 full-time Google software engineers; 93 (96.9%) completed (47 AI, 46 no-AI)	Google internal AI features in Cider V IDE: AI Code Completion, Smart Paste, Natural Language to Code (model not reported)	authors employed by vendor: All authors are Google employees evaluating Google's own internal AI developer tools
Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner, Bogdan Vasilescu, 2026. MSR '26 (journal)	C	staggered rollout, deployed work	806 GitHub repositories that adopted Cursor (identified by first commit of Cursor rule files) vs 1,380 propensity-matched never-adopting repositories	Cursor (Composer multi-file editing and agent mode; underlying LLMs not named) (Nov 2024)	none
AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development Shyam Agarwal, Hao He, Bogdan Vasilescu, 2026. MSR '26 (journal)	C	staggered rollout, deployed work	Agent-first repositories: 401 treated + 606 matched controls (15,664 repo-months); IDE-first repositories: 117 treated + 73 matched controls (3,572 repo-months)	Autonomous coding agents opening PRs: OpenAI Codex, Claude Code, GitHub Copilot agent, Cursor agent, Devin, Jules, OpenHands, Codegen and others	none
The Impact of AI Tool on Engineering at ANZ Bank: An Empirical Study on GitHub Copilot within Corporate Environment Sayan Chatterjee, Ching Louis Liu, Gareth Rowland, Tim Hogarth, 2024. arXiv 2402.05636 (preprint)	C	RCT, lab or benchmark	Over 100 ANZ engineers; 200 survey data points from the A/B phase, 172 analysed after removing 6 duplicates and 22 unsolved; Wilcoxon tests on 22 paired observations for time and 17 for unit tests	GitHub Copilot (VS Code extension; the paper says it is built on GPT technology but names no specific model) (Jun 2022)	none

Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.

Studies we found but do not count

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng et al. (2023), grade C). Fieldwork ran in May and June 2022 with GitHub Copilot on OpenAI Codex, before our November 2022 cut-off.
We are Changing our Developer Productivity Experiment Design (Becker et al. (2026), grade C). The authors say their own estimate is unreliable and at best a lower bound, so we do not count it.
Generative AI and the Nature of Work (Hoffmann et al. (2024), grade B). It measures how developers' mix of tasks shifted (more coding, less project management), not speed, output or quality; its main estimates also cover July 2022 to July 2023 with a GPT-3-based Copilot.

These studies are real and often cited, but they fall outside our rules for this rating. What counts as a study.

Does the rating hold up?

Run without company-run studies4 studies: strength 0, verdict no good evidence, headline band mixed.
Run without working papers and preprints4 studies: strength 2, verdict no good evidence, headline band no clear effect.

How up to date is this evidence?

Method →

Evidence searched to 2 Nov 2026. Studies cover models released Jun 2021 to Feb 2025. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).

How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.

Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.