Evidence · strength 1 of 5 · searched to 2 Nov 2026
Does AI make code review faster or better?
Not enough evidence yet to say whether AI makes code review faster or better. 1 of 4 studies found a clear gain in review time and 2 a clear loss.
What trials and field studies found when software teams used AI to review code, suggest fixes or describe changes: review time, issues found and merges.
VerdictNot enough evidence yet
No good evidence
ImpactMixed
1 of 4 studies found a clear gain in review time and 2 a clear loss.
Evidence strength1 of 5
We have very low confidence in this estimate.
Cost per user a year$$
Code review features come with AI coding plans of about $10–$40 a user a month (checked 1 Nov 2026).
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Time (headline)
Mixed- Maddila et al. (2025): +5.5% TimeInReview: time a diff spent in review when AI patches were shown to reviewers (Expt. 1) (p = .029).
We see statistically significant regressions in the time a diff was in the review state (5.5%, p = .029)
- Tufano et al. (2024): +33.3% Total review time, review starting from ChatGPT review (ACR) vs fully manual (MCR): mean 56 vs 42 minutes (% computed) (not statistically significant in regression (no p given)). Not statistically clear.
reviews performed completely manually (MCR) took less time (mean=42, median=37 minutes) than those supported by automation in the ACR (mean=56, median=42 minutes)
- Xiao et al. (2024): −19.3 hours of review time per PR Review time (PR creation to close), PRs with AI-generated descriptions vs without; ATT after propensity score weighting (SE 2.27; p = 1.64e-17).
we find that Copilot for PRs has an impact of decreasing the review time by 19.3 hours
- Cihan et al. (2024): +42% Average pull request closure time, before vs after the bot (5h52m to 8h20m; % computed) (t-test p < 0.001).
The overall average pull request closure duration increased from five hours and 52 minutes before the introduction of CodeReviewBot to eight hours and 20 minutes
Output
No clear effect- Xiao et al. (2024): +57% Odds of a PR being merged (95% CI 1.35-1.84; p < 0.001).
estimated odds ratio for the treatment variable (with Copilot for PRs) is 1.57, with a 95% confidence interval ranging from 1.35 to 1.84
- Cihan et al. (2024): −0.03 human comments per PR Human review comments per pull request (0.31 to 0.28) (Poisson regression: not significant). Not statistically clear.
The average number of human comments decreased from 0.31 to 0.28 with the introduction of CodeReviewBot. However ... this decrease was not statistically significant
Quality
Gain- Tufano et al. (2024): +4.1 issues reported per review Quality issues reported in final review, ACR vs MCR (11.8 vs 7.7 issues) (Dunn's test p = 0.0106 (Benjamini-Hochberg corrected)).
The final reviews of the ACR treatment identified, on average, 11.8 issues (median=10.5), against the 9.6 of CCR (median=9) and the 7.7 of MCR
Wellbeing
No clear effect- Tufano et al. (2024): +0.2 points on a 1-5 confidence scale Reviewer confidence (1-5 scale), ACR vs MCR (3.7 vs 3.5) (no significant effect reported). Self-reported. Not statistically clear.
Reviews from the MCR treatment were scored with an average confidence of 3.5 (median=4), those from the ACR treatment with an average confidence of 3.7
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level2 of 5, from 3 studies graded A, B or C.
- Down one: old models0 of 4 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Flag (no deduction)Mostly working papers or preprints, not yet peer-reviewed (flagged, not deducted).
- Publication biasnot testable with fewer than 10 studies.
- Result1 of 5. We have very low confidence in this estimate.
Behind the average
- After a redesign that showed AI fixes to the author instead (Maddila et al. (2025)): The slowdown disappeared (time in review +0.42%, not significant).
- Project #1 (Cihan et al. (2024)): Closure time rose from 2h48m to 4h38m (p < 0.001)
- Project #2 (Cihan et al. (2024)): Closure time fell from 6h06m to 3h07m (p < 0.001)
- Project #3 (Cihan et al. (2024)): closure time rose from 20h22m to 30h51m (p < 0.001)
How, where and why it works
Why it works
1 of 4. AI comments and suggested fixes add things for reviewers to check, which can slow review down.
For whom and where
1 of 4. Results differed between projects in one study; few studies split results.
What it takes
2 of 4. One large trial changed where AI fixes were shown and removed the slowdown.
The full cost
0 of 4. No study counts cost.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Treat AI review comments as a checklist, not a verdict.
- Keep a human reviewer for every change.
If you are rolling it out
- Measure time in review and defects found, against teams without AI.
- Test where AI suggestions appear before rolling out.
What it takes
- Where suggestions appear matters: in one trial, showing AI fixes to the author instead of the reviewer removed the slowdown.
What can go wrong
- Showing AI-suggested fixes to reviewers made reviews slower in a large field trial.
- AI review comments were mostly minor issues and missed planted serious ones in one experiment.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| AI-Assisted Fixes to Code Review Comments at Scale Chandra Maddila, Negar Ghorbani, James Saindon, Parth Thakkar, Vijayaraghavan Murali, Rui Abreu, Jingyue Shen, Brian Zhou, Nachiappan Nagappan, Peter C. Rigby, 2025. arXiv 2507.13499 (preprint) | B | field RCT, deployed work | Not stated (randomised production code reviews at Meta; paper says experiments ran 'with 10's of thousands of engineers') | GPT-4o (in the two randomised safety trials) | vendor-run: All authors work at Meta, which built MetaMateCR for internal use |
| Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword? Rosalia Tufano, Alberto Martin-Lopez, Ahmad Tayeb, Ozren Dabic, Sonia Haiduc, Gabriele Bavota, 2024. arXiv 2411.11401v3 (preprint) | C | other quasi-experiment, realistic task | 29 developers (28 professional), 72 reviews analysed (24 per treatment) of 12 small programs (six projects, each in Java and Python) with 48 injected issues | ChatGPT Plus (GPT-4) | none |
| Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions Tao Xiao, Hideaki Hata, Christoph Treude, Kenichi Matsumoto, 2024. Proceedings of the ACM on Software Engineering, Vol. 1, FSE, Article 47 (journal) | C | other quasi-experiment, deployed work | 18,256 Copilot-for-PRs pull requests from 146 GitHub repositories vs 54,188 other PRs from 139 of those repositories | GitHub Copilot for Pull Requests (GPT-4) | none |
| Automated Code Review In Practice Umut Cihan, Vahid Haratian, Arda Icoz, Mert Kaan Gul, Omercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Ucar, Eray Tuzun, 2024. arXiv 2412.18531v2 (preprint) | D | before-and-after, deployed work | 4,335 pull requests in three Beko projects (1,568 after the bot); 238 practitioners had access; 22 survey respondents | Qodo PR-Agent based 'CodeReviewBot' (GPT-4 Turbo; GPT-4-32K also named) | none |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Does the rating hold up?
- Run without company-run studies3 studies: strength 1, verdict no good evidence, headline band mixed.
- Run without working papers and preprints1 studies: strength 0, verdict no good evidence, headline band gain.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. The studies do not all state model release dates. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.