Evidence · strength 2 of 5 · searched to 2 Nov 2026
Does AI improve research papers and peer review?
AI in research writing and peer review may help, but the evidence is not yet strong enough for a firmer verdict. 2 of 3 studies found a clear gain in quality ratings.
What trials at major conferences, on preprint servers and with early-career researchers found when AI gave feedback on reviews and papers, or helped write them.
VerdictMay help
No good evidence. Studies point to a gain, but no factor named in advance explains who gains.
ImpactGain
2 of 3 studies found a clear gain in quality ratings.
Evidence strength2 of 5
We have low confidence in this estimate.
Cost per user a year$
Feedback tools in these trials were free to users; general assistants have free tiers.
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Quality (headline)
Gain- Kim et al. (2026): +0.08 points on a within-AC standardised rating Area-chair rating of review quality, papers reviewed under permissive vs no-LLM policy (paper-level randomisation) (95% CI 0.05 to 0.12; p < 0.001).
corresponding to a mean difference of 0.08 points on the standardized -2.5 to 2.5 rating scale (95% CI [0.05, 0.12])
- Kim et al. (2026): −0.007 points on a within-AC standardised rating Area-chair rating of review quality, reviewers randomised to permissive vs no-LLM policy, same papers (reviewer-level randomisation, 560 paired papers) (95% CI -0.112 to 0.097; p = 0.89). Not statistically clear.
standardized AC quality ratings likewise did not differ between groups (P-C = -0.007, 95% CI [-0.112, 0.097]
- Ratkovic et al. (2025): +0.486 points on a 1-5 expert rating scale Expert rating of clarity (change from baseline draft, AI vs no-AI) (Westfall-Young adjusted p = 0.009).
GPT assistance yielded statistically significant improvements in two dimensions: clarity (estimated effect = 0.486, adjusted p = 0.009) and coherence (estimated effect = 0.427, adjusted p = 0.036).
- Ratkovic et al. (2025): +0.427 points on a 1-5 expert rating scale Expert rating of coherence (Westfall-Young adjusted p = 0.036).
GPT assistance yielded statistically significant improvements in two dimensions: clarity (estimated effect = 0.486, adjusted p = 0.009) and coherence (estimated effect = 0.427, adjusted p = 0.036).
- Ratkovic et al. (2025): +0.114 points on a 1-5 expert rating scale Expert rating of originality (p = 0.875). Not statistically clear.
dimensions such as originality (estimate = 0.114, p = 0.875), analytical depth (estimate = 0.068, p = 0.935), methodological rigor (estimate = 0.273, p = 0.384)
- Ratkovic et al. (2025): +0.273 points on a 1-5 expert rating scale Expert rating of methodological rigour (p = 0.384). Not statistically clear.
dimensions such as originality (estimate = 0.114, p = 0.875), analytical depth (estimate = 0.068, p = 0.935), methodological rigor (estimate = 0.273, p = 0.384)
- Filimonovic et al. (2026): +1% Mean journal placement (SCImago Journal Rank fixed at 2019) of the author's papers, 2025 (significant at the 5% level (coefficient 0.0094, SE 0.0045)).
It rises to approximately 1% in both 2024 and 2025 and is statistically significant at the 5% level in both years.
Output
Gain- Thakkar et al. (2026): +14 extra words added per review Increase in review length (words in summary, strengths, weaknesses, questions), intent-to-treat: selected for feedback vs control (p below 0.01 (Figure 3 caption; comparison sign lost in PDF text)).
being selected to receive feedback caused the average review length to increase by about 14 words more than reviews that were not selected to receive feedback
- Thakkar et al. (2026): +6% Length of author rebuttals to reviews selected for feedback vs control (intent-to-treat) (p below 0.001 (Table 1 caption; comparison sign lost in PDF text)).
authors posted rebuttals that were, on average, 6% longer (48 words) to reviews written by reviewers who were selected to receive feedback
- Kim et al. (2026): +7% Review length (characters), paper-level randomisation (250 characters, 95% CI 204 to 296; p < 0.001).
corresponding to a mean difference of 250 characters (95% CI [204, 296]), or approximately 7% longer reviews
- Kim et al. (2026): +199 characters per review Review length (characters), reviewer-level randomisation within the same papers (95% CI 44 to 354; p = 0.01).
Randomized-P reviews were, on average, 199 characters longer than Randomized-C reviews (P-C = 199, 95% CI [44, 354]
- Wang et al. (2026): +12.55% Number of revised versions posted within one month of feedback delivery (intent-to-treat) (p < 0.05).
Treated papers exhibited an average increase of 0.005 revisions within one month relative to the control group ( < 0.05), corresponding to a 12.55% increase over the baseline revision rate.
- Kusumegi et al. (2025): +36.2% Monthly preprint output after first detected LLM use vs similar non-adopters, arXiv (reported as significant; intervals in supplement (not opened)).
we conservatively observe significant productivity increases, with boosts of 36.2% (arXiv), 52.9% (bioRxiv), and 59.8% (SSRN) relative to non-adopters.
- Kusumegi et al. (2025): +52.9% Monthly preprint output, bioRxiv (reported as significant; intervals in supplement (not opened)).
we conservatively observe significant productivity increases, with boosts of 36.2% (arXiv), 52.9% (bioRxiv), and 59.8% (SSRN) relative to non-adopters.
- Kusumegi et al. (2025): +59.8% Monthly preprint output, SSRN (reported as significant; intervals in supplement (not opened)).
we conservatively observe significant productivity increases, with boosts of 36.2% (arXiv), 52.9% (bioRxiv), and 59.8% (SSRN) relative to non-adopters.
- Filimonovic et al. (2026): +72% Annual peer-reviewed publications in 2025, GenAI-assisted writers vs matched comparison authors (PPML difference-in-differences) (PPML coefficient 0.5413 (SE 0.0103), 95% CI excludes zero; range 52-88% across specifications).
The transformed estimates indicate that productivity among GenAI-assisted writers increased by approximately 19% in 2023, 42% in 2024, and 72% in 2025 relative to matched comparison authors
- Bao et al. (2025): +2.2 percentile points of downloads within field and year Six-month PDF downloads of bioRxiv preprints flagged as ChatGPT-assisted, full period (last-author fixed effects) (coefficient +0.022 (SE 0.008), p < 0.01).
+0.022 (SE=0.008, p<0.01) for PDF downloads, +0.024 (SE=0.009, p<0.01) for abstract views in the longer full period.
- Bao et al. (2025): −0.7 percentile points of downloads within field and year Six-month PDF downloads, clean period (when ChatGPT was almost certainly the only LLM) (coefficient -0.007 (SE 0.030), p = 0.813). Not statistically clear.
-0.007 (SE=0.030, p=0.813) for PDF downloads
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level3 of 5, from 3 grade A or B studies.
- Down one: old models0 of 7 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Flag (no deduction)Mostly working papers or preprints, not yet peer-reviewed (flagged, not deducted).
- Publication biasnot testable with fewer than 10 studies.
- Result2 of 5. We have low confidence in this estimate.
Behind the average
- Papers whose authors are all in non-English-dominant countries (Wang et al. (2026)): Revisions rose 19.9% relative to control (p < 0.05); similar larger effects for manuscripts with lower scholarly embeddedness (26.4%), lower h-index (20.2%) and earlier-career teams (18.9%)
- English-dominant, more-cited, higher h-index and longer-career groups (Wang et al. (2026)): Effects small and not significant (p = 0.98, 0.63, 0.31 and 0.28)
- Authors with Asian names at institutions in Asia (Kusumegi et al. (2025)): Largest gains: 37.7% (arXiv), 89.3% (bioRxiv), 88.9% (SSRN)
- Authors with Caucasian names at institutions in English-speaking countries (Kusumegi et al. (2025)): Smallest gains: 23.7% (arXiv), 40.0% (bioRxiv), 46.2% (SSRN)
- Researchers in countries linguistically distant from English (Filimonovic et al. (2026)): Output up about 97% in 2025 vs 57% in native English-speaking countries (difference significant at 1%)
- Early-career researchers (7 years or less) (Filimonovic et al. (2026)): Output up about 82% in 2025 vs 64% for senior researchers (difference significant at 1%)
How, where and why it works
Why it works
1 of 4. AI feedback prompts reviewers and authors to revise; studies of output growth infer AI use from writing style.
For whom and where
2 of 4. Researchers writing outside English and early-career researchers gained most in several studies.
What it takes
2 of 4. Conference trials describe exactly how feedback was delivered to reviewers.
The full cost
0 of 4. No study counts cost.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Ask AI to check clarity and structure, and keep your own judgement on the science.
If you are rolling it out
- Pilot AI feedback for reviewers with a randomised comparison, as conferences have.
What it takes
- Feedback worked when delivered inside the reviewing system, with reviewers free to ignore it.
What can go wrong
- Studies that detect AI use from writing style may misclassify authors, so output gains may be overstated.
- Longer reviews are not always better reviews.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| A large-scale randomized study of large language model feedback in peer review (arXiv title: Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025) Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, James Zou, 2026. Nature Machine Intelligence (journal) | B | field RCT, deployed work | 44,831 ICLR 2025 reviews of 11,553 papers: 22,467 reviews randomly selected for LLM feedback (18,946 actually received it) and 22,364 control reviews; papers randomised in equal thirds to none, half or all of their reviews receiving feedback | Claude 3.5 Sonnet (June 20, 2024 version), five-call 'Review Feedback Agent' pipeline (Jun 2024) | none |
| Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026 Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan, Buxin Su, Weijie Su, Alekh Agarwal, Sharon Li, Martin Jaggi, Daniel G. Goldstein, Nihar B. Shah, Miroslav Dudík, 2026. arXiv 2609.19420v1 (preprint) | B | field RCT, deployed work | ICML 2026 main track (23,918 papers, 17,310 reviewers). Paper-level: 16,877 papers allowing either policy, 10% (1,686) randomly switched to the no-LLM policy vs 15,191 left permissive. Reviewer-level: 1,920 willing reviewers randomised (1,008 conservative, 912 permissive), compared within 940 papers reviewed by both groups. Survey N=1,486 | Any LLM the reviewer chose (not specified); permissive policy allowed only 'privacy-compliant' LLMs for understanding the paper and polishing text | none |
| Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment Binglu Wang, Weixin Liang, Jiahui Xue, Yuhui Zhang, Hancheng Cao, Dashun Wang, Yian Yin, 2026. arXiv 2605.24180v1 (preprint) | B | field RCT, deployed work | 31,020 arXiv preprints in the analysis sample (15,542 treatment, 15,478 control) from an initial 34,340 randomised; 45,466 contacted authors in 133 regions | Unnamed 'state-of-the-art' LLM in a multi-agent feedback pipeline (review analysis, title suggestions, grammar check) | none |
| Harnessing GPT for Enhanced Academic Writing: Evidence from a Field Experiment with Early-Career Researchers in the Social Sciences Marc Ratkovic, Thomas Gschwend, Leonard Wendering, Kevin Bauer, Anna-Sophia Kurella, Oliver Rittman, Melanie Sauter, Nicole Schwitter, 2025. Research Square rs-6937665 v2 (preprint) | C | RCT, realistic task | 22 doctoral and postdoctoral social scientists (11 per arm, stratified randomisation), working on their own manuscripts; 5 expert raters | Commercial ChatGPT (GPT-4); any AI tool allowed in the treatment arm | none |
| Scientific production in the era of large language models Keigo Kusumegi, Xinyu Yang, Paul Ginsparg, Mathijs de Vaan, Toby Stuart, Yian Yin, 2025. Science, 18 December 2025 (journal) | D | other quasi-experiment, deployed work | 168,553 authors tracked January 2022 to July 2024 (arXiv 109,965; bioRxiv 43,218; SSRN 15,370), drawn from 2.1 million preprints | Any LLM, inferred from abstracts with a word-distribution detector trained on GPT-3.5-turbo-0125 rewrites | none |
| GenAI-Assisted Writing and Academic Performance: Evidence from the Social and Behavioral Sciences Dragan Filimonovic, Christian Rutzer, Conny Wunsch, 2026. arXiv 2510.02408v2 (working paper) | C | other quasi-experiment, deployed work | Matched panel of 107,418 researchers (751,926 author-years) in economics, psychology and other social sciences (Scopus) | Any GenAI writing tool, inferred from rises in marker words (e.g. 'delve') in titles and abstracts | none |
| Where there's a will there's a way: ChatGPT is used more for science in countries where it is prohibited Honglin Bao, Mengyi Sun, Misha Teplitskiy, 2025. Quantitative Science Studies 6:716-731 (journal) | D | cross-sectional, deployed work | About 106,000 sampled arXiv preprints (April 2019 to March 2024) plus bioRxiv and medRxiv preprints (March 2019 to March 2024); outcome models use last authors with preprints before and after ChatGPT | ChatGPT, inferred with a classifier built on 'polished' abstracts | none |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Studies we found but do not count
- The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas (Si et al. (2025), grade C). It tests where research ideas came from (AI or experts), not researchers working with AI help.
These studies are real and often cited, but they fall outside our rules for this rating. What counts as a study.
Does the rating hold up?
- Run without company-run studiesNo change: no such studies.
- Run without working papers and preprints3 studies: strength 0, verdict no good evidence.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released Jun 2024 to Jun 2024. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.