Evidence · strength 1 of 5 · searched to 2 Nov 2026
Does AI speed up systematic reviews without more errors?
Not enough evidence yet to say whether AI speeds up systematic reviews without more errors. 0 of 2 studies found a clear gain in time per study.
What studies found when review teams used AI to extract data from research papers: time per study and the share of items extracted wrongly.
VerdictNot enough evidence yet
No good evidence
ImpactNo clear effect
0 of 2 studies found a clear gain in time per study.
Evidence strength1 of 5
We have very low confidence in this estimate.
Cost per user a year$
The studies used general AI models; costs per review are small but not reported.
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Time (headline)
No clear effect- Gartlehner et al. (2025): −41 minutes per study Median staff time per study for data extraction and verification, including prompt engineering (AI-assisted 84 vs human-only 125 minutes) (no test reported; range across reviews 78 minutes shorter to 25 minutes longer). Not statistically clear.
the AI-assisted approach reduced the median time required for data extraction by 41 minutes per study compared to human-only extraction (84 vs. 125 minutes).
- Ishikane et al. (2026): −23% Task completion time for data extraction per article (27.5 vs 34.5 minutes) (7.9 minutes faster, 95% CI -1.5 to 17.3 minutes; P = .099). Not statistically clear.
LLM assistance might reduce data extraction time by ∼23% (7.9 minutes per article; 95% CI −1.5 to 17.3 minutes) with no observed loss of accuracy.
Errors
No clear effect- Gartlehner et al. (2025): −2 points Share of extracted data items that were incorrect against the adjudicated reference (AI-assisted 9.0% (95% CI 8.4-9.6) vs human-only 11.0% (95% CI 10.4-11.7); intervals do not overlap but no test of the difference reported).
errors were identified by the adjudicated reference standard in 1,731 cases (AI-assisted 9.0% [95% CI 8.4% to 9.6%] vs. human-only 11.0% [95% CI 10.4% to 11.7%])
- Gartlehner et al. (2025): −0.2 points Share of items with major errors (AI-assisted 2.5% (2.2-2.8) vs human-only 2.7% (2.4-3.1) in the published version (medRxiv gave 2.4-3.2)). Not statistically clear.
was nearly identical between the two groups (AI-assisted 2.5% [2.2% to 2.8%] vs. human-only 2.7% [2.4% to 3.2%])
- Ishikane et al. (2026): +0 points Extraction accuracy judged by a blinded expert (9/9 vs 11/11 correct) (not estimable (no variation)). Not statistically clear.
Extraction accuracy was 100% in both conditions, and no adverse events were reported.
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level2 of 5, from 2 studies graded A, B or C.
- Down one: old models0 of 2 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Consistencywith 2 studies, never downgraded on this test alone.
- Publication biasnot testable with fewer than 10 studies.
- Result1 of 5. We have very low confidence in this estimate.
Behind the average
- One of the six reviews (Gartlehner et al. (2025)): AI-assisted extraction took up to 25 minutes longer per study than human-only; the other five were faster
How, where and why it works
Why it works
2 of 4. The AI extracts data first and a person verifies it, rather than extracting from scratch.
For whom and where
1 of 4. Time savings varied between reviews; one review took longer with AI.
What it takes
2 of 4. Studies ran inside real review teams with human verification of every item.
The full cost
1 of 4. Staff time is reported; licence costs are not.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Use AI for a first extraction pass, then check every item against the paper.
If you are rolling it out
- Compare time and error rates with your usual double extraction on a few reviews first.
What it takes
- Keep a person verifying every extracted item, as both studies did.
What can go wrong
- In one of six reviews, AI-assisted extraction took up to 25 minutes longer per study.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews Gerald Gartlehner, Shannon Kugley, Karen Crotty, Meera Viswanathan, Andreea Dobrescu, Barbara Nussbaumer-Streit, Graham Booth, Jonathan R. Treadwell, Jung Min Han, Jesse Wagner, Eric A. Apaydin, Erin L. Coppola, Margaret Maglione, Rainer Hilscher, Robert Chew, Meagan Pilar, Bryan Swanton, Leila C. Kahwati, 2025. Annals of Internal Medicine 178 (journal) | B | other quasi-experiment, deployed work | 6 ongoing AHRQ systematic reviews, 63 studies, 9,341 data items, each extracted by a human-only team and an AI-assisted team, with blinded adjudication of discrepancies | Claude 2.1; Claude 3 Opus; Claude 3.5 Sonnet | none |
| Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial Masahiro Ishikane, Yuki Kataoka, Yasushi Tsujimoto, Yuki Moriyama, Yukimasa Matsuzawa, Norio Ohmagari, 2026. Open Forum Infectious Diseases 13 (journal) | C | RCT, realistic task | 5 experienced reviewers (4 physicians, 1 pharmacist), 20 task-level evaluations (9 with LLM, 11 without) on mpox articles | OpenAI o3 via API, producing fixed structured summaries of each article | none |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Does the rating hold up?
- Run without company-run studiesNo change: no such studies.
- Run without working papers and preprintsNo change: no such studies.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. The studies do not all state model release dates. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.