Evidence · strength 1 of 5 · searched to 2 Nov 2026
Does AI help people understand long documents?
Not enough evidence yet to say whether AI helps people understand long documents. 2 of 5 studies found a clear gain in quality of understanding.
What trials found when clinicians, central bank staff and civil servants worked with AI summaries of long records and reports: quality, errors and time.
VerdictNot enough evidence yet
No good evidence
ImpactNo clear effect
2 of 5 studies found a clear gain in quality of understanding.
Evidence strength1 of 5
We have very low confidence in this estimate.
Cost per user a year$$
These studies mostly used general AI assistants; business plans list about $20–$30 a user a month (checked 2 Nov 2026).
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Quality (headline)
No clear effect- Chin et al. (2026): +0.2 points on a 5-point agreement scale Clinicians agreeing that pre-charting was efficient and effective (5-point agreement scale), adjusted difference at follow-up (95% CI -0.10 to 0.51 (exploratory); the other two pre-charting items were also null (-0.04 and -0.13)). Self-reported. Not statistically clear.
Between-arm differences on the three pre-charting perception items were small and crossed the null
- Tailor and colleagues (2025): +9 points Non-ophthalmology clinicians saying the note gave them a great deal of diagnostic understanding (33.0% with summary vs 24.0% without) (95% CI 0.3 to 18.2). Self-reported.
vs 24.0% [44 of 185] a great deal; percentage point difference, 9.0; 95% CI, 0.3-18.2
- Tailor and colleagues (2025): +23 points Non-ophthalmology clinicians rating the explanation of the eye condition 'very clear' (62.5% vs 39.5%) (95% CI 13.0 to 33.1 (abstract gives 12.0 to 33.1); P < .001). Self-reported.
62.5% [110 of 176] vs 39.5% [73 of 185] very clear; percentage point difference, 23.0
- Tailor and colleagues (2025): +21.5 points Non-ophthalmology clinicians satisfied with the level of detail in the note (63.6% with summary vs 42.2% without) (95% CI 11.4 to 31.5; P < .001). Self-reported.
satisfied; percentage point difference, 21.5; 95% CI, 11.4-31.5; P < .001
- Tailor and colleagues (2025): +12.1 points Non-ophthalmology clinicians satisfied with the conciseness of the note (70.5% with summary vs 58.4% without) (95% CI 2.3 to 21.9; P = .03). Self-reported.
satisfied; percentage point difference, 12.1; 95% CI, 2.3-21.9; P = .03
- Verma et al. (2025): −1.4 points Accuracy of answers against a gold-standard set, summary plus EHR (86.7%) vs EHR only (88.1%) (p = 0.42; no CI). Not statistically clear.
The Summary+EHR and EHR-only groups had comparable accuracy
- Fitzpatrick et al. (2025): +17% Answer quality, mean score on a 0-4 rubric over 12 questions graded blind by two evaluators (2.21 Gen AI vs 1.89 control) (two-tailed t-test p < 0.05; covariate-adjusted effect +0.227 (SE 0.112), p < 0.05 (Table 5, column 3)).
The average quality score for the treatment group was 2.21 compared to 1.89 in the control group, a mean difference of 0.32.
- Business et al. (2025): +1 points on a 1-5 scale Quality of the report summary, blind scored 1-5 (4 Copilot users vs 3 non-users) (no test or interval reported (3 vs 2 people)). Not statistically clear.
M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users.
Errors
No clear effect- Chin et al. (2026): +68.6% Clinicians saying clinically significant inaccuracy in summaries occurs never or rarely (intervention arm only, 118 respondents; no control comparison) (none reported). Self-reported. Not statistically clear.
clinically significant inaccuracy by 68.6% (median, 2 [IQR, 2 to 3])
- Tailor and colleagues (2025): +26% Ophthalmologist-reviewed summaries containing incorrect information before editing (no comparison group; independent overread found 15% (34 of 235)). Not statistically clear.
26% of PLSs (126 of 489) contained incorrect information prior to editing.
- Business et al. (2025): +1.5 points on a 1-5 scale Accuracy of the report summary, blind scored 1-5 (4 Copilot users vs 2.5 non-users) (no test or interval reported (3 vs 2 people)). Not statistically clear.
M365 Copilot users produced summaries of reports and wrote emails faster and to a higher quality and accuracy than non-users.
Time
No clear effect- Chin et al. (2026): −1.2 seconds per encounter Modified total chart time per encounter, steady state (months 2-3), adjusted difference (95% CI -19.0 to 16.6). Not statistically clear.
Charting time per encounter showed no significant between-arm difference during steady state (-1.2 seconds; 95% CI, -19.0 to 16.6)
- Chin et al. (2026): −0.2 minutes per day Time outside scheduled hours, steady state, adjusted difference (95% CI -7.6 to 7.2). Not statistically clear.
and -0.2 minutes per day during steady state (95% CI, -7.6 to 7.2)
- Tailor and colleagues (2025): +94.9% Ophthalmologists spending one minute or less reading and editing the summary (no comparison group). Self-reported. Not statistically clear.
A total of 94.9% of respondents (464 of 489) spent less than or equal to 1 minute reading and editing the PLS.
- Verma et al. (2025): −8.3% Time to complete the questionnaire per case, summary plus EHR (9.45 min) vs EHR only (10.30 min) (p = 0.46 (Mann-Whitney U); no CI). Self-reported. Not statistically clear.
Access to LLM-generated summaries did not significantly reduce overall questionnaire completion time
- Fitzpatrick et al. (2025): −34% Time to complete the task, self-reported (33 min Gen AI vs 50 min control) (p < 0.05; regression coefficient -17.1 min (SE 3.2), Table 6). Self-reported.
completed the task in 33 minutes, which was significantly shorter than the control group's average completion time of 50 minutes
- Business et al. (2025): −69.6% Time taken (12:37 Copilot users vs 41:34 non-users) (no test or interval reported (3 vs 2 people)). Not statistically clear.
Summarising reports 4 2.5 4 3 12:37 41:34
- Checking time not measured: none of these studies counts the time spent checking the AI’s output separately.
Wellbeing
Gain- Chin et al. (2026): −27.4 points on a 0-400 task-load scale Physician task load (NASA-TLX adapted for pre-charting, 0-400, lower is better), adjusted difference at follow-up; primary outcome (95% CI -49.4 to -5.3; P = 0.02). Self-reported.
The adjusted between-arm difference at follow-up was -27.4 points (95% CI, -49.4 to -5.3; P=0.02)
- Chin et al. (2026): −0.2 points on a 0-4 scale Professional Fulfillment Index overall burnout (0-4, lower is better), adjusted difference at follow-up (95% CI -0.38 to -0.01 (exploratory, not adjusted for multiple comparisons; no P value)). Self-reported.
Overall burnout was lower in the intervention arm (-0.20; 95% CI, -0.38 to -0.01)
- Chin et al. (2026): −0.24 points on a 0-4 scale Professional Fulfillment Index work exhaustion (0-4, lower is better), adjusted difference at follow-up (95% CI -0.47 to -0.02 (exploratory, not adjusted for multiple comparisons; no P value)). Self-reported.
as was the work exhaustion subscale (-0.24; 95% CI, -0.47 to -0.02)
- Chin et al. (2026): +0.04 points on a 0-4 scale Professional Fulfillment Index overall professional fulfillment (0-4, higher is better), adjusted difference at follow-up (95% CI -0.16 to 0.25 (exploratory)). Self-reported. Not statistically clear.
Overall professional fulfillment showed little between-arm difference at follow-up
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level2 of 5, from 4 studies graded A, B or C.
- Down one: old models0 of 5 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Flag (no deduction)Mostly working papers or preprints, not yet peer-reviewed (flagged, not deducted).
- Publication biasnot testable with fewer than 10 studies.
- Result1 of 5. We have very low confidence in this estimate.
Behind the average
- Physicians who used the summary more (Verma et al. (2025)): Time fell as share of answers drawn from the summary rose (Spearman rho -0.62, p = 0.0003); accuracy did not change (rho 0.137, p = 0.476)
- Participants in the bottom half of baseline (assessment) scores (Fitzpatrick et al. (2025)): Gained most: quality 1.38 on the assessment to 2.12 on the experiment task with Gen AI; time fell from 70% to 49% of the limit
- Participants in the top half of baseline scores (Fitzpatrick et al. (2025)): Quality fell from 2.67 to 2.28 with Gen AI (control fell from 2.72 to 2.05); time fell from 85% to 59% of the limit
- Questions needing information gathered from across the whole document (Fitzpatrick et al. (2025)): Gen AI group did worse on the business-model question, where control beat treatment
How, where and why it works
Why it works
1 of 4. AI condenses the text so people read less; it can leave out or invent details.
For whom and where
1 of 4. In one trial, people with lower baseline scores gained most, and questions needing facts from across a whole document did worse.
What it takes
2 of 4. One trial ran for 90 days in live clinics; use fell over that time.
The full cost
0 of 4. No study counts cost.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Ask for page references, then check them.
- Ask directly what the summary leaves out.
If you are rolling it out
- Compare AI summaries with the source on a sample of your own documents.
- Measure logged time, not only how people feel.
What it takes
- Keep the original document open and check the parts you act on.
- Most of this evidence comes from clinical records, so test it on your own documents.
What can go wrong
- In one clinical study, reviewers found incorrect information in about a quarter of summaries before editing.
- In the largest trial, clinicians felt less task load, but logged chart time did not change.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| A Pragmatic Randomized Trial of an EHR-Integrated Generative AI Chart Summarization Tool for Ambulatory Clinicians Aaron T. Chin, Nina Zhu, Sitaram Vangala, Hawkin Woo, Lauren E. Wisk, Thomas Kingsley, John N. Mafi, Paul J. Lukac, 2026. medRxiv preprint posted 31 August 2026 (preprint) | B | field RCT, deployed work | 142 per arm, stated in Table 1 (Intervention N=142, Control N=142) and in the Results text | Epic Outpatient Insights chart summarization (generative AI; underlying model not named in the main text, described in the Supplement) | none |
| Evaluation of AI Summaries on Interdisciplinary Understanding of Ophthalmology Notes Prashant D. Tailor and colleagues (Mayo Clinic, Rochester; full author list not read), 2025. JAMA Ophthalmology (journal) | C | field RCT, deployed work | Surveys sent: 540 intervention and 533 control (1,073 in all), so at least that many encounters were randomised; the paper gives no encounter count. Three randomisation schedules (inpatient, established outpatient, new outpatient), block size 6. | Mixtral 8x7B (Mistral AI), run locally, zero-shot (Dec 2023) | none |
| Verifiable Summarization of Electronic Health Records Using Large Language Models to Support Chart Review Ritchie Verma, Emily Alsentzer, Zachary Strasser, Leslie Chang, Kirollos Roman, Esteban Gershanik, Camellia Hernandez, Miguel Linares, Jorge Rodriguez, Durga Thakral, Ozan Unlu, Jacqueline You, Li Zhou, David Bates, 2025. medRxiv, posted 3 June 2025 (preprint) | C | RCT, realistic task | 8 physicians recruited by snowball sampling (75% attending, 25% fellow), 8 cases each (64 reviews); the regression df, F(17,41), imply about 59 reviews were analysed and the paper does not explain the gap | GPT-4o version 5-13-2024 via Azure OpenAI, temperature 0, extract-then-abstract pipeline (May 2024) | none |
| Assessing Generative AI value in a public sector context: evidence from a field experiment (Documents task: question answering on a ~100-page insurer report) Trevor Fitzpatrick, Seamus Kelly, Patrick Carey, David Walsh, Ruairi Nugent, 2025. arXiv 2502.09479v1 (preprint) | C | RCT, realistic task | Documents task: 76 participants in the analysis (regression tables, Num.Obs 76). Whole trial 143 analysed across both tasks, of 208 who registered interest and 175 who completed at least the assessment phase. Arm sizes within task are not reported in the text. Participants chose Documents or Data; random assignment to Gen AI or control happened within each task after a baseline assessment without AI | GPT-4o in a retrieval-augmented chatbot (web app built by the bank) (May 2024) | none |
| The Evaluation of the M365 Copilot Pilot in the Department for Business and Trade (observed Excel data analysis task only) UK Department for Business and Trade, Digital Data and Technology Monitoring and Evaluation team, 2025. DBT evaluation report, August 2025, Annex C and Findings (government report) | D | other quasi-experiment, realistic task | Summarising reports: 3 Copilot users and 2 non-users; the same five people also did the email task. All four observed tasks: 11 sessions, 6 users and 5 non-users. | Microsoft 365 Copilot (version not stated) | none |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Does the rating hold up?
- Run without company-run studiesNo change: no such studies.
- Run without working papers and preprints2 studies: strength 0, verdict no good evidence, headline band no clear effect.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released Dec 2023 to May 2024. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.