Evidence · strength 1 of 5 · searched to 2 Nov 2026
Does an AI chatbot help doctors diagnose and plan treatment?
Not enough evidence yet to say whether an AI chatbot helps doctors diagnose and plan treatment. 4 of 5 studies found a clear gain in reasoning scores.
What randomised trials found when physicians worked through clinical cases with a large language model beside their usual resources.
VerdictNot enough evidence yet
No good evidence. Studies so far point to a gain, but there are too few or too weak to rate it.
ImpactGain
4 of 5 studies found a clear gain in reasoning scores.
Evidence strength1 of 5
We have very low confidence in this estimate.
Cost per user a year$$
The trials used general AI assistants; paid plans list about $20 a user a month (checked 2 Nov 2026).
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Quality (headline)
Gain- Goh et al. (2024): +2 points Diagnostic reasoning score (structured reflection rubric), LLM plus usual resources versus usual resources (95% CI -4 to 8; P = .60). Not statistically clear.
for the conventional resources-only group, with an adjusted difference of 2 percentage points (95% CI, -4 to 8 percentage points; P = .60)
- Goh et al. (2024): +1.9 odds ratio Correct final diagnosis (binary) (95% CI 0.9 to 4.0; P = .10). Not statistically clear.
treating them as binary (correct vs incorrect) variables did not qualitatively change the results (odds ratio, 1.9; 95% CI, 0.9-4.0; P = .10)
- Goh et al. (2025): +6.5 points Management reasoning score (expert Delphi rubric), LLM plus usual resources versus usual resources (95% CI 2.7 to 10.2; p < 0.001).
Physicians randomized to use the LLM performed better than the control group (43.0% compared to 35.7%, difference: 6.5%, 95% CI: 2.7% to 10.2%, p<0.001)
- Qazi et al. (2025): +27.5 points Diagnostic reasoning score, GPT-4o plus usual resources versus usual resources (95% CI 22.8 to 32.2; P < 0.001).
yielding an adjusted difference of 27.5 percentage points (95% CI, 22.8 to 32.2; P < 0.001)
- Qazi et al. (2025): +34.4 points Correct final diagnosis (95% CI 31 to 37.9; P < .001).
Analysis of the final diagnosis showed a difference of 34.4 percentage points (95% CI, 31 to 37.9 pp; P < .001) between the treatment and control groups
- Everett et al. (2026): +9.9 points Diagnostic reasoning score, AI-as-first-opinion workflow versus conventional resources (85% vs 75%) (95% CI 4.7 to 15; p = 0.00039).
with the clinicians’ using AI as a first opinion (85%, p = 0.00039, mean difference=9.9%, 95% CI = 4.7–15%)
- Everett et al. (2026): +6.8 points Diagnostic reasoning score, AI-as-second-opinion workflow versus conventional resources (82% vs 75%) (95% CI 4.0 to 9.6; p = 3.3e-6).
or AI as a second opinion (82%, p = 3.3e−6, mean difference=6.8%, 95% CI = 4.0–9.6%)
- Everett et al. (2026): +14.9 points Score on clinically actionable decisions (final diagnosis and next steps), AI-as-second-opinion versus conventional resources (95% CI 6.0 to 23; p = 0.00092).
We found that there was a significant 14.9% (p = 0.00092, 95% CI 6.0–23%) increase in the score related to the clinically actionable decisions
- Healy et al. (2026): +8.6 points Diagnostic reasoning score with LLM access versus conventional resources (within physicians) (p = 0.001).
Access to the LLM was associated with improved physician performance compared to using conventional resources (74.3 vs. 65.7 %, p=0.001).
Errors
No clear effect- Goh et al. (2025): +0.9 points Responses with high likelihood of harm. Not statistically clear.
In the LLM-assisted group, 8.3% and 3.8% of physician responses carried medium and high likelihood of harm, respectively, compared to 11.4% and 2.9% in the conventional resources group.
Time
No clear effect- Goh et al. (2024): −82 seconds per case Time per case (95% CI -195 to 31; P = .20). Not statistically clear.
with a time difference of -82 (95% CI, -195 to 31; P = .20) seconds
- Goh et al. (2025): +119.3 seconds per case Time per case (95% CI 17.4 to 221.2; p = 0.022).
Physicians randomized to use the LLM spent 111.3 seconds more on each case (801.5 seconds versus 690.2, difference: 119.3 seconds, 95% CI: 17.4 to 221.2, p = 0.022)
- Qazi et al. (2025): −6.4 seconds per case Time per case (95% CI -68.2 to 55.3; P = 0.84). Not statistically clear.
Mean time per case was similar between groups (603.8 vs. 635 seconds; adjusted difference -6.4 seconds, 95% CI -68.2 to 55.3; P = 0.84).
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level3 of 5, from 3 grade A or B studies.
- Down one: inconsistent resultsThe first diagnostic trial found no clear gain; later trials, some with AI training first or set workflows, found gains, with no factor named in advance to explain the difference.
- Down one: old models0 of 5 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Publication biasnot testable with fewer than 10 studies.
- Result1 of 5. We have very low confidence in this estimate.
Behind the average
- Training level and prior LLM experience (Goh et al. (2024)): No subgroup differed from the overall null result
- Diagnostic-decision questions (Goh et al. (2025)): Largest gain: +12.1 points
- Gender (Qazi et al. (2025)): Men gained 41.6 points, women 17.1 (both positive)
- Years since MBBS (at or below 8.5 years vs more) (Qazi et al. (2025)): Less experienced gained 30 points, more experienced 24.5
- AI-as-second-opinion cases (Everett et al. (2026)): Actionable-decision score fell after seeing the AI in 12 of 147 cases (rose in 52, unchanged in 83)
- Individual physicians (Healy et al. (2026)): Benefit varied widely (SD 12.8 points); only 30% of case questions were put to the LLM
How, where and why it works
Why it works
1 of 4. The AI suggests diagnoses and plans; studies differ on whether doctors use the suggestions well.
For whom and where
2 of 4. One trial found large gains after 20 hours of AI training first; another found no difference by training level or prior AI experience.
What it takes
1 of 4. All trials used written cases, not real patients; training and workflow varied.
The full cost
0 of 4. No study counts cost; one found doctors took about two minutes longer per case.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Use AI as a second opinion, and check its reasoning against guidelines.
- Never enter identifiable patient data into a general chatbot.
If you are rolling it out
- Do not deploy for live diagnosis on vignette evidence alone.
- Train staff, and audit decisions where AI was consulted.
What it takes
- Every trial used written case vignettes, not live patients.
- Training doctors to use the AI may matter: the largest gain followed 20 hours of training.
What can go wrong
- In one trial, doctors with AI took about two minutes longer per case.
- In one study, seeing the AI's opinion lowered the doctor's score in some cases.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A. Cool, Zahir Kanjee, Andrew S. Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P. J. Olson, Adam Rodman, Jonathan H. Chen, 2024. JAMA Network Open 7 (journal) | B | RCT, realistic task | 50 physicians (26 attendings, 24 residents) in family, internal or emergency medicine; up to 6 vignettes each | GPT-4 via ChatGPT Plus (Mar 2023) | none |
| GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial Ethan Goh, Robert J. Gallo, Eric Strong, Yingjie Weng, Hannah Kerman, Jason A. Freed, Joséphine A. Cool, Zahir Kanjee, Kathleen P. Lane, Andrew S. Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P. J. Olson, Jason Hom, Jonathan H. Chen, Adam Rodman, 2025. Nature Medicine 31 (journal) | B | RCT, realistic task | 92 physicians (67 attendings, 25 residents), randomised evenly; 375 physician cases scored | GPT-4 via ChatGPT Plus (Mar 2023) | none |
| AI-literacy training enhances physician-LLM diagnostic collaboration in a resource-limited setting: a randomized controlled trial Ihsan Ayyub Qazi, Ayesha Ali, Asad Ullah Khawaja, Muhammad Junaid Akhtar, Ali Zafar Sheikh, Muhammad Hamad Alizai, 2025. medRxiv 2025.06.06.25329104 (preprint) | B | RCT, realistic task | 60 licensed physicians in Pakistan randomised after 20 hours of AI-literacy training; 58 completed (29 per arm), 342 cases | GPT-4o via ChatGPT (May 2024) | none |
| From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis Selin S. Everett, Bryan J. Bunning, Priyank Jain, Ivan Lopez, Anup Agarwal, Manisha Desai, Robert Gallo, Ethan Goh, Vinay B. Kadiyala, Zahir Kanjee, Jacob M. Koshy, Andrew Olson, Adam Rodman, Kevin Schulman, Eric Strong, Jonathan H. Chen, Eric Horvitz, 2026. npj Digital Medicine 9 (journal) | C | RCT, realistic task | 70 US-licensed physicians (39 residents, 31 attendings); 254 cases (27 AI-first, 43 AI-second participants) | Custom GPT built on GPT-4 (system prompt only, no fine-tuning) (Mar 2023) | authors employed by vendor: Corresponding author Eric Horvitz is Microsoft's Chief Scientific Officer (Microsoft is OpenAI's main investor and resells GPT-4) |
| Human-AI collaboration in clinical reasoning: a UK replication and interaction analysis Justin Healy, Jonathan Kossoff, Matthew Lee, Christian Hasford, 2026. Diagnosis (journal) | C | other quasi-experiment, realistic task | 22 UK physicians, 4 vignettes each (LLM available for 2) | Unnamed LLM via a custom web application (not named in the abstract) | none |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Does the rating hold up?
- Run without company-run studies4 studies: strength 1, verdict no good evidence, headline band gain.
- Run without working papers and preprints4 studies: strength 1, verdict no good evidence, headline band gain.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released Mar 2023 to May 2024. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. This page is not yet in search engines: health, legal and finance pages wait for a named domain reviewer, and none is appointed yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.