Evidence · strength 2 of 5 · searched to 2 Nov 2026
Do AI tutors help students learn?
No clear effect so far: the studies do not show that AI tutors help students learn. 2 of 6 studies found a clear gain in test scores.
What randomised trials in schools and universities found when students learned with AI tutors, and why unrestricted chatbots can harm learning.
VerdictNo clear effect so far
No good evidence. Studies so far find no clear effect either way on the headline outcome.
ImpactNo clear effect
2 of 6 studies found a clear gain in test scores.
Evidence strength2 of 5
We have low confidence in this estimate.
Cost per user a year?
Some tutors run on free AI tools; school programmes add devices, staff and supervision. Too varied to band.
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Learning (headline)
No clear effect- Bastani et al. (2025): −17% Unassisted closed-book exam score, GPT Base vs control (-0.054 on a 0-1 scale, SE 0.022 (clustered by class)).
students in the GPT Base arm perform statistically significantly worse than students in the control arm by 17%
- Bastani et al. (2025): −0.004 Unassisted closed-book exam score, GPT Tutor vs control (SE 0.013). Not statistically clear.
Student performance in the GPT Tutor arm was statistically indistinguishable from that of the control arm
- Kestin et al. (2025): +0.73 SD Post-test score vs in-class active learning, controlling for pre-test and background (quantile regression) (linear-regression estimate 0.63 SD, described as understated by a ceiling effect).
Such an analysis provides an effect size in the range of 0.73 to 1.3 standard deviations.
- De Simone et al. (2025): +0.31 SD Final assessment total score (English, AI knowledge, digital skills), intention to treat (SE 0.068).
The treatment effect on the total score (weighted) is 0.31 standard deviation (SE = 0.068)
- De Simone et al. (2025): +0.24 SD English skills (main outcome of interest) (SE 0.068).
followed by English skills (0.238 σ, SE = 0.068) and digital skills (0.139 σ, SE = 0.076).
- De Simone et al. (2025): +0.21 SD Regular third-term English exam (broader than the programme's content) (SE 0.067).
treatment also had a positive and significant impact on the third term exam score, with an effect size of 0.206 standard deviation (SE = 0.067)
- LearnLM team and Eedi (2025): +5.5 points Solving the first question of the next topic unaided (knowledge transfer), supervised LearnLM vs human tutor (95% credible interval -1.4 to +12.4 (posterior probability of benefit 93.6%)). Not statistically clear.
a supervised session with LearnLM increased the likelihood of learning transfer to a distinct topic by an additional 5.5 percentage points [-1.4%, +12.4%] relative to human tutoring
- LearnLM team and Eedi (2025): +10.1 points Knowledge transfer to the next topic, supervised LearnLM vs static pre-written hints (95% credible interval +4.6 to +15.4).
LearnLM improved a student's odds of successful knowledge transfer by a factor of 1.55 [1.21, 1.96] relative to static hints, corresponding to an ATE of +10.1% [+4.6%, +15.4%]
- LearnLM team and Eedi (2025): +1.8 points Correct on second attempt after help, LearnLM 93.0% vs human tutor 91.2% (static hints 65.4%) (LearnLM 93.0% [90.4, 95.3]; human 91.2% [88.5, 93.6]). Not statistically clear.
Students receiving guidance from LearnLM answered their second attempt correctly 93.0% [90.4%, 95.3%] of the time.
- Nie et al. (2025): −4.3 points Optional diagnostic exam participation, offered access vs control (intention-to-treat) (SE 1.34, 95% CI -7.10 to -1.82, P = 0.006).
giving students access to and advertising GPT-4 led to a substantial, statistically significant decrease in their exam participation. The effect of access to GPT was negative 4.3 percentage points.
- Nie et al. (2025): +0.7 points Exam score, offered access vs control (intention-to-treat, imputing missing) (90% CI -0.37 to 1.72). Not statistically clear.
Impute for Missingness +0.67 [-0.37, 1.72] 0.09
- Nie et al. (2025): +6.9 points Exam score for adopters (treatment effect on compliers, imputing missing) (90% CI 0.30 to 14.13).
We estimate adopters have a 6.8 percentage point average treatment effect improvement in their exam scores due to using the provided LLM.
- LearnLM team and Eedi (2026): −0.01 SD Renaissance Star Maths (external standardised test), baseline to endline: AI tutoring with question context v standard platform remediation (95% credible interval -0.15 to +0.14). Not statistically clear.
Students receiving AI tutoring with question context achieved outcomes almost identical to students receiving standard remediation, differing by only -0.01 standard deviations [-0.15, +0.14].
- LearnLM team and Eedi (2026): +0.02 SD Star Maths: AI tutoring with personalised context v standard remediation (95% credible interval -0.12 to +0.16). Not statistically clear.
tutoring with personalised context scored similarly, with an effect of +0.02 SD relative to standard remediation [-0.12, +0.16].
- LearnLM team and Eedi (2026): +13.5 points Correcting the immediate mistake (second diagnostic question): tutor-supervised AI with question context 56.9% v standard remediation 43.4% (56.9% [55.2, 58.6] v 43.4% [41.5, 45.3]; posterior probability of an advantage >99.9%).
students corrected their mistake 56.9% [55.2%, 58.6%] of the time when receiving AI tutoring with question context
Quality
Gain- Bastani et al. (2025): +48 Practice-problem grades while AI was available (0.137 (SE 0.031) and 0.361 (SE 0.032) on a 0-1 scale).
students in the GPT Tutor (resp., GPT Base) arm perform 127% (resp., 48%) better on the practice problems compared to students in the control arm.
Wellbeing
Gain- Kestin et al. (2025): +4.1 Engagement (5-point agreement scale) (SD 0.98 vs 0.92; t(311) = -4.5, p < 0.0001). Self-reported.
students in the AI group agreed more strongly (Mean = 4.1, SD = 0.98) than those in the in-class active learning group (Mean = 3.6, SD = 0.92)
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level4 of 5, from 4 grade A or B studies.
- Down one: inconsistent resultsClear gains in two trials, but no clear effect in three others (a guardrailed tutor, a supervised tutor against human tutors, and a coding-course assistant), with no factor named in advance to explain the difference.
- Down one: old models0 of 6 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Flag (no deduction)Mostly working papers or preprints, not yet peer-reviewed (flagged, not deducted).
- Publication biasnot testable with fewer than 10 studies.
- Result2 of 5. We have low confidence in this estimate.
Behind the average
- Prespecified ability, resources and effort groups (Bastani et al. (2025)): Little or no significant difference in effects.
- Students' own view (Bastani et al. (2025)): Students did not see that they had learned less; GPT Tutor students thought they did better.
- Students below and above 40% on the Force Concept Inventory pre-test (Kestin et al. (2025)): Both groups did significantly better with the AI tutor (p < 0.001).
- Female students (De Simone et al. (2025)): Larger effect (interaction 0.420, significant at 5%).
- Students with higher prior exam scores (De Simone et al. (2025)): Benefited more, though effects were positive across the baseline distribution.
- Students from low Human Development Index countries (Nie et al. (2025)): Exam participation rose (42.3% vs 27.5%, +14.8 points), but n = 180 and not significant after multiple-testing correction.
- Girls v boys (Star Maths) (LearnLM team and Eedi (2026)): Under standard remediation boys outpaced girls by +0.31 SD; AI tutoring narrowed the gap to +0.09 (question context) and +0.02 (personalised) because tutoring helped girls while tutored boys tended to trail
How, where and why it works
Why it works
2 of 4. Tutors that guide rather than give answers appear to help; an unrestricted chatbot that gave answers away lowered exam scores in one trial.
For whom and where
2 of 4. Trials report gains across ability levels; one found larger gains for girls.
What it takes
2 of 4. Trials describe teacher-designed prompts, supervised sessions and attendance.
The full cost
1 of 4. One programme reports costs; most do not.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Ask the AI for hints and questions, not answers.
- Test yourself without the AI afterwards.
If you are rolling it out
- Use tutors with guardrails set by teachers.
- Measure learning with a test taken without AI.
- Run a comparison group.
What it takes
- Design the tutor to guide, not hand over answers.
- Supervised sessions with a teacher present were common in the trials.
What can go wrong
- Students practising with an unrestricted chatbot scored 17% lower on a later exam without it in one trial.
- Students may think they learned more than they did.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Generative AI without guardrails can harm learning: Evidence from high school mathematics Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, Rei Mariman, 2025. PNAS 122 (journal) | A | field RCT, deployed work | Nearly 1,000 students in about 50 classes (grades 9-11), one high school in Turkey; 2,848 student-session observations | GPT-4, ChatGPT-like interface ('GPT Base') (Mar 2023); GPT-4 with teacher-designed guardrail prompts ('GPT Tutor') (Mar 2023) | none |
| AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, Gregorio Ponti, 2025. Scientific Reports 15: 17458 (journal) | C | RCT, deployed work | 194 students (of 233 enrolled) in Harvard's Physical Sciences 2; crossover over two lessons | GPT-4 (custom tutor 'PS2 Pal' with expert-written prompts and step-by-step solutions) (Mar 2023) | none |
| From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria Martín De Simone, Federico Tiberti, Maria Barron Rodriguez, Federico Manolio, Wuraola Mosuro, Eliot Jolomi Dikoru, 2025. World Bank Policy Research Working Paper 11125 (working paper) | B | field RCT, deployed work | 657 treatment and 671 control students assigned; 422 and 337 completed the final assessment (nine public schools, Benin City) | Microsoft Copilot (powered by GPT-4) | none |
| AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms LearnLM Team, Google & Eedi, 2025. arXiv:2512.23633 (preprint) | B | RCT, deployed work | 165 Year 9-10 students (ages 13-15) in five UK secondary schools; 17 supervising expert tutors | LearnLM (Google, pedagogy fine-tuned Gemini-family model), each message approved or edited by a human tutor | vendor-run: Written by Google's LearnLM team with Eedi (the platform) |
| The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters' Exam Performances Allen Nie, Yash Chandak, Miroslav Suzara, Ali Malik, Juliette Woodrow, Matt Peng, Mehran Sahami, Emma Brunskill, Chris Piech, 2025. arXiv:2407.09975 (preprint) | B | field RCT, deployed work | 5,831 active students from 146 countries in a free online Stanford coding course (3,581 offered GPT-4, 2,250 control); 14.2% of the offered group used it | GPT-4 (OpenAI) in a course-specific chat interface with prompts discouraging direct solutions (Mar 2023) | vendor-funded: OpenAI gave a $15,000 unrestricted grant |
| Measuring the impact of AI tutoring, from near transfer to standardised tests: An expanded RCT in UK mathematics classrooms LearnLM Team, Google & Eedi, 2026. Google technical report, 29 October 2026 (preprint) | C | field RCT, deployed work | 1,179 students in Years 8-10 in 10 UK state secondary schools (58 classrooms), individually randomised to 4 arms; 58 expert tutors | Google Gemini 3.1 Pro (13-21 April 2026), then Gemini 3 Flash (21 April-21 May), then Gemini 3.5 Flash (21 May to end of trial) | vendor-run: Run and written by Google's LearnLM team with Eedi (the platform provider) |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Studies we found but do not count
- Effective and Scalable Math Support: Evidence on the Impact of an AI-Tutor on Math Achievement in Ghana (Henkel et al. (2024), grade C). The paper does not show that the tutor used a generative AI model, and it was run by the tool's developer.
These studies are real and often cited, but they fall outside our rules for this rating. What counts as a study.
Does the rating hold up?
- Run without company-run studies3 studies: strength 1, verdict no good evidence, headline band gain.
- Run without working papers and preprints2 studies: strength 1, verdict no good evidence, headline band no clear effect.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released Mar 2023 to Mar 2023. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.