Evidence · strength 2 of 5 · searched to 2 Nov 2026
Does AI help people analyse data more accurately?
Mixed results: studies of AI for data analysis disagree on whether it helps. 3 of 7 studies found a clear gain in accuracy and 1 a clear loss.
What randomised trials found when researchers, consultants and analysts worked through data tasks with AI: accuracy, errors caught and time taken.
VerdictMixed results
Mixed
ImpactMixed
3 of 7 studies found a clear gain in accuracy and 1 a clear loss.
Evidence strength2 of 5
We have low confidence in this estimate.
Cost per user a year$$
The trials used general AI assistants; business plans list about $20–$30 a user a month (checked 2 Nov 2026).
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Quality (headline)
Mixed- Fitzpatrick et al. (2025): −12% Accuracy of answers to six questions on insurer reporting templates, 40 values in all, graded against ground truth (32.3 Gen AI vs 36.5 control out of 40) (two-tailed t-test p < 0.005; covariate-adjusted effect -3.860 (SE 1.419), p < 0.05 (Table 7, column 3)).
the control group outperformed the treatment group with an average score of 36.5 out of 40 for control and 32.3 out of 40 for treatment
- Business et al. (2025): −1.2 points on a 1-5 scale Accuracy of output on the fictional Excel dataset, blind scored 1-5 (1.5 Copilot users vs 2.7 non-users) (no test or interval reported (3 vs 3 people)). Not statistically clear.
M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users.
- Business et al. (2025): −1.2 points on a 1-5 scale Quality of output (formatting, tone, presentation), blind scored 1-5 (1.5 Copilot users vs 2.7 non-users) (no test or interval reported (3 vs 3 people)). Not statistically clear.
M365 Copilot users completed Excel data analysis more slowly and to a worse quality and accuracy than non-users.
- Purri et al. (2025): −46.2 points Error rate in flagging discrepancies: 8.48% with Octozi vs 54.67% with spreadsheets (paired t-test p < 0.001; Cohen's d 2.8 for accuracy).
the error rate plummeted from 54.7% with manual review to just 8.5% with Octozi assistance
- Brodeur et al. (2026): −2.5 points Share of teams that computationally reproduced the selected results, AI-assisted vs human-only (p = 0.697 (Table 1); OLS p = 0.771). Not statistically clear.
Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs 91%) and performed similarly on most outcomes
- Brodeur et al. (2026): −0.95 major errors detected per team Major coding errors detected per team, AI-assisted vs human-only (0.74 vs 1.70) (p = 0.049 (Table 1); OLS p = 0.006 (Table 2)).
human-only teams identified on average 1.00 minor and 1.70 major errors, compared with 1.40 minor and 0.74 major errors for AI-assisted teams
- Wiles et al. (2026): +0.49 score points on a scale where 0 equals the data-scientist benchmark Coding task score relative to the data-scientist benchmark (0 = benchmark; control mean about -0.63) (95% CI 0.42 to 0.56; p < 0.001).
On the coding task, treated workers performed 49 percentage points better than the workers in the control group
- Wiles et al. (2026): +0.201 score points on a scale where 0 equals the data-scientist benchmark Statistics task score relative to the data-scientist benchmark (control mean about -0.32) (95% CI 0.15 to 0.25; p < 0.001).
On the statistical understanding problem, the treatment group performed 20 percentage points better than the control group
- Wiles et al. (2026): +0.172 score points Prediction task score (minus mean absolute error; 0 = perfect; control mean about -0.43) (95% CI 0.09 to 0.25; p < 0.001).
On the prediction problem, compared with control group workers, treated workers obtain results 17 percentage points closer to those of data scientists
- Ipeirotis and Zheng (2025): −18 points Share of queries answered correctly against BIRD gold answers (46% vs 64%) (p = 0.08; CI from a 39-point deficit to rough parity). Not statistically clear.
SQL-LLM users answered 46% of queries correctly versus 64% for Snowflake, an 18-point difference favoring Snowflake
- Lu et al. (2025): +38.7 points Accuracy of sample size calculation (94.1% vs 55.4%) (95% CI 27.9-49.6; p < 0.001).
PowerGPT achieving 94.1% (88.3%-97.2%), significantly outperforming the reference group's 55.4% (45.7%-64.7%), with a difference of 38.7% (27.9%-49.6%, p < 0.001)
- Lu et al. (2025): +12 points Accuracy of statistical test selection (95.6% vs 83.6%) (95% CI 3.9-20.0; p < 0.001).
The accuracy of test selection was also higher in the PowerGPT group (95.6% [90.2%-98.2%]) compared to the reference group (83.6% [75.8%-89.3%])
Output
Gain- Purri et al. (2025): +503% Records correctly cleaned in a 30-minute review: median 20.5 with Octozi vs mean 3.4 with spreadsheets (range 3.7x to 15x across the 10 participants; p < 0.001 (paired t-test, stated for all primary measures)).
this increased to a median of 20.5 data points (range: 7-35) within the same 30-minute review period
- Lu et al. (2025): +21.5 points Sample size calculation completion rate (99.3% vs 77.8%) (95% CI 13.8-29.1; p < 0.001).
the completion rate was 99.3% (95.4%-100.0%) in the PowerGPT group versus 77.8% (69.9%-84.1%) in the reference group, with a difference of 21.5%
Time
No clear effect−57% to +21.7%- Fitzpatrick et al. (2025): +3% Time to complete the task, self-reported (33.5 min Gen AI vs 32.5 min control) (two-tailed t-test p = 0.7048). Self-reported. Not statistically clear.
average time for participants in the control group (32.5 minutes) was marginally lower when compared to the treatment group (33.5 minutes)
- Business et al. (2025): +21.7% Time taken, from screen recordings (25:01 Copilot users vs 20:33 non-users) (no test or interval reported (3 vs 3 people)). Not statistically clear.
M365 Copilot users completed Excel data analysis more slowly
- Brodeur et al. (2026): +13.8% Minutes from event start to successful reproduction, AI-assisted vs human-only (93.3 vs 82.0 minutes; % computed from Table 1 means) (p = 0.505 (Table 1)). Not statistically clear.
Minutes to reproduction 82.0 93.3 179.7 -11.3 -97.7 -86.4
- Ipeirotis and Zheng (2025): −31% Completion time per query, LLM interface vs manual SQL (Snowflake) (p = 0.03 (log-scale mixed model); p = 0.04-0.07 raw-scale).
SQL-LLM cut completion time per query by 31% (p = 0.03), an estimated 212 s relative to the Snowflake group's mean
- Lu et al. (2025): −57% Mean minutes per question (4.0 vs 9.3; % computed from reported means) (mean difference 5.3 min (95% CI 3.1-7.6), p < 0.001).
participants using PowerGPT completed each question in an average of 4.0 minutes ... whereas those in the reference group took 9.3 minutes
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level3 of 5, from 2 grade A or B studies.
- Down one: old models0 of 7 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Flag (no deduction)Mostly working papers or preprints, not yet peer-reviewed (flagged, not deducted).
- Publication biasnot testable with fewer than 10 studies.
- Result2 of 5. We have low confidence in this estimate.
Behind the average
- Participants scoring under 100% on the baseline assessment (N=18) (Fitzpatrick et al. (2025)): With Gen AI, the correct-answer rate fell from 79.63% on the assessment to 65.83% on the experiment task
- Participants scoring 100% on the baseline assessment (N=49) (Fitzpatrick et al. (2025)): With Gen AI, the correct-answer rate fell from 100% to 84.48%; the baseline task was too easy to separate people
- Treated participants with more prior Gen AI experience (Fitzpatrick et al. (2025)): Higher quality scores than less experienced treated participants; numbers only in Figure 14
- False positive queries (Purri et al. (2025)): Share of flags that were false positives fell from 48.0% (36 of 75) with spreadsheets to 3.1% (7 of 224) with Octozi
- AI-assisted teams with more prior AI experience (exploratory, non-causal) (Brodeur et al. (2026)): Found coding errors faster than other AI-assisted teams, significant only at the 10% level (p = 0.067 and p = 0.070)
- Teams where ChatGPT led the work (minimal human oversight) (Brodeur et al. (2026)): Reproduced 56.8 points fewer papers than human-only teams (37%) and took about 119% longer.
- Competent coders (statistics task) (Wiles et al. (2026)): Treatment effect 0.16 (95% CI 0.08-0.23), smaller than for those who never coded (interaction -0.154, p = 0.014); treated competent coders not distinguishable from data scientists (Bayes factor 0.37)
- Competent coders (coding task) (Wiles et al. (2026)): treatment effect 0.51 (95% CI 0.38-0.64); treated competent coders not distinguishable from data scientists (Bayes factor 0.25)
- Competent coders (prediction task) (Wiles et al. (2026)): treatment effect 0.27 (95% CI 0.13-0.40), but still 20 points below the data-scientist benchmark
- Non-statisticians (Lu et al. (2025)): Without the tool they completed only 59.7% of sample size tasks; with it they reached 100% completion, close to statisticians
How, where and why it works
Why it works
1 of 4. AI writes and checks code for the analysis; trials differ on whether people catch its mistakes.
For whom and where
2 of 4. Gains were larger for people without coding skills in one trial; in another, teams that let the AI lead did far worse.
What it takes
1 of 4. Trials ran for a day or a few hours; little is known about longer use.
The full cost
0 of 4. No study counts cost.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Ask the AI to write the analysis code, then read and run it yourself.
- Check every number against the source data.
If you are rolling it out
- Pair AI use with review by someone who knows the data.
- Measure accuracy on tasks with known answers before relying on it.
What it takes
- Keep a person in charge of the analysis: in one trial, teams that let the AI lead reproduced far fewer results.
- Expect skills not to carry over once the AI is removed.
What can go wrong
- Teams where the AI led the work reproduced 37% of results, against far higher rates for human-led teams.
- Plain-language database tools were faster but less accurate in one small trial.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Assessing Generative AI value in a public sector context: evidence from a field experiment (Data task: questions answered from insurer regulatory reporting data) Trevor Fitzpatrick, Seamus Kelly, Patrick Carey, David Walsh, Ruairi Nugent, 2025. arXiv 2502.09479v1 (preprint) | C | RCT, realistic task | Data task: 67 participants in the analysis (regression tables, Num.Obs 67; 49 scored 100% on the baseline assessment and 18 scored lower). Of 208 who registered interest across both tasks, 175 completed at least the assessment and 143 (76 Documents + 67 Data) were analysed. Arm sizes not reported. Participants chose their task; random assignment to Gen AI or control happened within the Data task after a baseline assessment without AI | GPT-4o in a natural-language-to-SQL web app (Vanna-based, with a Solvency II schema and 20 example question-SQL pairs) over a SQLite database (May 2024) | none |
| The Evaluation of the M365 Copilot Pilot in the Department for Business and Trade (observed Excel data analysis task only) UK Department for Business and Trade, Digital Data and Technology Monitoring and Evaluation team, 2025. DBT evaluation report, August 2025, Annex C and Findings (government report) | D | other quasi-experiment, realistic task | Excel data analysis: 3 Copilot users and 3 non-users. The same six people also did the PowerPoint step of the same session. All four observed tasks together: 11 sessions, 6 users and 5 non-users. Pilot: 1,000 licences, October to December 2024. Users came from a stratified random sample of diary respondents in analysis, finance or delivery roles; non-users were colleagues named by respondents, balanced on grade where possible | Microsoft 365 Copilot in Excel (version not stated) | none |
| Leveraging AI to Accelerate Medical Data Cleaning: A Comparative Study of AI-Assisted vs. Traditional Methods Matthew Purri, Amit Patel, Erik Deurell, 2025. arXiv 2508.05519v2 (preprint) | D | before-and-after, realistic task | 10 experienced medical data reviewers (two or more years in medical data review), each doing both conditions in a fixed order on synthetic oncology trial data with 10% of data points seeded with discrepancies | Octozi platform: proprietary LLMs derived from Llama 4, fine-tuned for clinical trial data cleaning, plus heuristic rules (Apr 2025) | authors employed by vendor: All three authors list Octozi, the company that makes the tool |
| AI-Assisted Teams Outperform AI-Led Teams but Not Human-Only Teams in Assessing Research Reproducibility in Quantitative Social Science Abel Brodeur, David Valenta, Alexandru Marcoci, Juan P. Aparicio, Derek Mikola, Bruno Barbarioli, Rohan Alexander, Lachlan Deer, Tom Stafford, Lars Vilhuber, Gunther Bensch, Fabio Motoki, et al. (288 participating co-authors), 2026. Proceedings of the National Academy of Sciences (journal) | B | RCT, realistic task | 288 researchers in 103 teams of three (33 human-only, 35 AI-assisted, 35 AI-led); seven events | ChatGPT paid subscription (GPT-4, GPT-4o) | none |
| Generative AI and the temporary upskilling of knowledge workers (working title: GenAI as an Exoskeleton: Experimental Evidence on Knowledge Workers Using GenAI on New Skills) Emma Wiles, Lisa Krayer, Mohamed Abbadi, Urvi Awasthi, Ryan Kennedy, Pamela Mishkin, Daniel Sack, Francois Candelon, 2026. Nature Human Behaviour (journal) | B | RCT, realistic task | 986 BCG consultants randomised; 573 began; 487 submitted task output (graded: coding 300, statistics 330, prediction 298); benchmark of 44 BCG data scientists | ChatGPT (GPT-4) | authors employed by vendor: Co-author P |
| Natural Language Interfaces for Databases: What Changes for SQL-Literate Users? Panos Ipeirotis, Haotian Zheng, 2025. arXiv 2511.14718 (preprint) | C | RCT, lab or benchmark | 20 SQL-literate participants (10 per arm), 12 BIRD benchmark tasks each (240 participant-tasks) | GPT-4o (via OpenAI API) behind a commercial NLIDB, 'SQL-LLM' (pseudonym) | none |
| Empowering Clinical Trial Design through AI: A Randomized Evaluation of PowerGPT Yiwen Lu, Lu Li, Dazheng Zhang, Xinyao Jian, Tingyin Wang, Siqi Chen, Yuqing Lei, Jiayi Tong, Zhaohan Xi, Haitao Chu, Chongliang Luo, Alexis Ogdie, Brian Athey, Alparslan Turan, Michael Abramoff, Joseph C Cappelleri, Hua Xu, Yun Lu, Jesse Berlin, Daniel I. Sessler, David A. Asch, Xiaoqian Jiang, Yong Chen, 2025. arXiv 2509.12471 (preprint) | C | RCT, lab or benchmark | 36 randomised (stratified by institution, UPenn and UTHealth); 35 analysed (17 PowerGPT, 18 reference); 8 power-analysis scenarios each | PowerGPT (LLM agent with statistical engines, on an OpenAI model via the UTHealth Enterprise OpenAI Service; model version not named) | vendor-run: The authors built PowerGPT (a free tool they are deploying across NIH CTSA sites) and ran the evaluation themselves |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Does the rating hold up?
- Run without company-run studies4 studies: strength 1, verdict no good evidence, headline band no clear effect.
- Run without working papers and preprints3 studies: strength 2, verdict no good evidence, headline band no clear effect.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released May 2024 to Apr 2025. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.