Does AI help staff get rules and policies right?
No good evidence. Studies so far point to a gain, but there are too few or too weak to rate it.
We show no impact figure until at least two studies qualify.
There is not enough evidence to rate this yet.
These studies mostly used general AI assistants; business plans list about $20–$30 a user a month (checked 2 Nov 2026).
What the studies found
No impact figure yet. Fewer than two studies qualify, so under our rules we do not show how big the effect is. The studies we found are listed below, with what each measured.
How sure are we?
- Starting level2 of 5, from 2 studies graded A, B or C.
- Down one: old models0 of 3 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Down one: not independent3 of 3 studies were run, funded or written by the company whose AI was tested.
- Down one: mostly not peer-reviewedMost of the weight is on working papers or preprints, together with another problem above.
- Publication biasnot testable with fewer than 10 studies.
- Result0 of 5. There is not enough evidence to rate this yet.
Behind the average
- Easy questions (control accuracy about 75% or higher) shown an incorrect suggestion (Gosciak et al. (2026)): Accuracy fell by about two-thirds relative to control
- Hard questions shown a correct suggestion (Gosciak et al. (2026)): Accuracy rose from 22% to 68%
- Caseworkers generally, high-quality chatbot (Gosciak et al. (2026)): Accuracy plateaued below what following every suggestion would give (the 'AI underreliance plateau')
How, where and why it works
1 of 4. AI suggests an answer from the rules; staff decide whether to use it.
2 of 4. In one trial, gains grew with the chatbot's accuracy, and a wrong suggestion lowered accuracy.
1 of 4. One small pilot ran for 14 weeks; about two-thirds of those with access used it.
0 of 4. No study counts cost.
What to do next
Measure your pilot →If you are trying it yourself
- Find the rule itself before you give the answer.
- Treat any answer about money or eligibility as unchecked.
If you are rolling it out
- Measure accuracy on questions with known answers.
- Get qualified advice before staff use it on benefits, tax or legal questions.
What it takes
- Check the chatbot's accuracy on your own rules before staff rely on it.
- Train staff to doubt it on easy questions too.
What can go wrong
- Caseworkers shown a wrong suggestion got about 18 points less accurate than those with no chatbot.
- In a large trial with the public, people offered a chatbot were slightly less accurate.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Generative AI in Real-World Workplaces: Experiment with Licensing Chatbot for Sellers (Ngwe, Peckham, Gruber-Gremlich, Smith study) Report eds. Sonia Jaffe, Neha Parikh Shah, Jenna Butler, Alex Farach, Alexia Cambon, Brent Hecht, Michael Schwarz, Jaime Teevan; study by Donald Ngwe, Ried Peckham, Ulrike Gruber-Gremlich, Tyler Smith, 2024. Microsoft Research technical report, July 2024 (working paper) | C | RCT, lab or benchmark | 64 Microsoft sellers randomly assigned to have or not have the chatbot (arm sizes not stated) | Licensing chatbot trained on Microsoft licensing policy documents (model not stated) | authors employed by vendor: Microsoft researchers testing a chatbot about Microsoft licensing with Microsoft sellers (vendor-run) |
| LLMs in social services: How does chatbot accuracy affect human accuracy? Jennah Gosciak, Eric Giannella, Zhaowen Guo, Michael Chen, Allison Koenecke, 2026. arXiv 2603.11213v1 (preprint) | C | RCT, realistic task | 125 caseworkers at Los Angeles nonprofits: 31 control (no suggestions), 94 treated across 10 levels of chatbot accuracy (53% to 100%); 45 multiple-choice SNAP eligibility questions each. 79 passed an attention check; main results use all 125 | GPT-4 generated the benchmark questions and the suggestions; suggestions shown to caseworkers were hard-coded correct or incorrect answers checked by SNAP quality-control experts, not live chatbot output (Mar 2023) | authors employed by vendor: One author is at Nava Labs, whose custom LLM chatbot inspired the study and whose pilot supplied 41 of the 125 participants |
| Helping the Helpers: Evaluating a GenAI-powered assistive chatbot for caseworkers Nava Labs (Nava Public Benefit Corporation), with Better Government Lab (Georgetown) and Cornell Tech researchers for the offline experiment, 2026. Nava Labs report, 13 March 2026 (working paper) | D | other quasi-experiment, deployed work | 14-week pilot, 61 caseworkers at six Los Angeles County agencies; 57 baseline surveys, 43 endpoint surveys, 39 with both; 31 had chatbot access, 20 of them used it (271 prompts). Plus the 125-person offline experiment (see gosciak-2026-caseworker-chatbot-accuracy) | Foundational LLM with retrieval over pre-vetted sources, built into Amplifi's Benefit Navigator; model not named in the parts read | authors employed by vendor: Written by Nava Labs, which built the open-source chatbot |
Studies we found but do not count
- AI chatbots in public services (trial slides, 12 Dec 2023; companion blog: ChatGOV: Will people trust AI tools to help them use public services?) (Nesta (2023), grade C). Not work: members of the public on a web quiz. Those offered a bot were slightly less accurate, the only large randomised result on this question. Shown as context, not counted (section 4, rule 1).
- 5 things we learned testing GOV.UK Chat, an AI assistant for government (Dub and McDonald (2026), grade D). No comparison with people working without the AI (section 4, rules 1 and 3): the accuracy figure scores the AI alone.
Does the rating hold up?
- Run without company-run studies0 studies: strength 0, verdict no good evidence.
- Run without working papers and preprints0 studies: strength 0, verdict no good evidence.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released Mar 2023 to Mar 2023. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. This page is not yet in search engines: health, legal and finance pages wait for a named domain reviewer, and none is appointed yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.