Does AI help customer service agents?
No good evidence. Studies so far point to a gain, but there are too few or too weak to rate it.
We show no impact figure until at least two studies qualify.
There is not enough evidence to rate this yet.
Contact-centre AI is priced by contract and varies too much to band.
What the studies found
No impact figure yet. Fewer than two studies qualify, so under our rules we do not show how big the effect is. The studies we found are listed below, with what each measured.
How sure are we?
- Starting level2 of 5, from 2 studies graded A, B or C.
- Down one: old models0 of 2 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Down one: not independent2 of 2 studies were run, funded or written by the company whose AI was tested.
- Down one: mostly not peer-reviewedMost of the weight is on working papers or preprints, together with another problem above.
- Consistencywith 2 studies, never downgraded on this test alone.
- Publication biasnot testable with fewer than 10 studies.
- Result0 of 5. There is not enough evidence to rate this yet.
Behind the average
- Lowest performers (Q1) (Ni et al. (2026)): Dissatisfaction down 0.215 and ratings up 0.812 points (ITT)
- Top performers (Q5) (Ni et al. (2026)): Quality worsens: dissatisfaction up 0.070, ratings down 0.283 points; retrials up 0.009 (ITT)
- Usage by performance (Ni et al. (2026)): Usage falls from 39.3% (Q1) to 18.3% (Q5)
How, where and why it works
2 of 4. One trial points to the AI passing on good answers: newer and weaker agents gained most, top performers least.
2 of 4. One trial split results by agent skill: the lowest performers gained most, and the top performers' quality slipped.
1 of 4. Agents used the suggestions only part of the time, and less often the better they were.
0 of 4. No study reports the cost of the tool or of checking its output.
What to do next
Measure your pilot →If you are trying it yourself
- Treat suggestions as drafts: check facts and policy before you send.
- Note where the AI gets things wrong and report it.
If you are rolling it out
- Pilot with new agents first, with a comparison group.
- Measure repeat contacts and customer ratings, not only handle time.
- Watch whether your best agents' quality changes.
What it takes
- Expect uneven use: in one trial, the weakest agents used the AI about twice as often as the strongest.
- Suggestions need to be trained on your own past conversations and policies.
What can go wrong
- Top performers' customer ratings fell slightly with AI suggestions in one large trial.
- Faster handling can hide unresolved issues: measure repeat contacts, not just speed.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Generative AI in Action: Field Experimental Evidence from Alibaba's Customer Service Operations Xiao Ni, Yiwei Wang, Tianjun Feng, Lauren Xiaoyuan Lu, Yitong Wang, Congyi Zhou, 2026. arXiv:2603.29888, v2 31 Jul 2026 (preprint) | B | field RCT, deployed work | 5,940 new after-sales chat agents (tenure under one year) on Taobao: 2,895 treated, 3,045 control; about 2.56 million chats | Alibaba in-house gen AI assistant combining Qwen LLMs with fine-tuned ML models (Qwen version not stated); developed in 2023 | vendor-run: Alibaba built the assistant and ran the A/B test in its own operation |
| Generative AI and Sales Productivity: Field Experiments in Online Retail (Live Chat Translation experiment) Lu Fang, Zhe Yuan, Kaifu Zhang, Dante Donati, Miklos Sarvary, 2026. arXiv 2510.12049 v6 (preprint) | C | field RCT, deployed work | About 0.2 million non-English-speaking consumers randomised (two-thirds treated) to Filipino service agents with or without GenAI real-time translation; number of agents not reported | Platform's GenAI bidirectional real-time translation (model not named) | authors employed by vendor: The tool was built and run by the partner cross-border retail platform |
Studies we found but do not count
- Generative AI at Work (Brynjolfsson et al. (2025), grade B). The assistant was built on GPT-3 and rolled out from 2020, before our November 2022 cut-off. It is the best-known study of AI in customer service, so we show it as context.
- Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations (Tannenbaum (2026), grade D). The seller's own unreviewed paper, and an agent talking to shoppers rather than people using AI at work (section 4, rule 1). Very large, and shows a clear revenue gain on its own terms.
- Agentic AI and Human-in-the-Loop Interventions: Field Experimental Evidence from Alibaba's Customer Service Operations (Wang et al. (2026), grade B). Here the AI answered customers itself while staff supervised: a different use from AI that assists human agents. Shown as context.
Does the rating hold up?
- Run without company-run studies0 studies: strength 0, verdict no good evidence.
- Run without working papers and preprints0 studies: strength 0, verdict no good evidence.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. The studies do not all state model release dates. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.