Evidence · strength 0 of 5 · searched to 2 Nov 2026

Does AI help IT and security staff resolve problems faster?

Not enough evidence yet to say whether AI helps IT and security staff resolve problems faster.

What studies found when IT administrators, support engineers and security analysts used AI assistants to troubleshoot and resolve incidents.

The studies How sure are we?

VerdictNot enough evidence yet

No good evidence. Studies so far point to a gain, but there are too few or too weak to rate it.

ImpactNot shown

We show no impact figure until at least two studies qualify.

Evidence strength0 of 5

There is not enough evidence to rate this yet.

Cost per user a year?

Priced by contract or by usage; too varied to band.

Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.

What the studies found

No impact figure yet. Fewer than two studies qualify, so under our rules we do not show how big the effect is. The studies we found are listed below, with what each measured.

How sure are we?

Starting level2 of 5, from 2 studies graded A, B or C.
Down one: old models0 of 3 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
Down one: not independent3 of 3 studies were run, funded or written by the company whose AI was tested.
Down one: mostly not peer-reviewedMost of the weight is on working papers or preprints, together with another problem above.
Publication biasnot testable with fewer than 10 studies.
Result0 of 5. There is not enough evidence to rate this yet.

Behind the average

Less-experienced device managers vs more-experienced (Bono and Xu (2024)): Accuracy gain 33.5% for less-experienced vs 16.2% for more-experienced; difference not significant
Less-experienced identity managers (0-3 years) vs more-experienced (3+ years), sign-in troubleshooting (Bono and Xu (2024)): accuracy gains 57.1% vs 43.0%, both significant, difference not significant

How, where and why it works

Why it works

1 of 4. The assistant summarises incidents and suggests steps; studies do not show which part saves time.

For whom and where

1 of 4. One trial found larger accuracy gains for less experienced administrators.

What it takes

1 of 4. Gains in live security teams appeared after about three months, not in the first month.

The full cost

0 of 4. No study counts cost.

Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).

What to do next

Measure your pilot →

If you are trying it yourself

Check suggested fixes before running them on live systems.

If you are rolling it out

Compare resolution times with a team not using the tool, over several months.
Ask vendors for independent evidence.

What it takes

Expect a learning period: in one study, time savings appeared only after the first months.

What can go wrong

Every study so far was run by the company selling the tool.
Some tasks took longer with the assistant in one trial.

The studies

Study	Grade	Design	People	AI model (released)	Company role
Randomized Controlled Trials for Security Copilot for IT Administrators James Bono, Alec Xu, 2024. arXiv 2411.01067v2 (working paper)	C	RCT, lab or benchmark	182 recruited, 181 completed (about half per arm); Upwork freelancers with 0 to 5+ years of identity/device management experience	Microsoft Security Copilot in Entra and Intune admin centres	vendor-run: Both authors are Microsoft employees evaluating Microsoft's own product
Generative AI and Security Operations Center Productivity: Evidence from Live Operations James Bono, Justin Grana, Alec Xu, 2024. arXiv 2411.03116v2 (working paper)	C	other quasi-experiment, deployed work	89 adopting organisations vs 88 propensity-matched non-adopters; 95,522 security incidents	Microsoft Security Copilot	vendor-run: Microsoft authors using Microsoft Defender XDR telemetry about Microsoft's own product
Generative AI in Microsoft Customer Service and Support Microsoft (Customer Service and Support; statistics from Microsoft Office of the Chief Economist), 2024. Microsoft whitepaper (working paper)	D	other quasi-experiment, deployed work	9,900 Microsoft commercial support agents	Copilot in Dynamics 365 Customer Service	vendor-run: Microsoft studying its own support staff using its own product

Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.

Does the rating hold up?

Run without company-run studies0 studies: strength 0, verdict no good evidence.
Run without working papers and preprints0 studies: strength 0, verdict no good evidence.

How up to date is this evidence?

Method →

Evidence searched to 2 Nov 2026. The studies do not all state model release dates. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).

How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.

Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.