How we rate what works

A use case is one kind of work done with AI, such as answering customer queries or writing code. For each one we ask: when people use AI for it, what happens to their time, output and quality? We answer only from published studies, never from our own impressions, vendor claims or surveys of opinion.

Each rating has four parts, shown side by side and never combined into one score: a verdict (which way the evidence points), the impact (how big the effect is, outcome by outcome), the strength of the evidence (how sure we can be, 0 to 5) and the cost.

A study enters our evidence map when it measures what happened when people used generative AI (a model released from November 2022) for real or realistic work; it reports an outcome such as time, output, quality, errors, cost, wellbeing or learning; it compares against people without AI, or before and after; and it is public, as a journal article, working paper or published evaluation.

Vendor case studies and press releases can enter only at the lowest grade and never decide a verdict alone.

In practice, “a model released from November 2022” means the study's fieldwork ran mainly after that date with a generative AI tool. Studies whose tool and fieldwork came earlier, or that test a different use, are listed on each page as context but not counted.

Then the level drops by one for each problem: inconsistent results that nothing explains (never with only two studies); old models, when fewer than half the studies used a model released in the last 18 months (today: since May 2025); not independent, when most studies were run, funded or written by the company whose AI was tested, or the effect shrinks without them; not real work, when most studies used lab or benchmark tasks; self-reported, when the headline outcome is what people said rather than what was measured; and publication bias, tested once 10 or more studies allow it.

Evidence made only of grade D studies cannot start above 1; only C and D, not above 3; with no grade A, not above 4. Working papers and preprints are flagged but not deducted, because much of the best evidence on AI at work is not yet in journals.

One row per outcome (time, output, quality, errors, learning and so on), never one fused score. Each row shows the studies' own numbers and a plain band:

Each study adds one result to the headline outcome. Where a study tests two products or several tasks against one control, we count its least favourable result and show the others alongside.

These bands are provisional until the map holds 30 pooled effects; then they will come from the map's own spread of results. Where a study counts the time people spend checking the AI's output, the time figure includes it; where it does not, the page says so.

The verb carries the certainty: “works”, “probably works”, “may work”. We never say “proven”. A verdict is computed by these rules, not chosen; if we ever think the rules give the wrong answer, we publish the exception and the reason.

We count licences, setup, training and the time people spend checking the output, and leave out what an organisation already has. Bands are reviewed every year.

Every use case also gets four scores from 0 to 4, never added up: do the studies tell us why it works, for whom, what it takes (training, review steps, data) and what it really costs? This is the EMMIE method used by the UK College of Policing's What Works Centre.

These are pilot ratings. Three simplifications apply until the evidence map is bigger, and each page says so:

No pooling yet. Each impact row shows the range across studies and is banded on their median, instead of a pooled estimate with an interval. Equal weights. Each study counts once, whatever its size. Model age by date. The old-models test uses release dates only.

The rules are written down and dated here before any rating changes. We still plan to register them publicly (on the Open Science Framework), recruit an independent methods reviewer, and appoint named reviewers for health, legal and finance pages. Until a domain reviewer signs a health, legal or finance page off, that page is kept out of search engines.

Registered trials whose results are not public yet, checked in our monthly search (last checked 5 Nov 2026). When results appear, each is read and checked like any other study; some will give sectors such as finance their first rated use cases.

AI research agents search for studies and read each paper. A second, separate pass re-opens every paper and checks every number we show against it, recording the quote and where it appears. Ratings are then computed by code from the graded studies. Nothing is rated from memory, from news reports or from a company's own summary of a study.

Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating. Spot a mistake? Corrections.