88.6%
GPT-5's accuracy on the Cerkno dialect, the highest of 12 models, against 99.8% on standard Slovenian
What the chart shows
DIALECT-COPA tests commonsense reasoning in standard Slovenian and in the Cerkno dialect. All 12 models tested scored lower on the dialect. GPT-5 scored 99.8% in standard Slovenian and 88.6% in the dialect, the best dialect score, followed by Gemini 2.5 Pro (86.4%).
Most other models dropped further: Mistral Medium 3.1 fell from 90.0% to 53.2% and Llama 3.3 from 87.0% to 53.6%. Eight of the 12 scored below 60% on the dialect, close to the 50% expected from guessing between two answers.
Key numbers
| Measure | Value |
|---|---|
| GPT-5, standard Slovenian | 99.8% |
| GPT-5, Cerkno dialect | 88.6% |
| Gemini 2.5 Pro, Cerkno dialect | 86.4% |
| Mistral Medium 3.1, standard Slovenian | 90.0% |
| Mistral Medium 3.1, Cerkno dialect | 53.2% |
| DeepSeek-R1-Distill-Qwen-14B, Cerkno dialect | 49.2% |
How to read it
- COPA (Choice of Plausible Alternatives) asks a model to pick the more plausible of two answers, so random guessing scores about 50%.
- Results cover these 12 models and one dialect. The AI Index notes that dialects differ from standard languages in spelling, vocabulary and grammar and are rarely represented in training data.
- Model names are shortened; the CSV gives versions (for example gpt-5-2025-08-07).
Source: Slovene DIALECT-COPA benchmark leaderboard, 2026, via the AI Index Report 2026 (Stanford HAI), figure 3.7.7: Slobench: accuracy. Chart and text by AI Adoption.
Numbers checked against the published data on 20 October 2026.
Download the data (CSV) · Cite this chart
Numbers checked against the published data on 20 October 2026.
Download the data (CSV) · Cite this chart
Discussion