Evidence · strength 1 of 5 · searched to 2 Nov 2026
Does AI help people write requirements and designs?
Not enough evidence yet to say whether AI helps people write requirements and designs. 0 of 3 studies found a clear gain in expert-rated quality.
What experiments found when professionals and students wrote or checked software requirements and designs with AI: time, expert-rated quality and errors.
VerdictNot enough evidence yet
No good evidence
ImpactNo clear effect
0 of 3 studies found a clear gain in expert-rated quality.
Evidence strength1 of 5
We have very low confidence in this estimate.
Cost per user a year$$
General assistants list about $20–$30 a user a month (checked 2 Nov 2026). Specialist design platforms are priced separately.
Pilot rating under our v0.3 rules. The rules are published on our method page but not yet independently reviewed, so ratings may change as the method is checked and new studies arrive. Our method.
What the studies found
Quality (headline)
No clear effect- Broccia et al. (2026): −8% Smell-detection accuracy (macro F1) with ChatGPT support vs without, same people in crossover, Bayesian model (95% credibility intervals overlap (Fig 3b); abstract says 'negatively affects'). Not statistically clear.
Observations from the control group (i.e., participants not using ChatGPT) achieved about 8% higher F1-scores on average.
- Boxtel et al. (2025): None Assessor-rated depth and consistency (principal Pega architect and business owner), human plus GenAI teams vs the one human-only team assessed. Not statistically clear.
one of these two teams created the best design, with a coverage that was broader than any of the other teams
- Bluemelhuber and Junker (2026): +0.7 points on a 7-point scale Expert-rated quality, simple tasks (7-point scale, 4.9 vs 4.2).
For simple tasks, quality ratings were higher with GenAI assistance (p = .012)
- Bluemelhuber and Junker (2026): +0 points on a 7-point scale Expert-rated quality, medium tasks (3.7 vs 3.7). Not statistically clear.
no significant difference in quality was detected (p = .990), with identical mean ratings (3.7) for both conditions
- Bluemelhuber and Junker (2026): +0.4 points on a 7-point scale Expert-rated quality, complex tasks (3.6 vs 3.2). Not statistically clear.
For complex tasks, we observed a positive trend toward higher quality with GenAI assistance
Errors
No clear effect- Broccia et al. (2026): +0 no clear difference in F1 Accuracy classifying correctly found smells as harmful or harmless. Not statistically clear.
using ChatGPT has virtually no effect on accuracy when classifying correctly detected smells as nocuous.
Output
No clear effect- Boxtel et al. (2025): +177% Design elements delivered (case types, processes, data objects, integration points, personas): human plus GenAI teams (43 and 54) vs human-only teams (22 and 13). Not statistically clear.
Looking at the numbers alone, the GENAI application contains the most elements. The HUMAN1 and HUMAN2 applications contain the least.
Time
No clear effect- Broccia et al. (2026): +0 minutes, no clear difference Task duration with vs without ChatGPT (95% credibility intervals overlap (Fig 3a)). Not statistically clear.
the use of ChatGPT does not significantly change the task duration.
- Boxtel et al. (2025): +0 qualitative Completion within the 4 hours. Self-reported. Not statistically clear.
had completed the assignment by the end of the time allotted to them.
- Bluemelhuber and Junker (2026): −30.4% Time on simple tasks, GenAI vs manual (mean 184.7 s vs 265.3 s).
with participants completing tasks on average 80.6 seconds faster with GenAI support
- Bluemelhuber and Junker (2026): −31% Time on medium tasks, GenAI vs manual (246.9 s vs 357.8 s).
an average reduction of 110.9 seconds
- Bluemelhuber and Junker (2026): −25.8% Time on complex tasks, GenAI vs manual (318.2 s vs 428.6 s).
participants completing tasks 110.4 seconds faster on average
- Checking time not measured: none of these studies counts the time spent checking the AI’s output separately.
A minus sign on time means the task took less time. Bands are provisional; see how impact is banded. Effects are not pooled yet: each band is set from one result per study, and where studies use different units we show only the direction.
How sure are we?
- Starting level2 of 5, from 2 studies graded A, B or C.
- Down one: old models0 of 3 studies show they used a model released since May 2025 (the 18-month window); the rest used older models or do not say which.
- Flag (no deduction)Mostly working papers or preprints, not yet peer-reviewed (flagged, not deducted).
- Publication biasnot testable with fewer than 10 studies.
- Result1 of 5. We have very low confidence in this estimate.
Behind the average
- Group that did the first session WITH ChatGPT (Broccia et al. (2026)): Improved about 6-7% between sessions vs about 12% for the group that started without ChatGPT; the authors read this as slower skill learning
- Complex requirements tasks (Bluemelhuber and Junker (2026)): Quality gain not statistically clear (p=.090); time gain still clear
How, where and why it works
Why it works
1 of 4. AI drafts requirements or design elements quickly; people judge whether they fit.
For whom and where
1 of 4. One experiment found a clear quality gain on simple tasks only.
What it takes
0 of 4. No study follows real project work over time.
The full cost
0 of 4. No study counts cost.
Four separate scores from 0 to 4, never added up: how much the studies tell us about why it works, for whom, what it takes and what it really costs (the EMMIE method).
What to do next
Measure your pilot →If you are trying it yourself
- Compare the draft with what the users really asked for.
- Ask what it assumed, then check those assumptions.
If you are rolling it out
- Score drafts with and without AI against your own review checklist.
What it takes
- Have someone who knows the system review every draft.
- Use AI for first drafts on simple items before harder ones.
What can go wrong
- Quality gains were not clear on medium or complex requirements in the main experiment.
- Students checking requirements with ChatGPT did no better, and possibly worse, than without it.
The studies
| Study | Grade | Design | People | AI model (released) | Company role |
|---|---|---|---|---|---|
| Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection Giovanna Broccia, Julian Frattini, Chetan Arora, Maurice H. ter Beek, Alessandro Fantechi, Andreas Vogelsang, Alessio Ferrari, 2026. arXiv 2608.21298v1 (preprint) | C | other quasi-experiment, lab or benchmark | 34 BSc computer engineering students at the University of Florence (17 per group), crossover with a one-week washout; each inspected two requirements documents (40 and 39 requirements, 21 and 19 smells) | ChatGPT (GPT-4o / GPT-4.1 models) (May 2024) | none |
| GenAI in application design: does it make a difference? A field experiment using Pega's Blueprint Colette van Boxtel, Robert-Jan Streng, Thijs van Werven, Rik van der Zwan (Atos Amplify), 2025. Atos Group white paper (working paper) | D | other quasi-experiment, realistic task | 9 Atos staff (4 business architects, 4 system architects, 1 tool operator) in 5 teams: GenAI only, 2 human-only, 2 human plus GenAI; one 4-hour design task from a fictitious case | Pega Blueprint (Pegasystems' generative design tool); underlying model not named | vendor-run: Run and written by Atos, a Pega implementation partner whose staff are Pega-certified, to promote its AI services (the document carries sales calls to action) |
| Engineering Better Requirements: Understanding the Impact of GenAI on Task Performance and Quality in Requirements Engineering Benedikt Bluemelhuber, Sebastian Junker, 2026. Proceedings of the 59th Hawaii International Conference on System Sciences (journal) | C | RCT, realistic task | 41 requirements-engineering professionals at one manufacturing company (from 127 invited), each doing 6 tasks at three difficulty levels with and without ChatGPT; start condition randomised (20 vs 21) | ChatGPT (version not stated; participants used it in a separate window with no standard prompts) | none |
Grades: A, a well-run randomised trial in real work; B, a strong natural experiment in real work or a well-run trial on realistic tasks; C, a trial with clear limits or a matched comparison; D, before-and-after, surveys or case studies. How studies are graded.
Studies we found but do not count
- The Perfection Paradox: From Architect to Curator in AI-Assisted API Design (Ahmad et al. (2026), grade D). The AI wrote the specification alone and experts rated it; nobody worked with the AI (section 4, rule 1).
- Using LLMs in Software Requirements Specifications: An Empirical Evaluation (Krishna et al. (2024), grade D). AI output rated alone, and effort compared with three students' own reports (section 4, rule 1).
- Exploring the Use of LLMs for Requirements Specification in an IT Consulting Company (Pasquale et al. (2025), grade D). AI drafts compared with analysts' drafts on one project, with no one working with and without the AI (section 4, rules 1 and 3).
These studies are real and often cited, but they fall outside our rules for this rating. What counts as a study.
Does the rating hold up?
- Run without company-run studies2 studies: strength 1, verdict no good evidence, headline band no clear effect.
- Run without working papers and preprints1 studies: strength 0, verdict no good evidence, headline band no clear effect.
How up to date is this evidence?
Method →Evidence searched to 2 Nov 2026. Studies cover models released May 2024 to May 2024. Rating computed 5 Nov 2026 under Module 15 v0.3 (pilot).
How this page was made: AI research agents searched for studies and read each paper; a second pass checked every number on this page against the paper itself, with the quote and where it appears. The rating is computed by fixed rules, not chosen. No person outside AI Adoption has reviewed it yet. Editorial owner: AI Adoption editors. Spot a mistake? Corrections.
Disclosure: AI Adoption uses AI tools, including Claude from Anthropic, in its research and editorial work. No company pays for or has any say in a rating.