Research
AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace
The Decoder · By Jonathan Kemper ·
The Brief
- A new test places AI coding agents in a simulated developer job with colleagues, project boards, and incomplete handoffs.
- In this setting, agents complete fewer than one in ten tasks, despite performing well on isolated tests.
- The source reports that even Opus 5.5 and Astra fall far short in this workflow.
Why it matters
Benchmarks based on isolated coding tasks may overstate how reliably agents can carry work through messy, multi-step team workflows. Developers should not assume agents can close tickets autonomously without oversight.
Who’s affected
- Teams evaluating or deploying AI coding agents
- Opus 5.5
- #ai-agents
- #coding-agents
- #benchmarks
- #evaluation
- #workflow-automation
A new real-world test puts AI coding agents in a simulated developer job with colleagues, project boards, and incomplete handoffs, where they have to keep developing software on their own and reliably close open tickets in a messy daily workflow. Even Opus 5.5 and Astra fall far…