International
The Morning Dev

Only the news worth knowing

‹
 

Research

AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace

The Decoder · By Jonathan Kemper ·

AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace

The Brief

  • A new test places AI coding agents in a simulated developer job with colleagues, project boards, and incomplete handoffs.
  • In this setting, agents complete fewer than one in ten tasks, despite performing well on isolated tests.
  • The source reports that even Opus 5.5 and Astra fall far short in this workflow.

Why it matters

Benchmarks based on isolated coding tasks may overstate how reliably agents can carry work through messy, multi-step team workflows. Developers should not assume agents can close tickets autonomously without oversight.

Who’s affected

  • Teams evaluating or deploying AI coding agents

A new real-world test puts AI coding agents in a simulated developer job with colleagues, project boards, and incomplete handoffs, where they have to keep developing software on their own and reliably close open tickets in a messy daily workflow. Even Opus 5.5 and Astra fall far…

How do you feel about this?

0 Comments

    AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace – The Morning Dev