AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test

Researchers from Alibaba and several Asian universities built a coding-agent benchmark from real user change requests. Even the best model fully completes only about one in four tasks, putting the near-perfect scores of older tests into perspective.

The team developed Co-Design Bench, a new test for coding agents. The agents must rework existing web apps based on user requests, sometimes using images, audio recordings, videos, or documents as input.

According to the paper, GPT-6 Astra, the best of the ten models tested, meets about 70 percent of a task's requirements on average. But the result is fully correct in only 26.4 percent of tasks. In most cases the agent handles the bulk of the work but leaves at least one detail unfinished, so anyone using these tools almost always has to clean up by hand.

Coding benchmarks have grown too easy for agents

The authors justify the new test by pointing to the saturation of existing benchmarks. When models can run, check, and fix their own code, they hit scores above 99 percent on those tests, and differences between models can barely be measured.

Co-Design Bench draws on real requests from actual coding sessions instead. The researchers favored cases where users had to rephrase their request multiple times or roll back changes, so the test contains mainly tasks that already gave AI agents trouble in practice. They range from small tweaks, like pulling an amount from a receipt and updating every dependent total, to full-scale project overhauls.

Grading is strict to match. One AI judge tests the finished app in a browser, then a second re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.

Multiple change types in one task overwhelm the models

Performance drops across all models as a task combines more types of changes, including layout, design, interaction, and content. But these intertwined requirements are the norm in real projects