Fancy harnesses make coding agents more expensive but not better, study finds

Earlier comparisons were measuring noise

Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI, for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin that falls well within that noise range. Harnesses are also often wired into the benchmark differently, making comparisons even less reliable.

Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and skips context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts, and automatic history compression.

The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise.

All runs happened offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task themselves. Claude models almost never did this. The researchers discarded those results and don't claim any maker trained on the test data, but the pattern was strong enough that they cut off web access for all remaining runs.

Swapping harnesses moves results about as much as a rerun

On the larger pool of 447 tasks, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band, they're statistically equivalent.