SyncAI.news, a Varaisys broadcasting
Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors
SS

Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin H{\aa}kanson, Vikram Nair, Jens Pillgram-Larsen, Nachiappan Nagappan

· 1 min read

ResearcharXiv cs.AI

Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors

arXiv:2611.18890v1 Announce Type: cross Abstract: Teams deploying coding agents need benchmarks built from their own code, but curating and validating them by hand is resource intensive, and an unvalidated benchmark can mislead silently. We present Benchmark Factory, an autonomous meta-agent loop with human checkpoints: an LLM agent writes sourcing queries over landed production code changes, runs models of known relative strength on each candidate, diagnoses why a candidate fails, and repairs it by editing its queries, prompts, and configuration until it clears quality anchors that need no per-task human labels. We used Benchmark Factory to build benchmarks for five developer cohorts at our organization. Engineering raters scored a 5% sample 2.3 for task quality and 2.6 for grader verdicts on a 1-to-3 scale. On configurations never used to build them, the benchmarks ranked Claude Opus~4.7 at high reasoning effort above low effort (60.6% vs. 54.4% of tasks passed, $p=0.002$) and GPT-5.5 above GPT-5.4 (65.8% vs. 60.5%, $p=0.004$); 9 of the 10 per-cohort comparisons pointed the same way. A deliberately trivial benchmark failed the anchors; five broken graders failed them, even under a relaxed rule fixed in advance that the real grader passes; and with model labels shuffled so that no true ordering existed, random redraws of a candidate passed in at most 1.4% of simulations. The anchors are necessary, not sufficient, conditions for a valid benchmark, designed to focus human review on what the loop cannot repair.

Original source

This story was published by arXiv cs.AI and written by Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin H{\aa}kanson, Vikram Nair, Jens Pillgram-Larsen, Nachiappan Nagappan. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News