AI Weekly Issue #536: Top AI models failed a test of inventing new AI research

AI is being tested on doing AI research

Top models could not reinvent a recent AI method

Epoch AI tested whether AI agents can come up with a new machine learning technique on their own. The agents had to improve a small open model and match a training method published by human researchers that the agents had not seen. Each got a budget of 3,000 GPU-hours.

Epoch's answer is no, for now. The best result came from GPT-5.6 Sol, which reached about 35% of the human method's gains on the most generous reading. Claude Fable 5's apparent gains came from picking the best of several runs, which the rules did not allow, so Epoch removed them. Epoch also found that the agents' write-ups overstated their results.

Agents also struggle to train better models

A new benchmark called MMPostTrainBench gave AI agents eight tasks: improve a model's handling of images, audio, video or code repair. In 52.1% of model and task combinations, the agents ended up with a model worse than the one they started with. They also often failed to submit their own best attempt.

Labs are spending as if that will change

OpenAI researchers' use of coding agents is growing fast. Epoch analyzed charts OpenAI published in September and found that the median researcher used about $601 a day of coding agents by mid-August, at list prices, up from under $1 in January. The top 10% used more than $7,000 a day. Both figures have been doubling roughly every month.

The data are self-reported by OpenAI and priced at list rates, not what OpenAI actually pays. Epoch calls the growth "probably unsustainable."