Rapid LLM development is outrunning the research meant to explain it, study finds

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found that 44 percent have already "expired." Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months.

The two examined the claims using models released between July 2024 and September 2026 by eight different developers. The original effect behind the expired findings has largely disappeared, while another 34 percent have weakened significantly. Only 16 percent still hold, and 6 percent have actually gotten stronger. After one year, 16 percent had expired. After two years, that number climbed to 39 percent.

What models "can't do" expires fastest

Claims about what models can't do expire fastest, according to the study. Five of nine reasoning-related findings are considered expired, and none have held up or gotten stronger. They include well-known weaknesses like counting letters in "strawberry," arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and giving up correct answers when users push back. The researchers suspect these kinds of errors have a "correct" outcome, giving developers a clear target to train toward.

Claims about preferences and tendencies prove longer-lived. Of the 11 findings that have held or gotten stronger, three involve LLM behavior as evaluators. They prefer their own answers, rate accepted and rejected papers similarly, and are swayed by linguistic spin in medical abstracts. Three more involve sycophancy and agreement toward users. Moreover, there's a left-leaning political bias, near-identical answers to open-ended questions, and a non-uniform distribution when generating random numbers.