Artificial Analysis v5 and the benchmarker's dilemma

Housekeeping: I’m on the road so the voiceover will be late.

Hill-climbing is easy, knowing what to hill-climb is hard. Making scores go up is expensive, but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today.

Independent evaluation of language models is under-appreciated in how hard of a …