Announcing NeurIPS 2025 E2LM Competition: Early Training Evaluation of Language Models

Join us in building benchmarks that capture early-stage reasoning & scientific knowledge in LLMs!

The development of Large Language Models (LLMs) typically begins with a series of ablation experiments, wherein various model architectures, data mixtures, and training hyperparameters are systematically evaluated. This phase is commonly referred to as the early stages of training. During this period, researchers primarily monitor two key metrics: the training loss curve and evaluation scores. However, existing evaluation benchmarks often fail to provide meaningful or discriminative signals during these initial stages where LLMs are trained on a few tokens ~200B tokens, making it challenging to derive conclusive insights from ongoing experiments.

In this competition, we want to build together new benchmarks to effectively capture relevant signals in early training stages of LLMs, specifically for scientific knowledge domain.

How to participate

The competition will be hosted on a dedicated Hugging Face organization - to register to the competition please follow this registration link 👉 https://e2lmc.github.io/registration. Participants will have to submit their solutions, which will be based on lm-evaluation-harness library through a HuggingFace Space. An active leaderboard will be maintained during the competition to track promising submissions. The size of the models make them easily runnable for everyone, on free-tier Google Colab GPUs. We also provide a comprehensive starting kit including several notebooks to get started with the competition.

Evaluation metrics

Score = α1 × Score_SQ + α2 × Score_RC + α3 × Score_CS

where, α_SQ, α_RC and α_CS are weighting coefficients that reflect the relative importance of each criterion. We set the weights as α1 = 0.5, α2 = 0.1 and α3 = 0.4, thereby placing greater emphasis on signal quality and compliance to scientific knowledge, which we consider the most important metrics in evaluating submissions.