Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

In the rapidly evolving landscape of large language models (LLMs), comprehensive and robust evaluation methodologies remain a critical challenge, particularly for low-resource languages. In this blog, we introduce AraGen, a generative tasks benchmark and leaderboard for Arabic LLMs, based on 3C3H, a new evaluation measure for NLG which we hope will inspire work for other languages as well.

The AraGen leaderboard makes three key contributions:

• 3C3H Measure: The 3C3H measure scores a model's response and is central to this framework. It is a holistic approach assessing model responses across multiple dimensions -Correctness, Completeness, Conciseness, Helpfulness, Honesty, and Harmlessness- based on LLM-as-judge.

• Dynamic Evaluations: AraGen Leaderboard implements a dynamic evaluation strategy, which includes three-month blind testing cycles, where the datasets and the evaluation code remain private before being publicly released at the end of the cycle, and replaced by a new private benchmark.

• Arabic Evaluation Dataset: AraGen Benchmark offers a meticulously constructed evaluation dataset for Arabic LLM evaluation, combining multi-turn and single-turn scenarios, which tests the model capability across multiple domains and tasks.

We believe that AraGen addresses persistent issues of data contamination with its dynamic evaluation approach, preserving the benchmark's integrity. It also serves as the first application of a scalable, language-agnostic framework for a nuanced and fair model assessment, which represents an important effort in understanding LLM performance across diverse linguistic contexts and sets a new standard for comprehensive model benchmarking.

Summary

To address these limitations, we propose a new evaluation measure that aims to combine both approaches, offering a comprehensive mechanism to evaluate language models. It assesses two key aspects of model outputs:

More Detailed Scores