Arabic Leaderboards: Introducing Arabic Instruction Following, Updating AraGen, and More

At Inception, we have been working to enhance AI model evaluations within the Arabic language context. Previously, we introduced AraGen, one of the first generative Arabic leaderboards, serving as a benchmark for evaluating Arabic LLMs on generative tasks.

As part of our ongoing efforts, we are excited to share the following updates:

• Arabic-Leaderboards Space, launched in collaboration with Mohammed bin Zayed University of Artificial Intelligence (MBZUAI) to consolidate Arabic AI evaluations in one place. This platform currently supports AraGen-03-25 and Arabic Instruction Following, with plans to expand to leaderboards for Arabic AI models across various modalities.
• AraGen 03-25 release with improvements and updated benchmark.
• Instruction Following leaderboard, powered by Arabic IFEval Benchmark, the first publicly available benchmark for evaluating instruction-following capabilities in Arabic.

The following sections provide details about each of these updates.

Arabic-Leaderboards Space

Arabic-Leaderboards is a comprehensive and unified space for all Arabic evaluations and tasks. It is meant to serve as a central hub covering a broad spectrum of evaluations, for models across modalities. Currently, it has AraGen-03-25 and Arabic Instruction Following as live leaderboards. We plan to expand this space with more leaderboards and tasks for Arabic AI models across various modalities.

We invite interested contributors to reach out to us through the community tab or directly through email in order to discuss how to integrate their work/leaderboards as additional tabs into this space.

Latest Updates in AraGen Leaderboard

By sharing this benchmark and model responses, we aim to encourage the community to review them, identify any unexpected behaviors we may have missed and help us refine our evaluation framework.

AraGen-03-25 Release

• Question Answering: ~200 pairs
• Reasoning: 70 pairs
• Safety Questions: 40 pairs
• Orthographic and Grammatical Analysis: 30 pairs

More Detailed Scores Couple 1: System Prompt Effect (AraGen-03-25 SP1 vs. AraGen-03-25 SP2) Couple 2: Dataset and Prompt Update Effect (AraGen-12-24 SP1 (old) vs. AraGen-03-25 SP2 (new)) Instruction Following Leaderboard Sample