Introducing the Open Arabic LLM Leaderboard

The Open Arabic LLM Leaderboard (OALL) is designed to address the growing need for specialized benchmarks in the Arabic language processing domain. As the field of Natural Language Processing (NLP) progresses, the focus often remains heavily skewed towards English, leaving a significant gap in resources for other languages. The OALL aims to balance this by providing a platform specifically for evaluating and comparing the performance of Arabic Large Language Models (LLMs), thus promoting research and development in Arabic NLP.

This initiative is particularly significant given that it directly serves over 380 million Arabic speakers worldwide. By enhancing the ability to accurately evaluate and improve Arabic LLMs, we hope the OALL will play a crucial role in developing models and applications that are finely tuned to the nuances of the Arabic language, culture and heritage.

Benchmarks, Metrics & Technical setup

Benchmark Datasets

The Open Arabic LLM Leaderboard (OALL) utilizes an extensive and diverse collection of robust datasets to ensure comprehensive model evaluation.

• AlGhafa benchmark: created by the TII LLM team with the goal of evaluating models on a range of abilities including reading comprehension, sentiment analysis, and question answering. It was initially introduced with 11 native Arabic datasets and was later extended to include an additional 11 datasets that are translations of other widely adopted benchmarks within the English NLP community.
• ACVA and AceGPT benchmarks: feature 58 datasets from the paper "AceGPT, Localizing Large Language Models in Arabic", and translated versions of the MMLU and EXAMS benchmarks to broaden the evaluation spectrum and cover a comprehensive range of linguistic tasks. These benchmarks are meticulously curated and feature various subsets that precisely capture the complexities and subtleties of the Arabic language.

Evaluation Metrics

Technical setup

The technical setup for the Open Arabic LLM Leaderboard (OALL) uses: