📚 3LM: A Benchmark for Arabic LLMs in STEM and Code

Why 3LM?

Arabic Large Language Models (LLMs) have seen notable progress in recent years, yet existing benchmarks fall short when it comes to evaluating performance in high-value technical domains. Most evaluations to date have focused on general-purpose tasks like summarization, sentiment analysis, or generic question answering. However, scientific reasoning and programming are essential for a broad range of real-world applications, from education to technical problem-solving.

To address this gap, we introduce 3LM (علم), a multi-component benchmark tailored to evaluate Arabic LLMs on STEM (Science, Technology, Engineering, and Mathematics) subjects and code generation. 3LM is the first benchmark of its kind, designed specifically to test Arabic models in structured reasoning and formal logic which are domains traditionally underrepresented in Arabic NLP.

What’s in the Benchmark?

3LM is made up of three datasets, each targeting a specific evaluation axis: real-world multiple-choice STEM questions (MCQs), synthetic high-difficulty STEM questions, and translated code generation tasks.

1. Native STEM

The Native STEM benchmark consists of 865 MCQs extracted from authentic Arabic educational content, including textbooks, worksheets, and exam banks for grades 8 through 12. Questions span five core subjects: Physics, Chemistry, Biology, Mathematics, and Geography.

Each question is annotated with metadata including domain and difficulty (on a 1–10 scale). The data was sourced using a pipeline that combined OCR (including LaTeX math parsing via Pix2Tex), LLM-assisted question-answer extraction, and manual review. This dataset provides a realistic testbed for evaluating factual and conceptual understanding in Arabic models using real educational materials.

2. Synthetic STEM

3. Arabic Code Benchmarks

Building the Benchmark

Each dataset in 3LM went through a multi-stage development process to ensure data quality, fairness, and representativeness.