🇨🇿 BenCzechMark - Can your LLM Understand Czech?

The 🇨🇿 BenCzechMark is the first and most comprehensive evaluation suite for assessing the abilities of Large Language Models (LLMs) in the Czech language. It aims to test how well LLMs can:

• Reason and perform complex tasks in Czech.
• Generate and verify grammatically and semantically correct Czech.
• Extract information and store knowledge by answering questions about Czech culture and Czech-related facts.
• Do what language models were originally trained for—estimate the probability of Czech texts.

To achieve this, we've sourced 50 tasks spanning 9 categories, with 90% of tasks having native, non-translated content.

In this blog, we introduce both the evaluation suite itself and the BenCzechMark leaderboard, featuring over 25 open-source models of various sizes!

📋 Tasks and Categories

The 🇨🇿 BenCzechMark (in it’s current version) is divided into 9 categories to comprehensively assess LLM abilities. For each task,

The translated portion of the dataset (10% of the total) was mostly translated via CUBBITT LINDAT Translation, except for CsFever, where the authors used DeepL for translation.

This is the complete list of categories, alongside the datasets and metrics used:

⚔️ Model Duels and Average Score

Since we use different metrics for the tasks, simply averaging wouldn't work due to varying scales. Instead, we've introduced a novel way to determine a final score: we let the models fight!

For every task and metric, we compute a test for statistical significance at α=0.05. This means the probability that the performance of model A equals that of model B is estimated to be less than 0.05. We use the following tests, each with varying statistical power:

• ACC and EM: one-tailed paired t-test,
• AUROC: Bayesian test inspired by Goutte et al., 2005,
• Ppl: bootstrapping.

We then compute a model's duel win score (DWS) - the proportion of duels won against all other models on that task. Finally, we calculate aggregate scores as follows: