The Hallucinations Leaderboard, an Open Effort to Measure Hallucinations in Large Language Models

In the rapidly evolving field of Natural Language Processing (NLP), Large Language Models (LLMs) have become central to AI's ability to understand and generate human language. However, a significant challenge that persists is their tendency to hallucinate — i.e., producing content that may not align with real-world facts or the user's input. With the constant release of new open-source models, identifying the most reliable ones, particularly in terms of their propensity to generate hallucinated content, becomes crucial.

The Hallucinations Leaderboard aims to address this problem: it is a comprehensive platform that evaluates a wide array of LLMs against benchmarks specifically designed to assess hallucination-related issues via in-context learning.

UPDATE -- We released a paper on this project; you can find it in arxiv: The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models. Here's also the Hugging Face paper page for community discussions.

The Hallucinations Leaderboard is an open and ongoing project: if you have any ideas, comments, or feedback, or if you would like to contribute to this project (e.g., by modifying the current tasks, proposing new tasks, or providing computational resources) please reach out!

What are Hallucinations?

Hallucinations in LLMs can be broadly categorised into factuality and faithfulness hallucinations (reference).

Factuality hallucinations occur when the content generated by a model contradicts verifiable real-world facts. For instance, a model might erroneously state that Charles Lindbergh was the first person to walk on the moon in 1951, despite it being a well-known fact that Neil Armstrong earned this distinction in 1969 during the Apollo 11 mission. This type of hallucination can disseminate misinformation and undermine the model's credibility.

The Hallucinations Leaderboard

A glance at the results so far

Let’s look at the results a bit more in detail.