A guide to setting up your own Hugging Face leaderboard: an end-to-end example with Vectara's hallucination leaderboard

Hugging Face’s Open LLM Leaderboard (originally created by Ed Beeching and Lewis Tunstall, and maintained by Nathan Habib and Clémentine Fourrier) is well known for tracking the performance of open source LLMs, comparing their performance in a variety of tasks, such as TruthfulQA or HellaSwag.

This has been of tremendous value to the open-source community, as it provides a way for practitioners to keep track of the best open-source models.

In late 2023, at Vectara we introduced the Hughes Hallucination Evaluation Model (HHEM), an open-source model for measuring the extent to which an LLM hallucinates (generates text that is nonsensical or unfaithful to the provided source content). Covering both open source models like Llama 2 or Mistral 7B, as well as commercial models like OpenAI’s GPT-4, Anthropic Claude, or Google’s Gemini, this model highlighted the stark differences that currently exist between models in terms of their likelihood to hallucinate.

As we continue to add new models to HHEM, we were looking for an open-source solution to manage and update the HHEM leaderboard.

Quite recently, the Hugging Face leaderboard team released leaderboard templates (here and here). These are lightweight versions of the Open LLM Leaderboard itself, which are both open-source and simpler to use than the original code.

Today we’re happy to announce the release of the new HHEM leaderboard, powered by the HF leaderboard template.

Vectara’s Hughes Hallucination Evaluation Model (HHEM)

The Hughes Hallucination Evaluation Model (HHEM) Leaderboard is dedicated to assessing the frequency of hallucinations in document summaries generated by Large Language Models (LLMs) such as GPT-4, Google Gemini or Meta’s Llama 2. To use it you can follow the instructions here.

By doing an open-source release of this model, we at Vectara aim to democratize the evaluation of LLM hallucinations, driving awareness to the differences that exist in LLM performance in terms of propensity to hallucinate.