Introducing the Open Chain of Thought Leaderboard

Chain-of-thought prompting is emerging as a powerful and effective design pattern for LLM-based apps and agents. The basic idea of chain-of-thought prompting is to let a model generate a step-by-step solution (“reasoning trace”) before answering a question or taking a decision. With the Open CoT Leaderboard we’re tracking LLMs’ ability to generate effective chain-of-thought traces for challenging reasoning tasks.

Unlike most performance based leaderboards, we’re not scoring the absolute accuracy a model achieves on a given task, but the difference between the accuracy with and without chain-of-thought prompting:

accuracy gain Δ = accuracy with CoT – accuracy w/o CoT.

This allows us to truly inspect the impact that chain-of-thought has on model accuracy.

Note: without CoT prompting, we use the loglikelihood accuracy to score the model on multiple choice evaluation.

What’s the motivation behind such a leaderboard for chain-of-thought?

Chain-of-thought prompting is a universally applicable prompting strategy that may improve explainability and accuracy of LLM-based apps and agents (see, e.g., this collection for recent research and implementations)). With frameworks like Langchain or LMQL, it’s straightforward to insert sophisticated reasoning chains in your apps. But even if you’ve never heard about chain-of-thought before, you may have noticed, while using a ChatBot, that it tends to proceed step by step before answering your query. So, a systematic, up-to-date comparison of LLMs’ ability to generate effective chain-of-thought traces may inform the decisions of builders and users when choosing a model.

Which tasks are used?

The Open CoT Leaderboard evaluates LLMs’ ability to generate effective chain-of-thought reasoning traces for the following tasks:

• LogiQA (new translation of original version, and version 2.0 with new examples)
• LSAT dataset (including subsets on analytical reasoning, logical reasoning, and reading comprehension)

We’ve chosen these tasks because they