Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem

In the fast-evolving landscape of Large Language Models (LLMs), building an “ecosystem” has never been more important. This trend is evident in several major developments like Hugging Face's democratizing NLP and Upstage building a Generative AI ecosystem.

Inspired by these industry milestones, in September of 2023, at Upstage we initiated the Open Ko-LLM Leaderboard. Our goal was to quickly develop and introduce an evaluation ecosystem for Korean LLM data, aligning with the global movement towards open and collaborative AI development.

Our vision for the Open Ko-LLM Leaderboard is to cultivate a vibrant Korean LLM evaluation ecosystem, fostering transparency by enabling researchers to share their results and uncover hidden talents in the LLM field. In essence, we're striving to expand the playing field for Korean LLMs. To that end, we've developed an open platform where individuals can register their Korean LLM and engage in competitions with other models. Additionally, we aimed to create a leaderboard that captures the unique characteristics and culture of the Korean language. To achieve this goal, we made sure that our translated benchmark datasets such as Ko-MMLU reflect the distinctive attributes of Korean.

Leaderboard design choices: creating a new private test set for fairness

The Open Ko-LLM Leaderboard is characterized by its unique approach to benchmarking, particularly:

• its adoption of Korean language datasets, as opposed to the prevalent use of English-based benchmarks.
• the non-disclosure of test sets, contrasting with the open test sets of most leaderboards: we decided to construct entirely new datasets dedicated to Open Ko-LLM and maintain them as private, to prevent test set contamination and ensure a more equitable comparison framework.

Evaluation Tasks

The Open Ko-LLM Leaderboard adopts the following five types of evaluation methods: