Open R1: Update #3

Over the last few weeks, we have focused our efforts on reproducing the competitive programming (code reasoning) aspects of the DeepSeek-R1 recipe.

In this post, we are excited to share:

The construction of CodeForces-CoTs: a dataset of nearly 100k high-quality samples distilled from R1 to produce solutions in C++ and Python.
The IOI benchmark: a new benchmark of challenging problems from the 2024 International Olympiad in Informatics (IOI).
OlympicCoder: two fine-tuned 7B and 32B code models that outperform closed-source frontier models like Claude 3.7 Sonnet on IOI problems.

Here’s an overview of how the OlympicCoder models stack up against various instruction fine-tuned and reasoning models. We find that training models on CodeForces-CoTs produces top-tier performance, with OlympicCoder-32B outperforming all open-weight models we tested, including some that are over 100x larger 🤯.

Read on to learn how we built the dataset, benchmark and models!

Key links

CodeForces

Problems dataset: open-r1/codeforces
DeepSeek-R1 cots dataset: open-r1/codeforces-cots

International Olympiad in Informatics (IOI)

Problem statements dataset (IOI’2020 - IOI’2024): open-r1/ioi
Test cases: open-r1/ioi-test-cases
Official (ground truth) solutions: open-r1/ioi-sample-solutions
DeepSeek-R1 cots dataset (IOI’2020-IOI’2023): open-r1/ioi-cots
Evaluation data for 40+ leading models on IOI’2024: open-r1/ioi-2024-model-solutions
Code to run generations and evaluations

OlympicCoder

7B Model: open-r1/OlympicCoder-7B
32B Model: open-r1/OlympicCoder-32B

CodeForces-CoTs Dataset

CodeForces is one of the most popular websites among competitive programmers, hosting regular contests where participants must solve challenging algorithmic optimization problems. The challenging nature of these problems makes them an interesting dataset to improve and test models’ code reasoning capabilities.