Open R1: How to use OlympicCoder locally for coding

Everyone’s been using Claude and OpenAI as coding assistants for the last few years, but there’s less appeal if you look at the developments coming out of open source projects like Open R1. If we look at the evaluation on LiveCodeBench below, we can see that the 7B parameter variant outperforms Claude 3.7 Sonnet and GPT-4o. These models are the daily drivers of many engineers in applications like Cursor and VSCode.

Evals are great and all, but I want to get my hands dirty and feel the commits! This blog post focuses on how you can integrate these models in your IDE now. We will set up OlympicCoder 7B, the smaller of the two OlympicCoder variants, and we’ll use a quantized variant for optimum local inference. Here’s the stack we’re going to use:

• OlympicCoder 7B. The 4bit GGUF version from the LMStudio Community
• LM Studio: A tool that simplifies running AI models
• Visual Studio Code (VS Code)
• Continue a VS Code extension for local models

It’s important to say that we chose this stack purely for simplicity. You might want to experiment with the larger model and/or different GGUF files. Or even alternative inference engines like llama.cpp.

1. Install LM Studio

LM Studio is like a control panel for AI models. It integrates with the Hugging Face hub to pull models, helps you find the right GGUF file, and exposes an API that other applications can use to interact with the model.

In short, it lets you download and run them without any complicated setup.

1. Go to the LM Studio website: Open your web browser and go to https://lmstudio.ai/download.
2. Choose your operating system: Click the download button for your computer (Windows, Mac, or Linux).
3. Install LM Studio: Run the downloaded file and follow the instructions. It’s just like installing any other program.

2. Get OlympicCoder 7B

The GGUF files that we need are hosted on the hub. We can open the model from the hub in LMStudio, using the ‘Use this model’ button: