Codestral 25.01

Among all the innovations in AI over the past year, code generation has arguably been the most significant. Akin to how the assembly line streamlined manufacturing and the calculator transformed mathematics, coding models represent a significant step change in software development.

Mistral AI has been at the forefront of this change with Codestral, a state of the art (SOTA) coding model released earlier this year. Lightweight, fast, and proficient in over 80 programming languages, Codestral is optimized for low-latency, high-frequency usecases and supports tasks such as fill-in-the-middle (FIM), code correction and test generation. Codestral has been used by thousands of developers as a highly capable coding companion, regularly boosting productivity several times over. And today, Codestral is getting a big upgrade.

Codestral 25.01 features a more efficient architecture and an improved tokenizer than the original, generating and completing code about 2 times faster. The model is now the clear leader for coding in its weight class, and SOTA for FIM use cases across the board.

Benchmarks

We have benchmarked the new Codestral with the leading sub-100B parameter coding models that are widely considered to be best-in-class for FIM tasks.

Overview

		Python	SQL		Average on several languages
Model	Context length	HumanEval	MBPP	CruxEval	LiveCodeBench	RepoBench	Spider	CanItEdit	HumanEval (average)	HumanEvalFIM (average)
Codestral-2501	256k	86.6%	80.2%	55.5%	37.9%	38.0%	66.5%	50.5%	71.4%	85.9%
Codestral-2405 22B	32k	81.1%	78.2%	51.3%	31.5%	34.0%	63.5%	50.5%	65.6%	82.1%
Codellama 70B instruct	4k	67.1%	70.8%	47.3%	20.0%	11.4%	37.0%	29.5%	55.3%	-
DeepSeek Coder 33B instruct	16k	77.4%	80.2%	49.5%	27.0%	28.4%	60.0%	47.6%	65.1%	85.3%
DeepSeek Coder V2 lite	128k	83.5%	83.2%	49.7%	28.1%	20.0%	72.0%	41.0%	65.9%	84.1%