Ettin Suite: SoTA Paired Encoders and Decoders

TL;DR

What would happen if you took the ModernBERT recipe and applied it to a decoder-only model? Turns out, a state-of-the-art decoder language model that beats Llama 3.2 1B and SmolLM2!

We introduce a new open-data training recipe to reproduce the encoder-only ModernBERT model (and actually beat it!). We then apply the exact same recipe to decoder-only models. For the first time, we have two state-of-the-art models trained in the same setup but with two different training objectives: masked language modeling (MLM), and causal language modeling (CLM).

This blog post introduces Ettin, the first suite of SoTA paired encoder-only and decoder-only models (17M-1B params) trained with identical data (2T tokens), architecture, and training recipes. Ettin enables true apples-to-apples comparisons between architectures and delivers state-of-the-art performance for open-data models in both categories. We then further explore whether it is possible to get a competitive encoder starting from the decoder and vice-versa.

If you are interested in trying out the models, some boilerplates are available at the end of this blogpost!

Encoders vs Decoders: The Architecture Divide

The LLM community has largely converged on decoder-only models like GPT, Llama, and Qwen. Their generative capabilities are impressive, but this focus is detracting attention from other categories, such as encoder-only models like BERT.

However, encoder BERT-like models remain the workhorses of production systems for classification, retrieval, and embedding tasks. They're faster, more memory-efficient, and often more accurate for discriminative tasks. The key difference lies in their attention patterns:

• Encoder models use bidirectional attention, allowing each token to "see" all other tokens in the sequence (fully visible)
• Decoder models use causal attention, where tokens can only "see" previous tokens to enable autoregressive generation

Click to see how to finetune this into a dense embedding model using Sentence Transformers Click to see how to finetune this into a multi-vector embedding model with PyLate Click to see how to finetune this into a sparse retrieval model using Sentence Transformers Click to see how to finetune this into a reranker model using Sentence Transformers Click to expand decoder training code