Mixture of Experts (MoEs) in Transformers

Introduction

Over the past few years, scaling dense language models has driven most progress in LLMs. From early models like the original ULMFiT (~30M parameters) or GPT-2 (1.5B parameters, which at the time was considered "too dangerous to release" 🧌), and eventually to today’s hundred-billion–parameter systems, the recipe was simple:

More data + more parameters gives better performance.

Scaling laws reinforced this trend, but dense scaling has practical limits:

Training becomes increasingly expensive.
Inference latency grows.
Deployment requires significant memory and hardware.

This is where Mixture of Experts (MoEs) enter the picture.

If you're already familiar with MoEs and want to jump straight into the engineering work done in transformers, you can head directly to Transformers and MoEs.

From Dense to Sparse: What Are MoEs?

A Mixture of Experts model keeps the Transformer backbone, but replaces certain dense feed-forward layers with a set of experts. An “expert” is not a topic-specialized module (e.g., "math expert", "code expert"). It is simply a learnable sub-network. For each token, a router selects a small subset of experts to process it.


Figure 1: Expert 1 among 4 experts is activated (Source: Maarten Grootendorst)

Different tokens activate different experts, based on their hidden representations.

Model capacity depends on total parameters, but inference speed depends on active parameters.

This is the key idea.

Your browser does not support the video tag.

This super fast speed confirms the model works approximately as a 3.6B parameter one, but it has the same capacity (or quality) as a 21B parameter model.

(Note: speed would be even faster if we used kernels for the native mxfp4 quantization the model uses).

MoEs are attractive for these reasons:

Transformers and MoEs

Most tooling in the ecosystem, including model loading, device placement, quantization, and backend execution was originally designed for dense models. MoEs challenge these assumptions.