Mixture of Experts Explained

There is a second iteration (Feb 2026) of the blog post where we cover how the transformers library has built around MoEs to make them "first class citizens" of the library and the Hub. Here is the link to the post: Mixture of Experts (MoEs) in Transformers

With the release of Mixtral 8x7B (announcement, model card), a class of transformer has become the hottest topic in the open AI community: Mixture of Experts, or MoEs for short. In this blog post, we take a look at the building blocks of MoEs, how they’re trained, and the tradeoffs to consider when serving them for inference.

Let’s dive in!

Table of Contents

• What is a Mixture of Experts?
• A Brief History of MoEs
• What is Sparsity?
• Load Balancing tokens for MoEs
• MoEs and Transformers
• Switch Transformers
• Stabilizing training with router Z-loss
• What does an expert learn?
• How does scaling the number of experts impact pretraining?
• Fine-tuning MoEs
• When to use sparse MoEs vs dense models?
• Making MoEs go brrr
  • Expert Parallelism
  • Capacity Factor and Communication costs
  • Serving Techniques
  • Efficient Training
• Open Source MoEs
• Exciting directions of work
• Some resources

TL;DR

MoEs:

• Are pretrained much faster vs. dense models
• Have faster inference compared to a model with the same number of parameters
• Require high VRAM as all experts are loaded in memory
• Face many challenges in fine-tuning, but recent work with MoE instruction-tuning is promising

Let’s dive in!

What is a Mixture of Experts (MoE)?

The scale of a model is one of the most important axes for better model quality. Given a fixed computing budget, training a larger model for fewer steps is better than training a smaller model for more steps.

Mixture of Experts enable models to be pretrained with far less compute, which means you can dramatically scale up the model or dataset size with the same compute budget as a dense model. In particular, a MoE model should achieve the same quality as its dense counterpart much faster during pretraining.