Welcome Mixtral - a SOTA Mixture of Experts on Hugging Face

Mixtral 8x7b is an exciting large language model released by Mistral today, which sets a new state-of-the-art for open-access models and outperforms GPT-3.5 across many benchmarks. We’re excited to support the launch with a comprehensive integration of Mixtral in the Hugging Face ecosystem 🔥!

Among the features and integrations being released today, we have:

• Models on the Hub, with their model cards and licenses (Apache 2.0)
• 🤗 Transformers integration
• Integration with Inference Endpoints
• Integration with Text Generation Inference for fast and efficient production-ready inference
• An example of fine-tuning Mixtral on a single GPU with 🤗 TRL.

Table of Contents

• What is Mixtral 8x7b
  • About the name
  • Prompt format
  • What we don't know
• Demo
• Inference
  • Using 🤗 Transformers
  • Using Text Generation Inference
• Fine-tuning with 🤗 TRL
• Quantizing Mixtral
  • Load Mixtral with 4-bit quantization
  • Load Mixtral with GPTQ
• Disclaimers and ongoing work
• Additional Resources
• Conclusion

What is Mixtral 8x7b?

Mixtral has a similar architecture to Mistral 7B, but comes with a twist: it’s actually 8 “expert” models in one, thanks to a technique called Mixture of Experts (MoE). For transformers models, the way this works is by replacing some Feed-Forward layers with a sparse MoE layer. A MoE layer contains a router network to select which experts process which tokens most efficiently. In the case of Mixtral, two experts are selected for each timestep, which allows the model to decode at the speed of a 12B parameter-dense model, despite containing 4x the number of effective parameters!

For more details on MoEs, see our accompanying blog post: hf.co/blog/moe

Mixtral release TL;DR;

• Release of base and Instruct versions
• Supports a context length of 32k tokens.
• Outperforms Llama 2 70B and matches or beats GPT3.5 on most benchmarks
• Speaks English, French, German, Spanish, and Italian.
• Good at coding, with 40.2% on HumanEval
• Commercially permissive with an Apache 2.0 license