Llama 3.1 - 405B, 70B & 8B with multilinguality and long context

Llama 3.1 is out! Today we welcome the next iteration of the Llama family to Hugging Face. We are excited to collaborate with Meta to ensure the best integration in the Hugging Face ecosystem. Eight open-weight models (3 base models and 5 fine-tuned ones) are available on the Hub.

Llama 3.1 comes in three sizes: 8B for efficient deployment and development on consumer-size GPU, 70B for large-scale AI native applications, and 405B for synthetic data, LLM as a Judge or distillation. All three come in base and instruction-tuned variants.

In addition to the six generative models, Meta released two new models: Llama Guard 3 and Prompt Guard. Prompt Guard is a small classifier that detects prompt injections and jailbreaks. Llama Guard 3 is a safeguard model that can classify LLM inputs and generations.

Among the features and integrations being released, we have:

• Models on the Hub
• Hugging Face Transformers and TGI integration
• Hugging Chat integration for Meta Llama 3.1 405B Instruct
• Inference & Deployment Integration with Inference Endpoints, Google Cloud, Amazon SageMaker & DELL Enterprise Hub
• Quantization for FP8, AWQ and GPTQ for easier inference
• Fine-tuning Llama 3.1 8B on a single GPU with 🤗 TRL
• Generate synthetic data using Llama 3.1 70B and 405B with Distilabel

Table of contents

• What’s new with Llama 3.1?
• How much memory does Llama 3.1 need?
  • Inference Memory Requirements
  • Training Memory Requirements
• Llama 3.1 evaluation
• Using Hugging Face Transformers
• How to prompt Llama 3.1
  • Built-in Tool calling
• Custom Tool calling
• Demo
• Llama 3.1 405B quantization with FP8, AWQ, and GPTQ
• Inference Integrations
  • Hugging Face Inference API
  • Hugging Face Inference Endpoints
• Hugging Face Partner Integrations
• Fine-tuning with Hugging Face TRL
• Synthetic data generation with distilabel
• Additional Resources
• Acknowledgments

What’s new with Llama 3.1?

Why is Llama 3.1 so exciting? On top of the features the predecessor offers, Llama 3.1 has some key new features:

Let’s dive into these!

Built-in tool calling example Custom Tool Calling JSON Functions Fine-Tuning Example with Hugging Face TRL