Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

Introduction

Check out also our official blogpost

Today, we are proud to introduce the Falcon-H1 series, a collection of six open-source models ranging from 0.5B to 34B parameters, each available in both base and instruction-tuned variants. At the core of these models lies a hybrid architecture that combines the strengths of the classical Transformer-based attention mechanism with the State Space Model (SSM), known for its superior long-context memory and computational efficiency. This architectural innovation is further enhanced by fundamental advancements in training dynamics and data utilization, enabling Falcon-H1 models to deliver uncompromised performance that rivals the top Transformer-based models across all covered size tiers.

In this release, we feature six open-weight models: 0.5B, 1.5B, 1.5B-Deep, 3B, 7B, and 34B, along with their instruct versions. All our open-source models are with a permissive license based on Apache 2.0.

Model Size	Base Model	Instruct Model
0.5B	🤗 Falcon-H1-0.5B-Base	🤗 Falcon-H1-0.5B-Instruct
1.5B	🤗 Falcon-H1-1.5B-Base	🤗 Falcon-H1-1.5B-Instruct
1.5B-Deep	🤗 Falcon-H1-1.5B-Deep-Base	🤗 Falcon-H1-1.5B-Deep-Instruct
3B	🤗 Falcon-H1-3B-Base	🤗 Falcon-H1-3B-Instruct
7B	🤗 Falcon-H1-7B-Base	🤗 Falcon-H1-7B-Instruct
34B	🤗 Falcon-H1-34B-Base	🤗 Falcon-H1-34B-Instruct

Key Features of Falcon-H1

Main Principles behind building Falcon-H1

Given this uncertainty, we conducted an extensive experimentation phase, systematically revisiting nearly every aspect of model design and training methodology before launching our final training runs. While we will provide comprehensive details in our upcoming technical report, we'd like to share the key insights that shaped the Falcon-H1 models.