Welcome to the Falcon 3 Family of Open Models!

We introduce Falcon3, a family of decoder-only large language models under 10 billion parameters, developed by Technology Innovation Institute (TII) in Abu Dhabi. By pushing the boundaries of performance and training efficiency, this release reflects our ongoing commitment to advancing open and accessible large foundation models.

Falcon3 represents a natural evolution from previous releases, emphasizing expanding the models' science, math, and code capabilities.

This iteration includes five base models:

1. Falcon3-1B-Base
2. Falcon3-3B-Base
3. Falcon3-Mamba-7B-Base
4. Falcon3-7B-Base
5. Falcon3-10B-Base

In developing these models, we incorporated several key innovations aimed at improving the models' performances while reducing training costs:

Key Highlights

Falcon3 featured the limits within the small and medium scales of large language models by demonstrating high performance on common benchmarks:

Enhanced Capabilities

We evaluated models with our internal evaluation pipeline (based on lm-evaluation-harness) and we report raw scores. Our evaluations highlight key areas where the Falcon3 family of models excel, reflecting the emphasis on enhancing performance in scientific domains, reasoning, and general knowledge capabilities:

Models' Specs and Benchmark Results

Detailed specifications of the Falcon3 family of models are summarized in the following table. The architecture of Falcon3-7B-Base is characterized by a head dimension of 256 which yields high throughput when using FlashAttention-3 as it is optimized for this dimension. These decoder-only models span 18 to 40 layers for the transformer-based ones, and 64 layers for the mamba one, all models share the SwiGLU activation function, with vocabulary size of 131K tokens (65Kfor Mamba-7B). The Falcon3-7B-Base is trained on the largest amount of data ensuring comprehensive coverage of concepts and knowledge, the other variants require way less data.