Welcome Falcon Mamba: The first strong attention-free 7B model

Falcon Mamba is a new model by Technology Innovation Institute (TII) in Abu Dhabi released under the TII Falcon Mamba 7B License 1.0. The model is open access and available within the Hugging Face ecosystem here for anyone to use for their research or application purposes.

In this blog, we will go through the design decisions behind the model, how the model is competitive with respect to other existing SoTA models, and how to use it within the Hugging Face ecosystem.

First general purpose large-scale pure Mamba model

Transformers, based on the attention mechanism, are the dominant architecture used in all the strongest large language models today. Yet, the attention mechanism is fundamentally limited in processing large sequences due to the increase in compute and memory costs with sequence length. Various alternative architectures, in particular State Space Language Models (SSLMs), tried to address the sequence scaling limitation but fell back in performance compared to SoTA transformers.

With Falcon Mamba, we demonstrate that sequence scaling limitation can indeed be overcome without loss in performance. Falcon Mamba is based on the original Mamba architecture, proposed in Mamba: Linear-Time Sequence Modeling with Selective State Spaces, with the addition of extra RMS normalization layers to ensure stable training at scale. This choice of architecture ensures that Falcon Mamba:

• can process sequences of arbitrary length without any increase in memory storage, in particular, fitting on a single A10 24GB GPU.
• takes a constant amount of time to generate a new token, regardless of the size of the context (see this section)

Model training

Evaluations

We evaluate our model on all benchmarks of the new leaderboard's version using the lm-evaluation-harness package and then normalize the evaluation results with Hugging Face score normalization.

Also, we evaluate our model on the benchmarks of the first version of the LLM Leaderboard using lighteval.