The Falcon has landed in the Hugging Face ecosystem

Falcon is a new family of state-of-the-art language models created by the Technology Innovation Institute in Abu Dhabi, and released under the Apache 2.0 license. Notably, Falcon-40B is the first “truly open” model with capabilities rivaling many current closed-source models. This is fantastic news for practitioners, enthusiasts, and industry, as it opens the door for many exciting use cases.

Note: Few months after this release, the Falcon team released a larger model of 180 billion parameters.

September 2023 Update: Falcon 180B has just been released! It's currently the largest openly available model, and rivals proprietary models like PaLM-2.

In this blog, we will be taking a deep dive into the Falcon models: first discussing what makes them unique and then showcasing how easy it is to build on top of them (inference, quantization, finetuning, and more) with tools from the Hugging Face ecosystem.

Table of Contents

• The Falcon models
• Demo
• Inference
• Evaluation
• Fine-tuning with PEFT
• Conclusion

The Falcon models

The Falcon family is composed of two base models: Falcon-40B and its little brother Falcon-7B. The 40B parameter model was at the top of the Open LLM Leaderboard at the time of its release, while the 7B model was the best in its weight class.

Note: the performance scores shown in the table below have been updated to account for the new methodology introduced in November 2023, which added new benchmarks. More details in this post.

Falcon-40B requires ~90GB of GPU memory — that’s a lot, but still less than LLaMA-65B, which Falcon outperforms. On the other hand, Falcon-7B only needs ~15GB, making inference and finetuning accessible even on consumer hardware. (Later in this blog, we will discuss how we can leverage quantization to make Falcon-40B accessible even on cheaper GPUs!)

Multi-Query Attention shares keys and value embeddings across attention heads. Courtesy Harm de Vries.

Demo

Video: Falcon 7B Instruct running on an M1 MacBook Pro with Core ML.

Inference

And then, you'd run text generation using code like the following: