2023, year of open LLMs

2023 has seen a surge of public interest in Large Language Models (LLMs), and now that most people have an idea of what they are and can do, the public debates around open versus closed source have reached a wide audience as well. At Hugging Face, we follow open models with great interest, as they allow research to be reproducible, empower the community to participate in the development of AI models, permit the easier scrutiny of model biases and limitations, and lower the overall carbon impact of our field by favoring checkpoint reuse (among many other benefits).

So let's do a retrospective of the year in open LLMs!

To keep this document manageable in length, we won't look at code models.

🍜 Recipe for a pretrained Large Language Model

First, how do you get a Large Language Model? (Feel free to skim this section if you already know!)

The model architecture (its code) describes its specific implementation and mathematical shape: it is a list of all its parameters, as well as how they interact with inputs. At the moment, most highly performing LLMs are variations on the "decoder-only" Transformer architecture (more details in the original transformers paper).

The training dataset contains all examples and documents on which the model is trained (aka the parameters are learned), therefore, the specific patterns learned. Most of the time, these documents contain text, either in natural language (ex: French, English, Chinese), a programming language (ex: Python, C), or any kind of structured data expressible as text (ex: tables in markdown or latex, equations, ...).

Training hyperparameters then define how the model is trained. How much should the parameters change to fit each new example? How fast should the model be updated?

🗝️ 2022, from a race for size to a race for data

What open models were available to the community before 2023?

This paradigm shift, while probably already known in closed labs took the open science community by storm.