The Transformers Library: standardizing model definitions

TLDR: Going forward, we're aiming for Transformers to be the pivot across frameworks: if a model architecture is supported by transformers, you can expect it to be supported in the rest of the ecosystem.

Transformers was created in 2019, shortly following the release of the BERT Transformer model. Since then, we've continuously aimed to add state-of-the-art architectures, initially focused on NLP, then growing to Audio and computer vision. Today, transformers is the default library for LLMs and VLMs in the Python ecosystem.

Transformers now supports 300+ model architectures, with an average of ~3 new architectures added every week. We have aimed for these architectures to be released in a timely manner; having day-0 support for the most sought-after architectures (Llamas, Qwens, GLMs, etc.).

A model-definition library

Over time, Transformers has become a central component in the ML ecosystem, becoming one of the most complete toolkits in terms of model diversity; it's integrated in all popular training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP, PyTorch-Lightning, TRL, Nanotron, etc.

Recently, we've been working hand in hand with the most popular inference engines (vLLM, SGLang, TGI, ...) for them to use transformers as a backend. The value added is significant: as soon as a model is added to transformers, it becomes available in these inference engines, while taking advantage of the strengths each engine provides: inference optimizations, specialized kernels, dynamic batching, etc.

As an example, here is how you would work with the transformers backend in vLLM:

from vllm import LLM

llm = LLM(model="new-transformers-model", model_impl="transformers")

That's all it takes for a new model to enjoy super-fast and production-grade serving with vLLM!

The same is true for MLX, where the transformers' safetensors files are directly compatible with MLX's models.