Accelerate 1.0.0

What is Accelerate today?

3.5 years ago, Accelerate was a simple framework aimed at making training on multi-GPU and TPU systems easier by having a low-level abstraction that simplified a raw PyTorch training loop:

Since then, Accelerate has expanded into a multi-faceted library aimed at tackling many common problems with large-scale training and large models in an age where 405 billion parameters (Llama) are the new language model size. This involves:

• A flexible low-level training API, allowing for training on six different hardware accelerators (CPU, GPU, TPU, XPU, NPU, MLU) while maintaining 99% of your original training loop
• An easy-to-use command-line interface aimed at configuring and running scripts across different hardware configurations
• The birthplace of Big Model Inference or device_map="auto", allowing users to not only perform inference on LLMs with multi-devices but now also aiding in training LLMs on small compute through techniques like parameter-efficient fine-tuning (PEFT)

These three facets have allowed Accelerate to become the foundation of nearly every package at Hugging Face, including transformers, diffusers, peft, trl, and more!

As the package has been stable for nearly a year, we're excited to announce that, as of today, we've published the first release candidates for Accelerate 1.0.0!

This blog will detail:

1. Why did we decide to do 1.0?
2. What is the future for Accelerate, and where do we see PyTorch as a whole going?
3. What are the breaking changes and deprecations that occurred, and how can you migrate over easily?

Why 1.0?

The plans to release 1.0.0 have been in the works for over a year. The API has been roughly at a point where we wanted, centering on the Accelerator side, simplifying much of the configuration and making it more extensible. However, we knew there were a few missing pieces before we could call the "base" of Accelerate "feature complete":

The future of Accelerate

How to try it out

• pip:

pip install --pre accelerate

• Docker: