Making LLMs lighter with AutoGPTQ and transformers

Large language models have demonstrated remarkable capabilities in understanding and generating human-like text, revolutionizing applications across various domains. However, the demands they place on consumer hardware for training and deployment have become increasingly challenging to meet.

🤗 Hugging Face's core mission is to democratize good machine learning, and this includes making large models as accessible as possible for everyone. In the same spirit as our bitsandbytes collaboration, we have just integrated the AutoGPTQ library in Transformers, making it possible for users to quantize and run models in 8, 4, 3, or even 2-bit precision using the GPTQ algorithm (Frantar et al. 2023). There is negligible accuracy degradation with 4-bit quantization, with inference speed comparable to the fp16 baseline for small batch sizes. Note that GPTQ method slightly differs from post-training quantization methods proposed by bitsandbytes as it requires to pass a calibration dataset.

This integration is available both for Nvidia GPUs, and RoCm-powered AMD GPUs.

Table of contents

• Resources
• A gentle summary of the GPTQ paper
• AutoGPTQ library – the one-stop library for efficiently leveraging GPTQ for LLMs
• Native support of GPTQ models in 🤗 Transformers
• Quantizing models with the Optimum library
• Running GPTQ models through Text-Generation-Inference
• Fine-tune quantized models with PEFT
• Room for improvement
  • Supported models
• Conclusion and final words
• Acknowledgements

Resources

This blogpost and release come with several resources to get started with GPTQ quantization:

• Original Paper
• Basic usage Google Colab notebook - This notebook shows how to quantize your transformers model with GPTQ method, how to do inference, and how to do fine-tuning with the quantized model.
• Transformers integration documentation
• Optimum integration documentation
• The Bloke repositories with compatible GPTQ models.

A gentle summary of the GPTQ paper

Quantization methods usually belong to one of two categories: