Transformers now runs llama.cpp quants

We're adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop's memory through the familiar transformers APIs. Pick a GGUF from the Hub, load it with from_pretrained, and start generating on your own machine.

Running AI models on your laptop has become much easier, and llama.cpp has been a big part of that. Its inference engine powers local AI tools such as Ollama, LM Studio, and Jan. Alongside projects like MLX, it has helped make local inference a practical option for everyday use.

A recent example of what local AI can feel like:

This is where we are right now. And i’m not gonna lie it feels pretty magical 🧙‍♀️

Qwen3.6 27B running inside of Pi coding agent via Llama.cpp on the MacBook Pro

For non-trivial tasks on the @huggingface codebases, this feels very, very close to hitting the latest Opus in Claude… pic.twitter.com/lsIxLoUneU

— Julien Chaumond (@julien_c) April 24, 2026

GGUF, developed by the llama.cpp team, is a widely used format for local inference. The team also shares quantized checkpoints under ggml-org on the Hub. Publishers such as Unsloth, LM Studio Community, and bartowski also provide ready-to-use GGUF checkpoints in a range of quantizations, so users can pick the version that fits their machine. GGUF models have been downloaded millions of times.

We want to make it easier to run these models locally with transformers, too. Compatibility is only useful if the model is pleasant to run. To bring performance close to llama.cpp, we're reusing its underlying ggml kernels through the kernels library, and reducing overhead in generate. Our initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture.

What is the GGUF file format?

Here's how quantization changes the file size of Unsloth's Qwen3.5-4B:

Load GGUF with transformers

To get started, you need:

pip install -U "git+https://github.com/huggingface/transformers.git" kernels

The benchmark script