Transformers backend integration in SGLang

Hugging Face transformers library is the standard for working with state-of-the-art models — from experimenting with cutting-edge research to fine-tuning on custom data. Its simplicity, flexibility, and expansive model zoo make it a powerful tool for rapid development.

But once you're ready to move from notebooks to production, inference performance becomes mission-critical. That’s where SGLang comes in.

Designed for high-throughput, low-latency inference, SGLang now offers seamless integration with transformers as a backend. This means you can pair the flexibility of transformers with the raw performance of SGLang.

Let’s dive into what this integration enables and how you can use it.

TL;DR

SGLang now supports Hugging Face transformers as a backend, letting you run any transformers-compatible model with high-performance inference out of the box.

import sglang as sgl

llm = sgl.Engine("meta-llama/Llama-3.2-1B-Instruct", impl="transformers")
print(llm.generate(["The capital of France is"], {"max_new_tokens": 20})[0])

No native support needed — SGLang automatically falls back to Transformers when needed, or you can set impl="transformers" explicitly.

Transformers and SGLang

Let’s walk through a simple text generation example with meta-llama/Llama-3.2-1B-Instruct to compare both approaches.

Transformers

transformers library is great for experimentation, small-scale tasks and training, but it's not optimized for high-volume or low-latency scenarios.

from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Llama-3.2-1B-Instruct")
generate_kwargs = {
    "top_p": 0.95,
    "top_k": 20,
    "temperature": 0.8,
    "max_new_tokens": 256
}
result = pipe("The future of AI is", **generate_kwargs)
print(result[0]["generated_text"])

SGLang

Or you can spin a server and send requests:

python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Instruct \
  --host 0.0.0.0 \
  --port 30000