Diffusers welcomes Stable Diffusion 3.5 Large

Stable Diffusion 3.5 is the improved variant of its predecessor, Stable Diffusion 3. As of today, the models are available on the Hugging Face Hub and can be used with 🧨 Diffusers.

The release comes with two checkpoints:

• A large (8B) model
• A large (8B) timestep-distilled model enabling few-step inference

In this post, we will focus on how to use Stable Diffusion 3.5 (SD3.5) with Diffusers, covering both inference and training.

Table Of Contents

• Architectural changes
• Using SD3.5 with Diffusers
• Performing inference with quantization
• Training LoRAs with quantization
• Using single-file loading
• Important links

Architectural changes

The transformer architecture of SD3.5 (large) is very similar to SD3 (medium), with the following changes:

• QK normalization: For training large transformer models, QK normalization has now become a standard, and SD3.5 Large is no exception.
• Dual attention layers: Instead of using single attention layers for each stream of modality in the MMDiT blocks, SD3.5 uses double attention layers.

The rest of the details in terms of the text encoders, VAE, and noise scheduler stay exactly the same as in SD3 Medium. For more on SD3, we recommend checking out the original paper.

Using SD3.5 with Diffusers

Make sure you install the latest version of diffusers:

pip install -U diffusers

As the model is gated, before using it with diffusers, you first need to go to the Stable Diffusion 3.5 Large Hugging Face page, fill in the form and accept the gate. Once you are in, you need to log in so that your system knows you’ve accepted the gate. Use the command below to log in:

huggingface-cli login

The following snippet will download the 8B parameter version of SD3.5 in torch.bfloat16 precision. This is the format used in the original checkpoint published by Stability AI, and is the recommended way to run inference.

The release also comes with a “timestep-distilled” model that eliminates classifier-free guidance and lets us generate images in fewer steps (typically in 4-8 steps).