LoRA training scripts of the world, unite!

A community derived guide to some of the SOTA practices for SD-XL Dreambooth LoRA fine tuning

TL;DR

We combined the Pivotal Tuning technique used on Replicate's SDXL Cog trainer with the Prodigy optimizer used in the Kohya trainer (plus a bunch of other optimizations) to achieve very good results on training Dreambooth LoRAs for SDXL. Check out the training script on diffusers🧨. Try it out on Colab.

If you want to skip the technical talk, you can use all the techniques in this blog and train on Hugging Face Spaces with a simple UI and curated parameters (that you can meddle with).

Overview

Stable Diffusion XL (SDXL) models fine-tuned with LoRA dreambooth achieve incredible results at capturing new concepts using only a handful of images, while simultaneously maintaining the aesthetic and image quality of SDXL and requiring relatively little compute and resources. Check out some of the awesome SDXL LoRAs here.
In this blog, we'll review some of the popular practices and techniques to make your LoRA finetunes go brrr, and show how you can run or train yours now with diffusers!

Recap: LoRA (Low-Rank Adaptation) is a fine-tuning technique for Stable Diffusion models that makes slight adjustments to the crucial cross-attention layers where images and prompts intersect. It achieves quality on par with full fine-tuned models while being much faster and requiring less compute. To learn more on how LoRAs work, please see our previous post - Using LoRA for Efficient Stable Diffusion Fine-Tuning.

Contents:

1. Techniques/tricks
   1. Pivotal tuning
   2. Adaptive optimizers
   3. Recommended practices - Text encoder learning rate, custom captions, dataset repeats, min snr gamma, training set creation
2. Experiments Settings and Results
3. Inference
   1. Diffusers inference
   2. Automatic1111/ComfyUI inference

Pivotal Tuning

Training

In our new training script, you can do textual inversion training by providing the following arguments

Adaptive Optimizers

Training

--optimizer="prodigy"

--learning_rate=1.0

Training

Training