Accelerating SD Turbo and SDXL Turbo Inference with ONNX Runtime and Olive

Introduction

SD Turbo and SDXL Turbo are two fast generative text-to-image models capable of generating viable images in as little as one step, a significant improvement over the 30+ steps often required with previous Stable Diffusion models. SD Turbo is a distilled version of Stable Diffusion 2.1, and SDXL Turbo is a distilled version of SDXL 1.0. We’ve previously shown how to accelerate Stable Diffusion inference with ONNX Runtime. Not only does ONNX Runtime provide performance benefits when used with SD Turbo and SDXL Turbo, but it also makes the models accessible in languages other than Python, like C# and Java.

Performance gains

In this post, we will introduce optimizations in the ONNX Runtime CUDA and TensorRT execution providers that speed up inference of SD Turbo and SDXL Turbo on NVIDIA GPUs significantly.

ONNX Runtime outperformed PyTorch for all (batch size, number of steps) combinations tested, with throughput gains as high as 229% for the SDXL Turbo model and 120% for the SD Turbo model. ONNX Runtime CUDA has particularly good performance for dynamic shape but demonstrates a marked improvement over PyTorch for static shape as well.

How to run SD Turbo and SDXL Turbo

To accelerate inference with the ONNX Runtime CUDA execution provider, access our optimized versions of SD Turbo and SDXL Turbo on Hugging Face.

The models are generated by Olive, an easy-to-use model optimization tool that is hardware aware. Note that fp16 VAE must be enabled through the command line for best performance, as shown in the optimized versions shared. For instructions on how to run the SD and SDXL pipelines with the ONNX files hosted on Hugging Face, see the SD Turbo usage example and the SDXL Turbo usage example.

To accelerate inference with the ONNX Runtime TensorRT execution provider instead, follow the instructions found here.

The following is an example of image generation with the SDXL Turbo model guided by a text prompt: