ControlNet in 🧨 Diffusers

Ever since Stable Diffusion took the world by storm, people have been looking for ways to have more control over the results of the generation process. ControlNet provides a minimal interface allowing users to customize the generation process up to a great extent. With ControlNet, users can easily condition the generation with different spatial contexts such as a depth map, a segmentation map, a scribble, keypoints, and so on!

We can turn a cartoon drawing into a realistic photo with incredible coherence.

Realistic Lofi Girl

Or even use it as your interior designer.

Before	After

You can turn your sketch scribble into an artistic drawing.

Before	After

Also, make some of the famous logos coming to life.

Before	After

With ControlNet, the sky is the limit 🌠

In this blog post, we first introduce the StableDiffusionControlNetPipeline and then show how it can be applied for various control conditionings. Let’s get controlling!

ControlNet: TL;DR

ControlNet was introduced in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang and Maneesh Agrawala. It introduces a framework that allows for supporting various spatial contexts that can serve as additional conditionings to Diffusion models such as Stable Diffusion. The diffusers implementation is adapted from the original source code.

Training ControlNet is comprised of the following steps:

Pictorially, training a ControlNet looks like so:

The diagram is taken from here.

A sample from the training set for ControlNet-like training looks like this (additional conditioning is via edge maps):

Prompt	Original Image	Conditioning
"bird"

Similarly, if we were to condition ControlNet with semantic segmentation maps, a training sample would be like so:

Prompt	Original Image	Conditioning
"big house"

Every new type of conditioning requires training a new copy of ControlNet weights. The paper proposed 8 different conditioning models that are all supported in Diffusers!

Now it's yoga time!