Constitutional AI with Open LLMs

Since the launch of ChatGPT in 2022, we have seen tremendous progress in LLMs, ranging from the release of powerful pretrained models like Llama 2 and Mixtral, to the development of new alignment techniques like Direct Preference Optimization. However, deploying LLMs in consumer applications poses several challenges, including the need to add guardrails that prevent the model from generating undesirable responses. For example, if you are building an AI tutor for children, then you don’t want it to generate toxic answers or teach them to write scam emails!

To align these LLMs according to a set of values, researchers at Anthropic have proposed a technique called Constitutional AI (CAI), which asks the models to critique their outputs and self-improve according to a set of user-defined principles. This is exciting because the practitioners only need to define the principles instead of having to collect expensive human feedback to improve the model.

In this work, we present an end-to-end recipe for doing Constitutional AI with open models. We are also releasing a new tool called llm-swarm to leverage GPU Slurm clusters for scalable synthetic data generation.

Here are the various artifacts:

Let’s start by taking a look at how CAI works!

Constitutional AI: learn to self-align

Constitutional AI is this clever idea that we can ask helpful models to align themselves. Below is an illustration of the CAI training process:

To make it more concrete, here is an example of a conversation that shows how the self-critique works:

The process goes as follows:

Given this conversation, we can then build Constitutional AI datasets:

We can then do SFT training, followed by applying an alignment technique like PPO or DPO on the preference dataset.

Note that the self-critique process doesn’t work perfectly every time. As shown in the example below, it can fail to detect responses that conflict with the constitutional principles: