Chat Templates: An End to the Silent Performance Killer

A spectre is haunting chat models - the spectre of incorrect formatting!

tl;dr

Chat models have been trained with very different formats for converting conversations into a single tokenizable string. Using a format different from the format a model was trained with will usually cause severe, silent performance degradation, so matching the format used during training is extremely important! Hugging Face tokenizers now have a chat_template attribute that can be used to save the chat format the model was trained with. This attribute contains a Jinja template that converts conversation histories into a correctly formatted string. Please see the technical documentation for information on how to write and apply chat templates in your code.

Introduction

If you're familiar with the 🤗 Transformers library, you've probably written code like this:

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModel.from_pretrained(checkpoint)

By loading the tokenizer and model from the same checkpoint, you ensure that inputs are tokenized in the way the model expects. If you pick a tokenizer from a different model, the input tokenization might be completely different, and the result will be that your model's performance will be seriously damaged. The term for this is a distribution shift - the model has been learning data from one distribution (the tokenization it was trained with), and suddenly it has shifted to a completely different one.

Whether you're fine-tuning a model or using it directly for inference, it's always a good idea to minimize these distribution shifts and keep the input you give it as similar as possible to the input it was trained on. With regular language models, it's relatively easy to do that - simply load your tokenizer and model from the same checkpoint, and you're good to go.

If that all seems a bit abstract, here's an example chat to make it more concrete:

User: Hey there!
Bot: Nice to meet you!

Or you could add special tokens to indicate the roles: