How to generate text: using different decoding methods for language generation with Transformers

Note: Edited on July 2023 with up-to-date references and examples.

Introduction

In recent years, there has been an increasing interest in open-ended language generation thanks to the rise of large transformer-based language models trained on millions of webpages, including OpenAI's ChatGPT and Meta's LLaMA. The results on conditioned open-ended language generation are impressive, having shown to generalize to new tasks, handle code, or take non-text data as input. Besides the improved transformer architecture and massive unsupervised training data, better decoding methods have also played an important role.

This blog post gives a brief overview of different decoding strategies and more importantly shows how you can implement them with very little effort using the popular transformers library!

All of the following functionalities can be used for auto-regressive language generation (here a refresher). In short, auto-regressive language generation is based on the assumption that the probability distribution of a word sequence can be decomposed into the product of conditional next word distributions:

P(w1:T∣W0)=∏t=1TP(wt∣w1:t−1,W0) ,with w1:0=∅, P(w_{1:T} | W_0 ) = \prod_{t=1}^T P(w_{t} | w_{1: t-1}, W_0) \text{ ,with } w_{1: 0} = \emptyset,

and W0W_0 being the initial context word sequence. The length TT of the word sequence is usually determined on-the-fly and corresponds to the timestep t=Tt=T the EOS token is generated from P(wt∣w1:t−1,W0)P(w_{t} | w_{1: t-1}, W_{0}).

We will give a tour of the currently most prominent decoding methods, mainly Greedy search, Beam search, and Sampling.

Let's quickly install transformers and load the model. We will use GPT2 in PyTorch for demonstration, but the API is 1-to-1 the same for TensorFlow and JAX.

!pip install -q transformers

Greedy Search

The major drawback of greedy search though is that it misses high probability words hidden behind a low probability word as can be seen in our sketch above: