Guiding Text Generation with Constrained Beam Search in 🤗 Transformers

Introduction

This blog post assumes that the reader is familiar with text generation methods using the different variants of beam search, as explained in the blog post: "How to generate text: using different decoding methods for language generation with Transformers"

Unlike ordinary beam search, constrained beam search allows us to exert control over the output of text generation. This is useful because we sometimes know exactly what we want inside the output. For example, in a Neural Machine Translation task, we might know which words must be included in the final translation with a dictionary lookup. Sometimes, generation outputs that are almost equally possible to a language model might not be equally desirable for the end-user due to the particular context. Both of these situations could be solved by allowing the users to tell the model which words must be included in the end output.

Why It's Difficult

However, this is actually a very non-trivial problem. This is because the task requires us to force the generation of certain subsequences somewhere in the final output, at some point during the generation.

Let's say that we're want to generate a sentence S that has to include the phrase p1={t1,t2} p_1=\{ t_1, t_2 \} with tokens t1,t2 t_1, t_2 in order. Let's define the expected sentence S S as:

Sexpected={s1,s2,...,sk,t1,t2,sk+1,...,sn} S_{expected} = \{ s_1, s_2, ..., s_k, t_1, t_2, s_{k+1}, ..., s_n \}

And what if you have multiple constraints with varying requirements? What if you want to force the phrase p1={t1,t2} p_1=\{t_1, t_2\} and also the phrase p2={t3,t4,t5,t6} p_2=\{ t_3, t_4, t_5, t_6\} ? What if you want the model to choose between the two phrases? What if we want to force the phrase p1 p_1 and force just one phrase among the list of phrases {p21,p22,p23} \{p_{21}, p_{22}, p_{23}\} ?

The above examples are actually very reasonable use-cases, as it will be shown below, and the new constrained beam search feature allows for all of them!

Banks

or: