Faster Text Generation with Self-Speculative Decoding

Self-speculative decoding, proposed in LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding is a novel approach to text generation. It combines the strengths of speculative decoding with early exiting from a large language model (LLM). This method allows for efficient generation by using the same model's early layers for drafting tokens, and later layers for verification.

This technique not only speeds up text generation, but it also achieves significant memory savings and reduces computational latency. In order to obtain an end-to-end speedup, the output of the earlier layers need to be close enough to the last layer. This is achieved by a training recipe which, as described in the paper, can be applied during pretraining, and also while fine-tuning on a specific domain. Self-speculative decoding is especially efficient for real-world applications, enabling deployment on smaller GPUs and lowering the overall hardware footprint needed for large-scale inference.

In this blog post, we explore the concept of self-speculative decoding, its implementation, and practical applications using the 🤗 transformers library. You’ll learn about the technical underpinnings, including early exit layers, unembedding, and training modifications. To ground these concepts in practice, we offer code examples, benchmark comparisons with traditional speculative decoding, and insights into performance trade-offs.

Dive straight into the following Hugging Face artifacts to know more about the method and try it out yourself:

1. Hugging Face Paper Discussion Forum
2. LayerSkip Model Collections
3. Colab Notebook showcasing the in-depth working of self-speculative decoding

Speculative Decoding and Self-Speculative Decoding

Illustration of LayerSkip inference on facebook/layerskip-llama2-7B (Llama2 7B continually pretrained with the LayerSkip recipe).

Usage with transformers

Here is a simple code snippet showcasing the functionality.

pip install transformers