Zero-shot image-to-text generation with BLIP-2

This guide introduces BLIP-2 from Salesforce Research that enables a suite of state-of-the-art visual-language models that are now available in 🤗 Transformers. We'll show you how to use it for image captioning, prompted image captioning, visual question-answering, and chat-based prompting.

Table of contents

1. Introduction
2. What's under the hood in BLIP-2?
3. Using BLIP-2 with Hugging Face Transformers
   1. Image Captioning
   2. Prompted image captioning
   3. Visual question answering
   4. Chat-based prompting
4. Conclusion
5. Acknowledgments

Introduction

Recent years have seen rapid advancements in computer vision and natural language processing. Still, many real-world problems are inherently multimodal - they involve several distinct forms of data, such as images and text. Visual-language models face the challenge of combining modalities so that they can open the door to a wide range of applications. Some of the image-to-text tasks that visual language models can tackle include image captioning, image-text retrieval, and visual question answering. Image captioning can aid the visually impaired, create useful product descriptions, identify inappropriate content beyond text, and more. Image-text retrieval can be applied in multimodal search, as well as in applications such as autonomous driving. Visual question-answering can aid in education, enable multimodal chatbots, and assist in various domain-specific information retrieval applications.

What's under the hood in BLIP-2?

BLIP-2 bridges the modality gap between vision and language models by adding a lightweight Querying Transformer (Q-Former) between an off-the-shelf frozen pre-trained image encoder and a frozen large language model. Q-Former is the only trainable part of BLIP-2; both the image encoder and language model remain frozen.

Q-Former is a transformer model that consists of two submodules that share the same self-attention layers:

Using BLIP-2 with Hugging Face Transformers

Let's use GPU to make text generation faster:

"A torch"