A Dive into Vision-Language Models

Human learning is inherently multi-modal as jointly leveraging multiple senses helps us understand and analyze new information better. Unsurprisingly, recent advances in multi-modal learning take inspiration from the effectiveness of this process to create models that can process and link information using various modalities such as image, video, text, audio, body gestures, facial expressions, and physiological signals.

Since 2021, we’ve seen an increased interest in models that combine vision and language modalities (also called joint vision-language models), such as OpenAI’s CLIP. Joint vision-language models have shown particularly impressive capabilities in very challenging tasks such as image captioning, text-guided image generation and manipulation, and visual question-answering. This field continues to evolve, and so does its effectiveness in improving zero-shot generalization leading to various practical use cases.

In this blog post, we'll introduce joint vision-language models focusing on how they're trained. We'll also show how you can leverage 🤗 Transformers to experiment with the latest advances in this domain.

Table of contents

1. Introduction
2. Learning Strategies
   1. Contrastive Learning
   2. PrefixLM
   3. Multi-modal Fusing with Cross Attention
   4. MLM / ITM
   5. No Training
3. Datasets
4. Supporting Vision-Language Models in 🤗 Transformers
5. Emerging Areas of Research
6. Conclusion

Introduction

What does it mean to call a model a “vision-language” model? A model that combines both the vision and language modalities? But what exactly does that mean?

One characteristic that helps define these models is their ability to process both images (vision) and natural language text (language). This process depends on the inputs, outputs, and the task these models are asked to perform.

Take, for example, the task of zero-shot image classification. We’ll pass an image and a few prompts like so to obtain the most probable prompt for the input image.

The cat and dog image has been taken from here.