Vision Language Models (Better, faster, stronger)

Motivation

Vision Language Models (VLMs) are the talk of the town. In a previous blog post (from April 2024), we talked a lot about VLMs. A major chunk was about LLaVA, the first successful and easily reproducible open-source vision language model, along with tips on how to discover, evaluate, and fine-tune open models.

Since then, so much has changed. Models have become smaller yet more powerful. We've seen the rise of new architectures and capabilities (reasoning, agency, long video understanding, etc.). In parallel, entirely new paradigms, such as multimodal Retrieval Augmented Generation (RAG) and multimodal agents have taken shape.

In this blog post, we’ll take a look back and unpack everything that happened with vision language models the past year. You’ll discover key changes, emerging trends, and notable developments.

We highly recommend reading the first blog post if you want a good primer on how vision language models work.

Table of Contents

• New Model Trends
  • Any-to-any models
  • Reasoning models
  • Smol Yet Capable Models
  • Mixture-of-Experts as Decoders
  • Vision Language Action Models
• Specialized Capabilities
  • Object Detection, Segmentation, Counting with Vision Language Models
  • Multimodal Safety Models
  • Multimodal RAG: retrievers, rerankers
• Multimodal Agents
• Video Language Models
• New Alignment Techniques for Vision Language Models
• New Benchmarks
  • MMT-Bench
  • MMMU-Pro
• Extra: Our Model Picks
• Useful Resources

New model trends

In this section, we will look at the new types of VLMs. While some are absolutely new, others are improved versions of previous research.

Any-to-any models

Any-to-any models, as the name suggests, are models that can take in any modality and output any modality (image, text, audio). They do it by aligning the modalities, where an input from one modality can be translated to another (e.g. the word “dog” would be associated with an image of a dog, or with the utterance of the word).