Multimodal Embedding & Reranker Models with Sentence Transformers

Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v5.4 update, you can now encode and compare texts, images, audio, and videos using the same familiar API. In this blogpost, I'll show you how to use these new multimodal capabilities for both embedding and reranking.

Multimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines.

If you want to train your own multimodal models, check out the companion blogpost: Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers.

Table of Contents

• What are Multimodal Models?
• Installation
• Multimodal Embedding Models
  • Loading a Model
  • Encoding Images
  • Cross-Modal Similarity
  • Encoding Queries and Documents
• Multimodal Reranker Models
  • Ranking Mixed-Modality Documents
  • Predicting Pair Scores
• Retrieve and Rerank
• Input Formats and Configuration
  • Supported Input Types
  • Checking Modality Support
  • Processor and Model kwargs
• Supported Models
• Additional Resources

What are Multimodal Models?

Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you can compare a text query against image documents (or vice versa) using the same similarity functions you're already familiar with.

Similarly, traditional reranker (Cross Encoder) models compute relevance scores between pairs of texts. Multimodal rerankers can score pairs where one or both elements are images, combined text-image documents, or other modalities.

Click here if you need to pass raw message inputs Click here for a text-only reranker usage example Click here for a CLIP usage example