Visual Document Retrieval Goes Multilingual

TL;DR: We present vdr-2b-multi-v1, the best multilingual embedding model for visual document retrieval. We also release its English-only twin vdr-2b-v1 and open-source the new vdr-multilingual-train dataset. With 500k high-quality samples, it's the largest open-source multilingual synthetic dataset for visual document retrieval.

Introducing vdr-2b-multi-v1 (🤗), a multilingual embedding model designed for visual document retrieval across multiple languages and domains. This model is designed to encode document page screenshots into dense single-vector representations, this will effectively allow to search and query visually rich multilingual documents without the need for any OCR, data extraction pipelines, chunking...

The vdr-2b-multi-v1 model is based on MrLight/dse-qwen2-2b-mrl-v1 and is trained on an extensive self-made dataset of multilingual query-image pairs. This model is built in collaboration with LlamaIndex and is the next iteration of mcdse-2b-v1. Our vdr-2b-multi-v1 extends and improves the learning and methods used to train it, resulting in a much more powerful and better model.

• Trained on 🇮🇹 Italian, 🇪🇸 Spanish, 🇬🇧 English, 🇫🇷 French and 🇩🇪 German: Together they form a new large, open-source, multilingual training dataset of 500k high-quality samples.

• Low VRAM and Faster Inference: On synthetic Visual Document Retrieval (ViDoRe) benchmarks, our English-only model with 768 image patches performs better than the base model with 2560 image patches. This results in 3x faster inference and much lower VRAM usage.

• Cross-lingual Retrieval: Substantially better on real-world scenarios. For example, you can search for German documents with Italian queries.

• Matryoshka Representation Learning: You can reduce the vectors size 3x and still keep 98% of the embeddings quality. This allows for notably faster retrieval speeds while reducing storage costs.

Usage

🎲 Try out vdr-2b-multi-v1 now, available on this Hugging Face Space!

via LlamaIndex via SentenceTransformers