🪆 Introduction to Matryoshka Embedding Models

In this blogpost, we will introduce you to the concept of Matryoshka Embeddings and explain why they are useful. We will discuss how these models are theoretically trained and how you can train them using Sentence Transformers.

Additionally, we will provide practical guidance on how to use Matryoshka Embedding models and share a comparison between a Matryoshka embedding model and a regular embedding model. Finally, we invite you to check out our interactive demo that showcases the power of these models.

Table of Contents

• Understanding Embeddings
• 🪆 Matryoshka Embeddings
• 🪆 Matryoshka Dolls
• Why would you use 🪆 Matryoshka Embedding models?
• How are 🪆 Matryoshka Embedding models trained?
  • Theoretically
  • In Sentence Transformers
• How do I use 🪆 Matryoshka Embedding models?
  • Theoretically
  • In Sentence Transformers
• Results
• Demo
• References

Understanding Embeddings

Embeddings are one of the most versatile tools in natural language processing, enabling practitioners to solve a large variety of tasks. In essence, an embedding is a numerical representation of a more complex object, like text, images, audio, etc.

The embedding model will always produce embeddings of the same fixed size. You can then compute the similarity of complex objects by computing the similarity of the respective embeddings!

This has an enormous amount of use cases, and serves as the backbone for recommendation systems, retrieval, one-shot or few-shot learning, outlier detection, similarity search, paraphrase detection, clustering, classification, and much more!

🪆 Matryoshka Embeddings

As research progressed, new state-of-the-art (text) embedding models started producing embeddings with increasingly higher output dimensions, i.e., every input text is represented using more values. Although this improves performance, it comes at the cost of efficiency of downstream tasks such as search or classification.