Training and Finetuning Sparse Embedding Models with Sentence Transformers

Sentence Transformers is a Python library for using and training dense embedding, reranker (cross encoder), and sparse embedding models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, paraphrase mining, and more. In this blogpost, I'll show you how to use it to finetune a sparse encoder/embedding model and explain why you might want to do so. This results in sparse-encoder/example-inference-free-splade-distilbert-base-uncased-nq, a cheap model that works especially well in hybrid search or retrieve and rerank scenarios.

Finetuning sparse embedding models involves several components: the model, datasets, loss functions, training arguments, evaluators, and the trainer class. I'll have a look at each of these components, accompanied by practical examples of how they can be used for finetuning strong sparse embedding models.

In addition to training your own models, you can choose from a wide range of pretrained sparse encoders available on the Hugging Face Hub. To help navigate this growing space, we’ve curated a SPLADE Models collection highlighting some of the most relevant models.
We list the most prominent ones along with their benchmark results in Pretrained Models in the documentation.

Table of Contents

• What are Sparse Embedding models?
  • Query and Document Expansion
  • Why Use Sparse Embedding Models?
• Why Finetune?
• Training Components
• Model
  • Splade
  • Inference-free Splade
  • Contrastive Sparse Representation (CSR)
  • Architecture Picker Guide
• Dataset
  • Data on the Hugging Face Hub
  • Local Data (CSV, JSON, Parquet, Arrow, SQL)
  • Local Data that requires pre-processing
  • Dataset Format
• Loss Function
• Training Arguments
• Evaluator
  • SparseNanoBEIREvaluator
  • SparseEmbeddingSimilarityEvaluator with STSb
  • SparseTripletEvaluator with AllNLI
• Trainer
  • Callbacks
  • Multi-Dataset Training
• Evaluation
• Training Tips
• Vector Database Integration
• Additional Resources
  • Training Examples
  • Documentation