Training and Finetuning Embedding Models with Sentence Transformers

Sentence Transformers is a Python library for using and training embedding models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, paraphrase mining, and more. In this blogpost, I'll show you how to use it to finetune Sentence Transformer models to improve their performance on specific tasks. You can also use this method to train new Sentence Transformer models from scratch.

Finetuning Sentence Transformers involves several components, including datasets, loss functions, training arguments, evaluators, and the trainer itself. I'll go through each of these components in detail and provide examples of how to use them to train effective models.

Table of Contents

• Why Finetune?
• Training Components
• Dataset
  • Data on Hugging Face Hub
  • Local Data (CSV, JSON, Parquet, Arrow, SQL)
  • Local Data that requires pre-processing
  • Dataset Format
• Loss Function
• Training Arguments
• Evaluator
  • EmbeddingSimilarityEvaluator with STSb
  • TripletEvaluator with AllNLI
• Trainer
  • Callbacks
• Multi-Dataset Training
• Deprecation
• Additional Resources
  • Training Examples
  • Documentation

Why Finetune?

Finetuning Sentence Transformer models can significantly enhance their performance on specific tasks. This is because each task requires a unique notion of similarity. Let's consider a couple of news article headlines as an example:

• "Apple launches the new iPad"
• "NVIDIA is gearing up for the next GPU generation"

Depending on the use case, we might want similar or dissimilar embeddings for these texts. For instance, a classification model for news articles could treat these texts as similar since they both belong to the Technology category. On the other hand, a semantic textual similarity or retrieval model should consider them dissimilar due to their distinct meanings.

Training Components

Training Sentence Transformer models involves the following components:

Now, let's dive into each of these components in more detail.