Vision Language Model Alignment in TRL ⚡️

Introduction

Vision Language Models (VLMs) are getting stronger, but aligning them to human preferences still matters. In TRL, we already showed how to post-train VLMs with Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). This time, we’re going further.

tl;dr Here’s what’s new in TRL:

Mixed Preference Optimization (MPO)
Group Relative Policy Optimization (GRPO)
Group Sequence Policy Optimization (GSPO) (a variant of GRPO)

These go beyond pairwise DPO, extracting richer signals from preference data and scaling better with modern VLMs.

We’ve also extended existing methods to support VLMs:

Reinforce Leave One Out (RLOO)
Online Direct Preference Optimization (Online DPO)

This enables more efficient and scalable multimodal alignment.

Finally:

Native Supervised Fine-tuning support for Vision Language Models
Training scripts and demo notebooks to help you get started quickly

Multimodal Alignment for VLMs in TRL ⚡️
- Introduction
- Alignment for Vision Language Models
  - Mixed Preference Optimization (MPO)
  - Multimodal Group Relative Policy Optimization (GRPO)
  - Group Sequence Policy Optimization (GSPO)
  - Comparison
  - Further Extensions for VLMs
    - Reinforce Leave One Out (RLOO)
    - Online Direct Preference Optimization (Online DPO)
- Native Supervised Fine-tuning Support
- vLLM Integration in TRL
- Useful Resources

Alignment for Vision Language Models

Traditionally, you would take a base model, apply SFT to follow instructions, and then apply DPO to align it to preferential data. Previously, we adapted this approach to Vision Language Models (VLMs) and validated it on IDEFICS2, showing improvement in model responses.

DPO works by optimizing preferences between pairs of model responses using a contrastive loss: you have a chosen and a rejected answer and you optimize your preferences based on what you want and don’t want.