Preference Tuning LLMs with Direct Preference Optimization Methods

Addendum

After consulting with the authors of the IPO paper, we discovered that the implementation of IPO in TRL was incorrect; in particular, the loss over the log-likelihoods of the completions needs to be averaged instead of summed. We have added a fix in this PR and re-run the experiments. The results are now consistent with the paper, with IPO on par with DPO and performing better than KTO in the paired preference setting. We have updated the post to reflect these new results.

TL;DR

We evaluate three promising methods to align language models without reinforcement learning (or preference tuning) on a number of models and hyperparameter settings. In particular we train using different hyperparameters and evaluate on:

• Direct Preference Optimization (DPO)
• Identity Preference Optimisation (IPO)
• Kahneman-Tversky Optimisation (KTO)

Introduction

In this post, we perform an empirical evaluation of three promising LLM alignment algorithms: Direct Preference Optimization (DPO), Identity Preference Optimisation (IPO) and Kahneman-Tversky Optimisation (KTO). We conducted our experiments on two high quality 7b LLMs that have undergone a supervised fine-tuning step, but no preference alignment. We find that while one algorithm clearly outshines the others, there are key hyper-parameters that must be tuned to achieve the best results.

Alignment without Reinforcement Learning

Image from the DPO paper (https://arxiv.org/abs/2305.18290)

Direct Preference Optimization (DPO) has emerged as a promising alternative for aligning Large Language Models (LLMs) to human or AI preferences. Unlike traditional alignment methods, which are based on reinforcement learning, DPO recasts the alignment formulation as a simple loss function that can be optimised directly on a dataset of preferences {(x,yw,yl)} \{(x, y_w, y_l)\} , where xx is a prompt and yw,yly_w,y_l are the preferred and dispreferred responses.

Sample of a preference tuning dataset.

Let’s get started!