TRL v1.0: Post-Training Library Built to Move with the Field

We’re releasing TRL v1.0, and it marks a real shift in what TRL is. What started as a research codebase has become a dependable library people build on, with clearer expectations around stability. This isn't just a version bump. It reflects the reality that TRL now powers production systems, and embraces that responsibility.

TRL now implements more than 75 post-training methods. But coverage isn’t the goal by itself. What matters is making these methods easy to try, compare, and actually use in practice. The design of the library wasn’t decided upfront. It is the result of years of iteration — the first commit goes back more than six years — and it has been shaped by everything the field threw at it: new algorithms, new models, shifting paradigms. Over time, this pressure forced the codebase toward a very specific design. Parts of it might look unusual at first, but like in many evolutionary codebases, they exist for a reason.

TRL is built for a field that doesn’t sit still. So the question is not how to design the perfect abstraction. It is how to make stable software in a domain that keeps invalidating its own assumptions. This is what we tried to solve in TRL v1.0, and this post explains how.

1. A moving target: post-training as a shifting field

Post-training has not evolved as a smooth refinement of one recipe. It has moved through successive centers of gravity, each changing not just the objective, but the shape of the stack.

PPO [Schulman et al., (2017); Ziegler et al., (2019)] made one architecture look canonical: a policy, a reference model, a learned reward model, sampled rollouts, and an RL loop.

Then DPO-style methods such as the original DPO [Rafailov et al., (2023)], ORPO [Hong et al., (2024)], and KTO [Ethayarajh et al., (2024)] cut through that stack: preference optimization could work without a separate reward model, value model, or any online RL. Components that had looked fundamental suddenly looked optional.