No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL

🚀 Introduction

TRL supports training LLMs using GRPO, an online learning algorithm recently introduced in the DeepSeekMath paper. In GRPO, the model learns from its own outputs: it generates responses during training, receives feedback, and uses that feedback to improve itself over time.

This makes generation a critical step in the training loop — and also a major bottleneck. To speed up generation, TRL integrates with vLLM. This combination lets you train powerful models more efficiently in GRPO setup. However, there’s a catch.

🧨 The Problem

Before TRL v0.18.0, vLLM was only supported in server mode, running as a separate process on different GPUs from the training job. It communicated with the training script over HTTP, which made the setup modular and easy to use — but also introduced GPU inefficiencies.

Here’s what happens:

During training, the model needs to generate completions frequently.
The trainer sends a request to the vLLM server, which runs on its own GPUs.
While vLLM generates, the training GPUs sit idle and wait.
Once generation is done, vLLM GPUs become idle, and training resumes.

This “ping-pong” between training and generation causes:

Wasted GPU time on both sides
Increased demand for extra GPUs just to run inference
Reduced overall throughput and higher cost

In online learning methods like GRPO — where generation happens constantly — this inefficiency becomes even more painful. You spend more on hardware, but don't get the performance you'd expect.

So, the key question becomes: Can we share the same GPUs for both training and generation, instead of separating them?

💡 The Opportunity

This approach is what we refer to as colocation. Training and inference are co-located on the same GPUs and coordinated via the same process group, allowing them to take turns smoothly — no extra hardware needed.