Efficient Request Queueing – Optimizing LLM Performance

Serving LLMs to many applications and users in parallel is challenging because they compete for limited GPU resources. This article is the first in a series on LLM performance, based on our experience with serving self-hosted LLMs at TNG Technology Consulting GmbH. In the first part, we focus on the impact of queuing and discuss different scheduling strategies.

Starting Point: A Bare Inference Engine

An inference engine like vLLM or HuggingFace TGI consists of

• a worker that does the actual work of calculating the next token in a request
• a queue to which requests are added when they first arrive
• a scheduler that takes requests from the queue and moves them to the worker

Why do we need a queue here? Because calculations on the GPU are more performant and resource-efficient when they are done batch-wise instead of isolated for individual requests. This backend queue allows the scheduler to pick multiple requests and put them on the same batch to be processed.

Note that typically each inference engine serves only a single model, and we have multiple deployments running for different models in parallel.

Problem: "Power Users" Can Block Other Users

When a single user "A" sends a large number of requests, they will quickly fill up the queue. Other users ("B" and "C") that send their requests only shortly after are blocked from using the same model (until all requests from "A" have been processed). Note that the picture focuses on vLLM as inference engine, but the problem is more general and applies to other backends as well.

Solution: Fair Scheduling

The key idea is: prioritize requests from different users in our own component and not in the inference backend! Typically, you can't change the order of requests once they have been sent to the inference engine, so you have to bring them in the right order while they are still in the LLM-Server, where we have full control.