5 Fun Projects for LLM Inference Engineering

Training large language models gets most of the attention, but once a model is trained, another engineering problem begins: how do you serve it quickly and efficiently?

LLM inference engineering focuses on reducing latency, increasing throughput, lowering memory consumption, and getting more useful work from the available hardware. This involves techniques such as KV caching, continuous batching, quantization, prefix caching, speculative decoding, optimized attention kernels, and distributed inference. The best way to learn these concepts is not by reading about them individually — it is by building something and seeing what happens. In this article, we will look at 5 projects that progressively introduce the most important ideas behind modern LLM inference systems.

1. Building a Mini-vLLM From Scratch

If you want one project that exposes you to almost the entire inference stack, this is probably the best place to start. Nano-vLLM is a small educational implementation of a vLLM-style inference engine, with its core implementation small enough to actually read. It includes scheduling, KV-cache block management, prefix caching, tensor parallelism, CUDA graphs, and other optimizations while using a real Qwen model. Instead of simply cloning it and running the example, rebuild one component at a time: first perform normal autoregressive generation, then add KV caching, then batching, then block-based KV-cache allocation. Benchmark tokens per second after every addition. By the end, you will understand concepts like PagedAttention and continuous batching in a much better way.

2. Writing Your Own FlashAttention Kernel With Triton

3. Building an LLM Quantization Lab

4. Implementing Speculative Decoding

5. Building Your Own Production LLM Inference Service

Wrapping Up

This is why building projects is such a good way to learn the field.