The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough

NVIDIA CUDA remains the foundation of GPU-accelerated computing, powering everything from scientific simulations to large-scale AI training.

But writing correct, maintainable, and performant CUDA code can be challenging: memory bugs hide in plain sight, performance bottlenecks can be invisible without the right instrumentation, and hand-rolled GPU algorithms rarely match the efficiency of optimized libraries. Fortunately, the modern CUDA toolchain has matured significantly, and many of these challenges now have straightforward solutions.

In this blog post, we will walk through the tools NVIDIA offers to debug, benchmark, and improve your code. With only small line changes each time, we are going to make the example code safer, easier to maintain, and faster.

Across six incremental steps, this post will cover:

1. How to easily find indexing bugs by adopting the modern CCCL API and Compute Sanitizer
2. How to improve Nsight Systems benchmarks with NVTX
3. How to use CUB’s optimized algorithms at the block and device level
4. How to manage GPU memory through pooled containers
5. How to speed up host-to-device transfers with pinned containers
6. How to parallelize GPU work by giving each thread its own stream and asynchronous transfers

As a companion to this blog post, we provide the code and the option to run on Google Colab.

Starting point: An image processing pipeline example

From an input stream of red, green, and blue images, start by transferring the data from the CPU to the GPU. Then convert those from RGB to grayscale.

Then, for each 32 by 32 pixel tile in the image, compute the median by sorting the pixels and selecting the middle value. Finally, copy the median of each tile back to the CPU.

The base code example

Below is the full starting code. Each step in this post improves on it.

This code starts by defining two kernels:

In the main, after defining the constants used for the example, the CPU memory is allocated for each image and for the medians.

Let’s fix it.