From Zero to GPU: A Guide to Building and Scaling Production-Ready CUDA Kernels

We have revamped the Kernels project since the publication of this post. Check out the updates here.

Custom CUDA kernels give your models a serious performance edge, but building them for the real world can feel daunting. How do you move beyond a simple GPU function to create a robust, scalable system without getting bogged down by endless build times and dependency nightmares?

We created the kernel-builder library for this purpose. You can develop a custom kernel locally, and then build it for multiple architectures and make it available for the world to use.

In this guide we'll show you how to build a complete, modern CUDA kernel from the ground up. Then, we’ll tackle the tough production and deployment challenges, drawing on real-world engineering strategies to show you how to build systems that are not just fast, but also efficient and maintainable.

What You’ll Learn

When you're done, other developers will be able to use your kernels directly from the hub like this:

import torch

from kernels import get_kernel

# Download custom kernel from the Hugging Face Hub
optimized_kernel = get_kernel("your-username/optimized-kernel")

# A sample input tensor
some_input = torch.randn((10, 10), device="cuda")

# Run the kernel
out = optimized_kernel.my_kernel_function(some_input)

print(out)

Rather watch a video? Check out the YouTube video that accompanies this guide.

Let's Get Started! 🚀

Part 1: Anatomy of a Modern CUDA Kernel

Let's build a practical kernel that converts an image from RGB to grayscale. This example uses PyTorch's modern C++ API to register our function as a first-class, native operator.

Step 1: Project Structure

A clean, predictable structure is the foundation of a good project. The Hugging Face Kernel Builder expects your files to be organized like this:

img2gray/
├── build.toml
├── csrc
│   └── img2gray.cu
├── flake.nix
└── torch-ext
    ├── torch_binding.cpp
    ├── torch_binding.h
    └── img2gray
        └── __init__.py

You can also follow a standard git-based process for the upload.