Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler

What you cannot profile, you cannot optimize.

Whether you are trying to squeeze more tokens per second out of a Large Language Model (LLM), shave milliseconds off inference, or just understand why your training loop runs slower than the spec sheet promises, the path eventually runs through profiling.

The catch is that profiling has a steep on-ramp. The traces are dense walls of colored rectangles. The events carry intimidating names. Most tutorials assume you can already read them. So even when we know we should be profiling, opening a trace can feel like a chore best left for later (or for someone else). This post, and the series it kicks off, is our attempt to lower that on-ramp.

We document the journey from a beginner's point of view. No prerequisites apart from basic PyTorch. Treat this as a leisurely read with some "Aha!" moments. The structure of the post is intentionally question-led: we open a trace, ask "wait, why is that happening?", and chase the answer until something clicks. By the end you should know:

• how to set up torch.profiler and what it actually hands back,
• how to read the profiler table and the trace (CPU lane, GPU lane, and the suspicious gaps in between),
• the chain of events from a Python call all the way down to a CUDA kernel,
• what changes (and, more interestingly, what does not change) when you slap torch.compile on top.

Before we begin, two definitions that will make everything below read better:

1. A GPU kernel is a program that runs in parallel on many threads of the GPU.
2. The CPU schedules and launches these kernels.

You don't usually have to write GPU kernels yourself; when you use a PyTorch operation, it is automatically translated to one or more kernels that do the job on GPU.

With those two ideas in your back pocket, let's start asking questions.

The matrix multiplication and addition operation

def fn(x, w, b):
  return torch.add(torch.matmul(x, w), b)

To profile, we will be using the torch.profiler module. The steps involved are:

1. Export the profile