How to read a PyTorch attention trace before optimizing it

On July 10, 2026, Hugging Face published the third part of its PyTorch profiling tutorial, devoted to reading attention traces before optimizing. When a Transformer model becomes slow, attention often appears early in the discussion: it combines matrix multiplications, masks, softmax, and intermediate memory; with long sequences, all of that grows quickly. The lesson is to inspect the trace before replacing a few lines with an “optimized” function.

Profiling is not about guessing which part of a program looks expensive. It is about observing which operators and kernels actually run, how long they take, and what memory movement comes with them. PyTorch provides torch.profiler to record CPU and CUDA activity, input shapes, and memory use. A table helps identify candidates; a trace explains order and dependencies.

Attention that looks simple

A direct implementation of causal attention computes products between queries and keys, scales the result, applies a mask, runs softmax, and multiplies by the values again. In a trace, those operations are not an abstraction: they become several GPU kernels and sometimes copies that did not appear in the pseudocode.

The Hugging Face tutorial shows one concrete example on an NVIDIA A100. The direct version launched a memory copy alongside the expected calculations. Replacing masked_fill with its in-place variant removed that copy and took the forward pass from six to five kernels. That is a useful observation, not a universal recipe: in-place operations can overwrite values that autograd needs for the backward pass. In the experiment they were safe because inference ran under torch.no_grad; that detail changes the decision completely in training.

A short function does not always mean less work

PyTorch packages this operation in torch.nn.functional.scaled_dot_product_attention, or SDPA. For CUDA tensors it can select among a math implementation, FlashAttention, a memory-efficient variant, and other compatible paths. The interface is convenient, but the selected backend depends on data type, dimensions, masking, and hardware.

In the article’s tests, pinning SDPA to the math backend produced a surprise: it launched 20 kernels per forward rather than the in-place naive version’s five, and it was about 3.7 times slower. That does not mean SDPA is bad. That route is intended as a safe, general reference. In this particular case it materialized a mask, used a protected softmax, and upcast operations to FP32, adding work and memory traffic.

The efficient, flash, and cuDNN routes looked different: one fused kernel per forward. Fusion avoids writing the full sequence-by-sequence attention matrix into the GPU’s main memory and reduces launches. On the A100 used in the experiment, Flash was the fastest CUDA path for that shape, averaging 146.8 microseconds on the GPU, compared with 277.9 for efficient and 186.3 for cuDNN. But cuDNN spent more CPU time preparing its plan and can win at other dimensions. A cleaner trace does not prove that work has vanished; sometimes it has only moved inside a library.

A method before a trick

To profile attention carefully, it helps to follow a short sequence. First, profile a representative case after warm-up, separating CPU and CUDA. Second, sort by self and total time: self time excludes child calls, while total time includes them. Third, open the trace and look for repeated kernels, copies, rebuilt masks, or gaps between CPU and GPU.

Then compare SDPA backends with the same inputs, measure latency and memory, and verify that output retains the needed accuracy. PyTorch’s documentation allows a backend to be temporarily forced precisely for that comparison. If a choice is unsupported for a shape or hardware target, the warning is useful information too.

Attention optimization is not just about saying “FlashAttention.” It is about learning to read what a program does on a specific GPU and preserving correctness while improving it. A profile does not hand over an automatic answer, but it prevents an elegant code change from making the real workload worse.