Hugging Face published the third entry in its PyTorch profiling series on July 10, 2026, detailing how distinct attention implementations interact with GPU kernels and system memory. Written by author ariG23498, the publication tested naive attention code alongside PyTorch's Scaled Dot Product Attention backends on an NVIDIA A100-SXM4-80GB GPU.
Testing a naive attention implementation revealed that PyTorch's standard out-of-place masked_fill operation inserts an extraneous GPU memory copy kernel into every forward pass. Replacing masked_fill with the in-place variant masked_fill_ removed the memory copy kernel entirely. Hugging Face noted this change saves compute time and memory without risking autograd corruption when executing under torch.no_grad.
Backend Trade-offs
PyTorch's built-in scaled_dot_product_attention function dispatched across several backends, yielding stark performance differences in profiler traces. The reference math backend ran roughly 3.7 times slower than naive attention, launching 20 GPU kernels per step compared to five in the naive code. Profiler traces showed the math backend upcasts tensors to FP32, uses general-purpose CUDA cores instead of bfloat16 Tensor Cores, and rebuilds causal masks on every call.
Fused attention backends consolidated the entire sequence of operations into single GPU kernels. Meta's efficient backend executed a single CUTLASS-based kernel in bfloat16 on Tensor Cores. The flash backend, implementing FlashAttention-2, kept attention scores on-chip using online softmax and tiling to prevent writing large score matrices to high-bandwidth memory.
Occupancy and Driver Launch Differences
The profiler reported a low 13 percent occupancy for the FlashAttention kernel because each thread block heavily consumes registers and shared memory. High resource usage per block limits how many warps reside on a Streaming Multiprocessor at once, though total execution time remains fast.
NVIDIA's cudnn backend generated customized kernels for input shapes, removing explicit tensor transpose operations on the CPU. However, CPU overhead increased to 214 microseconds per forward pass as cuDNN selected configuration plans. The driver-level cuLaunchKernelEx call caused CUPTI profiling tools to report zero percent achieved occupancy, despite true block occupancy reaching roughly 12.5 percent.
