Kernels and Compilation for Inference Engineers
Kernels and compilation are how model operations become fast GPU code. A kernel is a function that runs on the GPU, written in CUDA or in Python-embedded Triton. Compilers and runtimes such as TensorRT, torch.compile and XLA fuse operations and pick efficient implementations, reducing memory traffic and launch overhead.
Stage: Performance Engineering · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Kernels and Compilation
- Describe threads, blocks and warps in the CUDA execution model
- Write a simple vector addition kernel in Triton
- Explain how kernel fusion reduces memory reads and writes
- Use torch.compile and inspect what it changed
- Recognise why FlashAttention avoids materialising the attention matrix
Kernels and Compilation interview questions
What is kernel fusion and why does it speed things up?
Fusion combines several operations, such as a matrix multiply, bias and activation, into one kernel so intermediate results stay in fast on-chip memory instead of being written to and read from global memory. It cuts memory traffic and launch overhead.
What does FlashAttention do?
FlashAttention computes attention in tiles that fit in on-chip memory, using an online softmax, so the full sequence-by-sequence score matrix is never stored in high-bandwidth memory. It reduces memory use and traffic while producing exact results.
Where Kernels and Compilation fits in the Inference Engineer roadmap
Kernels and Compilation belongs to the Performance Engineering stage (advanced level), which takes about 5 hours. Make it fast.
Other units in this stage
Back to the full Inference Engineer roadmap