GPU Fundamentals for Deep Learning Engineers
GPU fundamentals describe how graphics processors execute deep learning workloads. A GPU has many streaming multiprocessors running thousands of threads in warps, with a memory hierarchy of registers, shared memory, L2 cache and high-bandwidth memory. Profilers reveal whether a kernel is limited by compute or by memory bandwidth.
Stage: GPU Performance and Deployment · Part of the Deep Learning Engineer roadmap · Last reviewed 19 November 2026
What to learn in GPU Fundamentals
- Describe threads, warps, blocks and streaming multiprocessors
- Explain the GPU memory hierarchy and its relative speeds
- Distinguish compute-bound from memory-bound operations
- Profile a training step with PyTorch Profiler or Nsight
- Read arithmetic intensity using the roofline model
- Explain why tensor cores need specific data types and shapes
GPU Fundamentals interview questions
What does it mean for an operation to be memory-bound?
Its runtime is limited by how fast data moves between memory and the compute units rather than by arithmetic speed. Elementwise operations like adding tensors are typical examples, and fusing them into one kernel reduces memory traffic.
What is a warp?
On NVIDIA GPUs a warp is a group of 32 threads that execute the same instruction together. If threads in a warp take different branches, the paths run one after another, which is called warp divergence and lowers efficiency.
Where GPU Fundamentals fits in the Deep Learning Engineer roadmap
GPU Fundamentals belongs to the GPU Performance and Deployment stage (advanced level), which takes about 4 hours. Make it fast, small and shippable.
Other units in this stage
Back to the full Deep Learning Engineer roadmap