Hardware for Inference Engineers
Hardware for inference is knowing the accelerators that run models. NVIDIA GPUs have streaming multiprocessors, tensor cores, high-bandwidth memory and NVLink interconnects, and their limits shape performance. Alternatives include AMD GPUs, Google TPUs and custom accelerators, each with different software stacks and trade-offs.
Stage: Foundations · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Hardware
- Describe GPU memory hierarchy from registers to HBM
- Read a spec sheet for memory bandwidth, FLOPs and capacity
- Use a roofline model to judge compute versus memory limits
- Compare NVLink with PCIe for multi-GPU communication
- Weigh TPU and AMD options against software ecosystem maturity
Hardware interview questions
What is the roofline model?
It plots achievable performance against arithmetic intensity, the FLOPs done per byte moved. Low-intensity workloads sit under the memory-bandwidth slope, and high-intensity ones hit the compute ceiling, showing which limit to optimise.
Why does HBM capacity matter for serving LLMs?
The GPU memory must hold model weights plus the KV cache for all active requests. Capacity decides how large a model fits and how many concurrent sequences or how much context can be served, so it often limits batch size and cost.
Where Hardware fits in the Inference Engineer roadmap
Hardware belongs to the Foundations stage (intermediate level), which takes about 7 hours. What happens in a forward pass.
Other units in this stage
Back to the full Inference Engineer roadmap