Skip to main content

Hardware for Inference Engineers

Hardware for inference is knowing the accelerators that run models. NVIDIA GPUs have streaming multiprocessors, tensor cores, high-bandwidth memory and NVLink interconnects, and their limits shape performance. Alternatives include AMD GPUs, Google TPUs and custom accelerators, each with different software stacks and trade-offs.

Stage: Foundations · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026

What to learn in Hardware

Hardware interview questions

What is the roofline model?

It plots achievable performance against arithmetic intensity, the FLOPs done per byte moved. Low-intensity workloads sit under the memory-bandwidth slope, and high-intensity ones hit the compute ceiling, showing which limit to optimise.

Why does HBM capacity matter for serving LLMs?

The GPU memory must hold model weights plus the KV cache for all active requests. Capacity decides how large a model fits and how many concurrent sequences or how much context can be served, so it often limits batch size and cost.

Where Hardware fits in the Inference Engineer roadmap

Hardware belongs to the Foundations stage (intermediate level), which takes about 7 hours. What happens in a forward pass.

Back to the full Inference Engineer roadmap