Skip to main content

Model Internals for Inference Engineers

Model internals for inference means understanding what a transformer does at serving time. Generation has a compute-heavy prefill phase that processes the prompt and a memory-bound decode phase that emits one token at a time. Mixture-of-experts models route tokens to expert subnetworks, and long-context models stress attention memory.

Stage: Foundations · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026

What to learn in Model Internals

Model Internals interview questions

Why is decoding usually memory-bandwidth bound?

Each new token requires reading all the model weights and the KV cache from GPU memory to do a small amount of arithmetic for a single sequence. The arithmetic intensity is low, so memory bandwidth, not compute, limits speed at small batch sizes.

How does grouped-query attention help inference?

Grouped-query attention shares key and value heads across several query heads, shrinking the KV cache and the memory traffic per token. This allows longer contexts and larger batches with only a small effect on model quality.

Where Model Internals fits in the Inference Engineer roadmap

Model Internals belongs to the Foundations stage (intermediate level), which takes about 7 hours. What happens in a forward pass.

Back to the full Inference Engineer roadmap