Model Internals for Inference Engineers
Model internals for inference means understanding what a transformer does at serving time. Generation has a compute-heavy prefill phase that processes the prompt and a memory-bound decode phase that emits one token at a time. Mixture-of-experts models route tokens to expert subnetworks, and long-context models stress attention memory.
Stage: Foundations · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Model Internals
- Explain the prefill and decode phases and which resource limits each
- Calculate parameter memory from model size and numeric precision
- Describe how mixture-of-experts routing changes active parameters per token
- Estimate why attention cost grows with context length
- Identify where multi-head, grouped-query and multi-query attention differ
Model Internals interview questions
Why is decoding usually memory-bandwidth bound?
Each new token requires reading all the model weights and the KV cache from GPU memory to do a small amount of arithmetic for a single sequence. The arithmetic intensity is low, so memory bandwidth, not compute, limits speed at small batch sizes.
How does grouped-query attention help inference?
Grouped-query attention shares key and value heads across several query heads, shrinking the KV cache and the memory traffic per token. This allows longer contexts and larger batches with only a small effect on model quality.
Where Model Internals fits in the Inference Engineer roadmap
Model Internals belongs to the Foundations stage (intermediate level), which takes about 7 hours. What happens in a forward pass.
Other units in this stage
Back to the full Inference Engineer roadmap