Compression for Inference Engineers
Compression beyond quantisation includes pruning, which removes weights or structures, and distillation, which trains a smaller student model to imitate a larger teacher. Speculative and parallel decoding reduce latency by having a cheap draft propose several tokens that the large model verifies in one pass.
Stage: Model Optimisation · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Compression
- Differentiate unstructured and structured pruning and their hardware effects
- Describe how knowledge distillation trains a student from teacher outputs
- Explain how speculative decoding keeps output identical to the target model
- Estimate speedup from draft acceptance rate and draft cost
- Compare Medusa-style multi-head and separate draft-model approaches
Compression interview questions
How does speculative decoding preserve the target model's output?
A small draft model proposes several tokens, and the large model scores them in one forward pass. Tokens are accepted or rejected using a rule that keeps the sampling distribution identical to the large model alone, so quality is unchanged.
Why is unstructured pruning often disappointing on GPUs?
Randomly zeroed weights leave dense matrix shapes unchanged, and GPUs gain speed only from regular structure. Unless the sparsity follows a supported pattern such as 2:4, the zeros save little time even though they reduce parameter counts.
Where Compression fits in the Inference Engineer roadmap
Compression belongs to the Model Optimisation stage (advanced level), which takes about 5 hours. Smaller, faster models.
Other units in this stage
Back to the full Inference Engineer roadmap