Skip to main content

Compression for Inference Engineers

Compression beyond quantisation includes pruning, which removes weights or structures, and distillation, which trains a smaller student model to imitate a larger teacher. Speculative and parallel decoding reduce latency by having a cheap draft propose several tokens that the large model verifies in one pass.

Stage: Model Optimisation · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026

What to learn in Compression

Compression interview questions

How does speculative decoding preserve the target model's output?

A small draft model proposes several tokens, and the large model scores them in one forward pass. Tokens are accepted or rejected using a rule that keeps the sampling distribution identical to the large model alone, so quality is unchanged.

Why is unstructured pruning often disappointing on GPUs?

Randomly zeroed weights leave dense matrix shapes unchanged, and GPUs gain speed only from regular structure. Unless the sparsity follows a supported pattern such as 2:4, the zeros save little time even though they reduce parameter counts.

Where Compression fits in the Inference Engineer roadmap

Compression belongs to the Model Optimisation stage (advanced level), which takes about 5 hours. Smaller, faster models.

Back to the full Inference Engineer roadmap