Quantisation for Inference Engineers
Quantisation stores model weights, and sometimes activations or the KV cache, in lower-precision numbers such as INT8, FP8 or 4-bit formats to cut memory and often raise speed. Methods like GPTQ, AWQ and SmoothQuant choose how to round with minimal accuracy loss, which must be measured on your tasks.
Stage: Model Optimisation · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Quantisation
- Compare FP16, BF16, FP8, INT8 and 4-bit formats by range and size
- Explain post-training quantisation versus quantisation-aware training
- Apply a method such as AWQ or GPTQ to a model
- Quantise the KV cache and judge the effect on long contexts
- Evaluate quality loss with perplexity and task benchmarks
Quantisation interview questions
Why can weight-only quantisation speed up decoding?
Decoding is memory-bandwidth bound, so shrinking weights from 16-bit to 4-bit reduces bytes read per token. Even if weights are dequantised on the fly, less data moves from memory, so throughput can improve.
How do you check whether quantisation hurt a model?
Compare the quantised model to the baseline on perplexity and on task-specific evaluations that reflect real use, including long-context and reasoning cases. Aggregate scores can hide regressions, so inspect specific failures and sample outputs.
Where Quantisation fits in the Inference Engineer roadmap
Quantisation belongs to the Model Optimisation stage (advanced level), which takes about 5 hours. Smaller, faster models.
Other units in this stage
Back to the full Inference Engineer roadmap