Skip to main content

Quantisation for Inference Engineers

Quantisation stores model weights, and sometimes activations or the KV cache, in lower-precision numbers such as INT8, FP8 or 4-bit formats to cut memory and often raise speed. Methods like GPTQ, AWQ and SmoothQuant choose how to round with minimal accuracy loss, which must be measured on your tasks.

Stage: Model Optimisation · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026

What to learn in Quantisation

Quantisation interview questions

Why can weight-only quantisation speed up decoding?

Decoding is memory-bandwidth bound, so shrinking weights from 16-bit to 4-bit reduces bytes read per token. Even if weights are dequantised on the fly, less data moves from memory, so throughput can improve.

How do you check whether quantisation hurt a model?

Compare the quantised model to the baseline on perplexity and on task-specific evaluations that reflect real use, including long-context and reasoning cases. Aggregate scores can hide regressions, so inspect specific failures and sample outputs.

Where Quantisation fits in the Inference Engineer roadmap

Quantisation belongs to the Model Optimisation stage (advanced level), which takes about 5 hours. Smaller, faster models.

Back to the full Inference Engineer roadmap