Skip to main content

Serving Engines for Inference Engineers

Serving engines are specialised runtimes for LLM inference. vLLM, SGLang and TGI provide continuous batching, paged KV cache memory, prefix caching and an OpenAI-compatible API. Disaggregated serving separates prefill and decode onto different workers so each phase can be scaled and tuned independently.

Stage: Serving Systems · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026

What to learn in Serving Engines

Serving Engines interview questions

What is continuous batching?

Continuous batching schedules work at the token-step level: finished sequences leave the batch immediately and waiting requests join. This keeps the GPU full and avoids idle time while one long request holds a static batch.

What problem does PagedAttention solve?

It stores the KV cache in fixed-size blocks that need not be contiguous, similar to virtual memory paging. This removes fragmentation and over-reservation, so more concurrent sequences fit in GPU memory and sequences can share blocks.

Where Serving Engines fits in the Inference Engineer roadmap

Serving Engines belongs to the Serving Systems stage (advanced level), which takes about 9 hours. Production inference.

Back to the full Inference Engineer roadmap