Serving Engines for Inference Engineers
Serving engines are specialised runtimes for LLM inference. vLLM, SGLang and TGI provide continuous batching, paged KV cache memory, prefix caching and an OpenAI-compatible API. Disaggregated serving separates prefill and decode onto different workers so each phase can be scaled and tuned independently.
Stage: Serving Systems · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Serving Engines
- Launch a model on vLLM and call its OpenAI-compatible endpoint
- Explain continuous batching versus static batching
- Describe PagedAttention and how it limits KV cache fragmentation
- Use prefix caching to reuse shared prompt prefixes
- Explain why splitting prefill and decode can improve latency targets
Serving Engines interview questions
What is continuous batching?
Continuous batching schedules work at the token-step level: finished sequences leave the batch immediately and waiting requests join. This keeps the GPU full and avoids idle time while one long request holds a static batch.
What problem does PagedAttention solve?
It stores the KV cache in fixed-size blocks that need not be contiguous, similar to virtual memory paging. This removes fragmentation and over-reservation, so more concurrent sequences fit in GPU memory and sequences can share blocks.
Where Serving Engines fits in the Inference Engineer roadmap
Serving Engines belongs to the Serving Systems stage (advanced level), which takes about 9 hours. Production inference.
Other units in this stage
Back to the full Inference Engineer roadmap