Model Serving for MLOps Engineers
Model serving is exposing a trained model for predictions with predictable latency and throughput. Serving stacks include FastAPI wrappers, BentoML, TorchServe, KServe and Triton, and optimisation techniques such as batching, quantisation and caching. This unit opens the Serving and Infrastructure stage.
Stage: Serving and Infrastructure · Part of the MLOps Engineer roadmap · Last reviewed 19 November 2026
What to learn in Model Serving
- Serve a model through an HTTP endpoint with request validation
- Use dynamic batching to improve throughput on a GPU
- Reduce model size and latency with quantisation or distillation
- Measure p50 and p99 latency and requests per second under load
- Choose between online, batch and streaming inference for a requirement
Model Serving interview questions
What is dynamic batching in model serving?
The server briefly holds incoming requests and groups them into one batch for a single forward pass. Accelerators are far more efficient on batches, so throughput rises, at the cost of a small extra wait added to each request.
Why look at p99 latency rather than the average?
Averages hide slow outliers. P99 is the time within which 99 percent of requests finish, so it reflects what the unluckiest users experience and often sets the service level objective, especially when many calls are chained.
Where Model Serving fits in the MLOps Engineer roadmap
Model Serving belongs to the Serving and Infrastructure stage (intermediate level), which takes about 6 hours. Low latency, high throughput.
Other units in this stage
Back to the full MLOps Engineer roadmap