Skip to main content

Model Serving for MLOps Engineers

Model serving is exposing a trained model for predictions with predictable latency and throughput. Serving stacks include FastAPI wrappers, BentoML, TorchServe, KServe and Triton, and optimisation techniques such as batching, quantisation and caching. This unit opens the Serving and Infrastructure stage.

Stage: Serving and Infrastructure · Part of the MLOps Engineer roadmap · Last reviewed 19 November 2026

What to learn in Model Serving

Model Serving interview questions

What is dynamic batching in model serving?

The server briefly holds incoming requests and groups them into one batch for a single forward pass. Accelerators are far more efficient on batches, so throughput rises, at the cost of a small extra wait added to each request.

Why look at p99 latency rather than the average?

Averages hide slow outliers. P99 is the time within which 99 percent of requests finish, so it reflects what the unluckiest users experience and often sets the service level objective, especially when many calls are chained.

Where Model Serving fits in the MLOps Engineer roadmap

Model Serving belongs to the Serving and Infrastructure stage (intermediate level), which takes about 6 hours. Low latency, high throughput.

Back to the full MLOps Engineer roadmap