Scaling for Inference Engineers
Scaling inference means serving more traffic or larger models across several GPUs and machines. Tensor, pipeline and expert parallelism split a model, while replicas multiply throughput. Autoscaling and scheduling match capacity to demand, and routers send requests to the right model with fallbacks when one fails.
Stage: Serving Systems · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Scaling
- Contrast tensor parallelism with pipeline parallelism and their communication costs
- Choose parallelism degree for a model that exceeds one GPU
- Scale replicas on queue depth or GPU utilisation signals
- Handle cold starts when loading large weights onto new nodes
- Route requests by model or cost and fall back on failure
Scaling interview questions
When would you use tensor parallelism rather than replicas?
Use tensor parallelism when a model's weights or KV cache do not fit on one GPU, or you need lower per-request latency. Use replicas when the model fits and the goal is more throughput, because replicas need no inter-GPU communication.
Why is autoscaling GPU inference harder than scaling web servers?
GPU instances are expensive and slow to start, since weights must be downloaded and loaded. Requests also vary in length, so CPU-style metrics mislead. Scaling needs queue-based signals, warm capacity and careful scale-down.
Where Scaling fits in the Inference Engineer roadmap
Scaling belongs to the Serving Systems stage (advanced level), which takes about 9 hours. Production inference.
Other units in this stage
Back to the full Inference Engineer roadmap