Serving and Cost for AI Engineers
Serving and cost covers how model inference is run in production and how to keep latency and spend under control. Options include hosted APIs, managed endpoints and self-hosting with engines such as vLLM, together with caching, batching, quantisation and model routing. It is the operating-cost stage of the AI Engineer path.
Stage: Customisation and Serving · Part of the AI Engineer roadmap · Last reviewed 19 November 2026
What to learn in Serving and Cost
- Compare API, managed endpoint and self-hosted inference on cost and control
- Estimate monthly spend from request volume and average input and output tokens
- Cache responses and prompt prefixes to cut repeated work
- Route easy requests to a small model and hard ones to a larger model
- Explain how quantisation trades a little accuracy for lower memory use
Serving and Cost interview questions
How would you estimate whether self-hosting beats a hosted API?
Compare per-token API pricing at your volume with GPU hourly cost divided by achieved throughput, including idle time, engineering effort and redundancy. Self-hosting usually wins only at high, steady utilisation or when data must remain inside your own environment.
What is continuous batching and why does it raise throughput?
Instead of waiting for a whole batch to finish, the server adds new requests and removes finished ones at each decoding step. The GPU stays busy and short requests are not held behind long ones. Engines such as vLLM pair it with paged attention for the KV cache.
Where Serving and Cost fits in the AI Engineer roadmap
Serving and Cost belongs to the Customisation and Serving stage (advanced level), which takes about 48 hours. When prompts are not enough.
Other units in this stage
Back to the full AI Engineer roadmap