Skip to main content

Serving and Cost for AI Engineers

Serving and cost covers how model inference is run in production and how to keep latency and spend under control. Options include hosted APIs, managed endpoints and self-hosting with engines such as vLLM, together with caching, batching, quantisation and model routing. It is the operating-cost stage of the AI Engineer path.

Stage: Customisation and Serving · Part of the AI Engineer roadmap · Last reviewed 19 November 2026

What to learn in Serving and Cost

Serving and Cost interview questions

How would you estimate whether self-hosting beats a hosted API?

Compare per-token API pricing at your volume with GPU hourly cost divided by achieved throughput, including idle time, engineering effort and redundancy. Self-hosting usually wins only at high, steady utilisation or when data must remain inside your own environment.

What is continuous batching and why does it raise throughput?

Instead of waiting for a whole batch to finish, the server adds new requests and removes finished ones at each decoding step. The GPU stays busy and short requests are not held behind long ones. Engines such as vLLM pair it with paged attention for the KV cache.

Where Serving and Cost fits in the AI Engineer roadmap

Serving and Cost belongs to the Customisation and Serving stage (advanced level), which takes about 48 hours. When prompts are not enough.

Back to the full AI Engineer roadmap