Profiling for Inference Engineers
Profiling is measuring inference performance correctly before optimising. Key metrics are time to first token, inter-token latency, throughput in tokens per second and GPU utilisation. Tools such as Nsight Systems and the PyTorch profiler show kernel timings, and correct benchmarks control warm-up, batch size and sequence lengths.
Stage: Performance Engineering · Part of the Inference Engineer roadmap · Last reviewed 19 November 2026
What to learn in Profiling
- Measure time to first token and inter-token latency separately
- Profile a model run with Nsight Systems or the PyTorch profiler
- Warm up and synchronise the GPU so timings are valid
- Benchmark with a realistic distribution of prompt and output lengths
- Tell latency-oriented from throughput-oriented measurement goals
Profiling interview questions
Why must you synchronise the GPU when timing code?
CUDA calls are asynchronous, so the CPU returns before the GPU finishes. Without a synchronise call or CUDA events, timers measure only launch time. Wrap measurements with synchronisation after warm-up iterations.
What is the difference between latency and throughput?
Latency is how long one request takes, such as time to first token. Throughput is how much work is completed per unit time, such as tokens per second across all requests. Larger batches usually raise throughput but increase per-request latency.
Where Profiling fits in the Inference Engineer roadmap
Profiling belongs to the Performance Engineering stage (advanced level), which takes about 5 hours. Make it fast.
Other units in this stage
Back to the full Inference Engineer roadmap