Observability for AI Engineers
Observability for AI systems is the ability to see what happened inside each request: prompts, retrieved chunks, tool calls, token counts, latency, cost and errors. It uses traces and metrics, and it feeds user feedback and failure cases back into evals. On the AI Engineer path it keeps a deployed system improvable.
Stage: Evaluation and Observability · Part of the AI Engineer roadmap · Last reviewed 19 November 2026
What to learn in Observability
- Instrument an LLM application so each request produces a trace with nested spans
- Record tokens, cost and latency per call and chart them over time
- Alert on error rate, latency percentiles and sudden cost increases
- Capture thumbs-up and thumbs-down feedback linked to the originating trace
- Turn a failing production trace into a new test case in the eval set
Observability interview questions
What should a trace of a RAG request contain to debug a bad answer?
The user query, any rewritten query, retrieved chunk IDs with scores, the final prompt as sent, model name and parameters, the response, tool calls, token counts and timings per span. With that you can replay the request and see whether retrieval or generation failed.
Why track latency percentiles such as p95 instead of only the average for an LLM feature?
LLM latency has a long tail because output length and queueing vary. An average hides the slow responses users actually notice. P95 and p99 expose the tail, and tracking time to first token separately shows the perceived speed of streamed output.
Where Observability fits in the AI Engineer roadmap
Observability belongs to the Evaluation and Observability stage (intermediate level), which takes about 48 hours. The skill that separates demos from products.
Other units in this stage
Back to the full AI Engineer roadmap