Evals for AI Engineers
Evals are repeatable tests that score an AI system's outputs against expectations, so changes to prompts, models or retrieval can be judged by data. They include datasets with reference answers, automatic graders, LLM-as-judge rubrics and specialised checks for RAG and agents. Evals are the stage that separates a demo from a product.
Stage: Evaluation and Observability · Part of the AI Engineer roadmap · Last reviewed 19 November 2026
What to learn in Evals
- Build a dataset of real inputs with expected outputs or grading rubrics
- Write a rubric for an LLM judge and check its agreement with human labels
- Measure retrieval with recall at k separately from answer faithfulness
- Score an agent on task success and on the steps or tool calls it needed
- Run evals in CI so a prompt change that lowers a score blocks the merge
Evals interview questions
How do you check that an LLM-as-judge is trustworthy?
Have humans label a sample, then measure agreement with the judge using accuracy or Cohen's kappa. Watch for known biases such as favouring longer answers or its own model family, use clear scored rubrics, and recalibrate whenever the judge model changes.
What is the difference between retrieval metrics and faithfulness in RAG evaluation?
Retrieval metrics such as recall at k and MRR test whether the right chunks were found. Faithfulness tests whether the generated answer is supported by the retrieved text. A system can retrieve well yet invent details, or answer faithfully from the wrong chunk, so both are needed.
Where Evals fits in the AI Engineer roadmap
Evals belongs to the Evaluation and Observability stage (intermediate level), which takes about 48 hours. The skill that separates demos from products.
Other units in this stage
Back to the full AI Engineer roadmap