Skip to main content

Evals for AI Engineers

Evals are repeatable tests that score an AI system's outputs against expectations, so changes to prompts, models or retrieval can be judged by data. They include datasets with reference answers, automatic graders, LLM-as-judge rubrics and specialised checks for RAG and agents. Evals are the stage that separates a demo from a product.

Stage: Evaluation and Observability · Part of the AI Engineer roadmap · Last reviewed 19 November 2026

What to learn in Evals

Evals interview questions

How do you check that an LLM-as-judge is trustworthy?

Have humans label a sample, then measure agreement with the judge using accuracy or Cohen's kappa. Watch for known biases such as favouring longer answers or its own model family, use clear scored rubrics, and recalibrate whenever the judge model changes.

What is the difference between retrieval metrics and faithfulness in RAG evaluation?

Retrieval metrics such as recall at k and MRR test whether the right chunks were found. Faithfulness tests whether the generated answer is supported by the retrieved text. A system can retrieve well yet invent details, or answer faithfully from the wrong chunk, so both are needed.

Where Evals fits in the AI Engineer roadmap

Evals belongs to the Evaluation and Observability stage (intermediate level), which takes about 48 hours. The skill that separates demos from products.

Back to the full AI Engineer roadmap