Evaluation for LLM and RAG Engineers
LLM evaluation is the systematic measurement of whether a model application gives correct, useful and safe output. It uses a curated test set, automatic metrics, human review and LLM-as-judge scoring, and for RAG it separates retrieval metrics like recall from answer metrics like faithfulness.
Stage: Production · Part of the LLM and RAG Engineer roadmap · Last reviewed 19 November 2026
What to learn in Evaluation
- Build a golden dataset of realistic questions with expected answers
- Measure retrieval with recall at k and mean reciprocal rank
- Score answers for faithfulness and relevance with a rubric-based LLM judge
- Run evaluations automatically on each prompt or model change
- Calibrate automatic judges against a sample of human labels
Evaluation interview questions
What is LLM-as-a-judge and what are its limits?
It uses a strong model to grade outputs against a rubric, which scales better than human review. Limits include bias toward longer answers, sensitivity to rubric wording and self preference. Validate judge scores against human labels on a sample before trusting them.
Why do you need an evaluation set before changing a prompt?
Without a fixed set of test cases, you cannot tell whether a change improved or broke behaviour, since outputs vary and anecdotes mislead. A stable set gives a baseline, so every prompt, model or retrieval change can be compared objectively.
Where Evaluation fits in the LLM and RAG Engineer roadmap
Evaluation belongs to the Production stage (advanced level). Operate it.
Other units in this stage
Back to the full LLM and RAG Engineer roadmap