You change a prompt. Is the system better or worse?
For ordinary software, tests answer this. For a model, there is no exact expected output — a hundred phrasings can be equally correct, and a fluent, well-formed answer can be entirely wrong.
Without evaluation you are shipping on vibes: someone tries five questions, decides it feels better, and deploys. That works until the change that felt better silently broke a case nobody tried.
Evaluation is the discipline that makes AI systems engineerable. It is also the part teams skip, and the reason prototypes stall before production.
The eval set
Everything starts here: a collection of inputs with known-good outcomes, representative of real traffic.
Building one is unglamorous and decisive. Practical guidance:
Start small and real. Fifty genuine user questions beat five hundred invented ones. Take them from production logs, support tickets, or user research.
Cover the tail deliberately. Naive sampling gives you the easy cases. Include ambiguous inputs, adversarial ones, questions with no good answer, and the edge cases that generated complaints.
Record expected outcomes, not exact strings. For factual questions, the fact. For classification, the label. For open generation, the criteria a good answer must meet.
Version it. The set will grow — every production bug should become a case, exactly as with regression tests.
Four ways to score
Deterministic checks
Where an exact answer exists, compare directly: classification labels, extracted fields, schema validity, whether a required citation is present.
Cheap, fast, unambiguous. Use these wherever the task permits — and structured outputs, from the earlier lesson, dramatically expand where that is.
Reference-based similarity
Compare against a known-good answer using overlap or embedding similarity. Useful for translation and summarisation; weak in general, because a correct answer phrased differently scores badly and a wrong answer phrased similarly scores well.
Model-as-judge
Use a strong model to score outputs against criteria. This is what makes evaluating open-ended generation practical at all.
It works — with discipline:
Give explicit criteria, not "is this good?" Score named dimensions: is it grounded in the provided context, does it answer the question asked, is the tone appropriate.
Use a small scale. Binary or three-point judgements are far more consistent than one-to-ten.
Require reasoning before the score, for the same reason as the structured outputs lesson — reasoning after a score is rationalisation.
Validate the judge against humans. Score a sample both ways and check agreement. An unvalidated judge is a number of unknown meaning.
Known biases worth guarding against: judges tend to favour longer answers, and favour outputs from the same model family. Randomise position when comparing two candidates.
Human review
The ground truth, and it does not scale. Use it to validate judges, to build the initial set, and to sample production periodically — not as your routine mechanism.
What to measure
| Dimension | Question | Method |
|---|---|---|
| Correctness | Is the answer right? | Deterministic or judge |
| Groundedness | Is every claim supported by the context? | Judge, per claim |
| Relevance | Does it answer what was asked? | Judge |
| Retrieval quality | Were the right passages fetched? | Recall@k against labels |
| Format | Valid schema, citations present? | Deterministic |
| Safety | Refuses what it should? | Deterministic plus judge |
| Cost and latency | Within budget? | Measured |
Groundedness deserves emphasis for RAG systems. It asks a narrower, far more checkable question than "is this correct": is each statement supported by the retrieved context? A judge can verify that claim by claim, and it catches the specific failure RAG exists to prevent.
Evaluating agents is different
For an agent, the final answer is not the whole story. A correct answer reached by calling nine unnecessary tools, or by taking a destructive action that happened to work, is not a success.
Evaluate the trajectory:
- Did it choose appropriate tools?
- Were the arguments valid?
- How many steps, and how many were wasted?
- What did it cost, and how long did it take?
- Did it respect policy — no forbidden actions, confirmation where required?
A system scoring well on final answers and badly on trajectory is expensive and dangerous, and you cannot see either from the output alone.
Putting it in the pipeline
Evaluation earns its keep when it gates deploys.
Diagram, as text
- Prompt or model change leads to Run eval set
- Run eval set leads to Meets thresholds?
- Meets thresholds? leads to Deploy (yes)
- Meets thresholds? leads to Block, show regressions (no)
Set thresholds per dimension, run automatically on every change to a prompt, model version, retrieval configuration or tool definition — all of which alter behaviour — and report per-case diffs rather than one aggregate number. "Score fell from 0.84 to 0.81" is not actionable; "these six cases regressed" is.
Online evaluation complements this. Sample real traffic, score it with a judge, and track the trend. Offline sets go stale; production tells you what is actually happening, and the observability lesson covers wiring that up.
Check your understanding
Common mistakes
No eval set. Shipping on impressions; the default state of stalled AI projects.
Only easy cases. High scores that predict nothing about production.
An unvalidated judge. A number whose relationship to quality is unknown.
One aggregate score. Hides which cases regressed.
Evaluating only final answers for agents. Misses wasteful and dangerous trajectories.
Never re-sampling from production. Offline sets drift away from real traffic.
Treating evaluation as a phase. It is continuous, like monitoring.
Practice
Build the evaluation strategy for a summarisation feature: what you measure, how, and how the set grows over time.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Summarisation has no single correct answer, so decide what 'wrong' means before choosing a method.
Show a strong answer
Define failure before defining quality. For summarisation the failures are specific and mostly checkable:
- Fabrication — a claim not supported by the source. The most serious, and the most checkable.
- Omission — a key point missing.
- Length and format violations.
- Tone or register wrong for the audience.
How to measure each, cheapest method first:
- Format and length — deterministic checks. Free, unambiguous, run on every case.
- Fabrication — a targeted check, not a general quality score. Extract the claims from the summary and verify each is entailed by the source. This can be done with a model in a narrow role, which is far more reliable than asking "is this summary good?"
- Omission — against a reference. For a subset of cases, have a human list the points that must appear, then check coverage. Expensive, so keep this set small and stable.
- Tone — model-as-judge, validated. Score a sample with both the judge and human raters and measure agreement before trusting it. If agreement is poor, the rubric is too vague; sharpen it rather than abandoning the method.
How the set grows — the part that determines long-term value:
- Every production failure becomes a case. A user reporting a bad summary contributes the input, the output and what was wrong with it. Within months this is the most valuable asset the team owns.
- Keep the set balanced. If every added case is a failure, the set stops representing typical traffic and scores become uninterpretable. Add routine cases too.
- Version it alongside the prompt, so a score is always attributable to a specific set and a specific prompt.
What I would resist: a single overall quality number. It moves for reasons nobody can attribute, and it hides a fabrication regression behind a tone improvement.
Your evaluation set has 200 cases and passes at 94%. A change takes it to 96%. Should you ship it?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Work out how many cases that difference actually represents.
Show a strong answer
Do the arithmetic first. 94% of 200 is 188 passes and 12 failures. 96% is 192 and 8. The difference is four cases.
With a sample of 200, that difference is well within what random variation produces. Re-running the same evaluation with a non-zero temperature could plausibly move it by as much. So the honest answer is that this result does not demonstrate an improvement.
What I would do before deciding:
- Run both arms several times and look at the spread. If 94% and 96% overlap across runs, there is no signal to act on.
- Look at which four cases changed. This matters more than the number. If the change fixed four cases in the same category, that is a real, explainable improvement even at this sample size. If four unrelated cases flipped — and especially if any previously-passing case now fails — it is noise, and the "improvement" may be a regression in disguise.
- Check whether anything regressed. A net +4 could be +6 and −2, and the two that broke may matter more than the six that were fixed.
Would I ship it? If the change is otherwise low-risk — a clearer prompt, a fix for a known problem — then yes, on its own merits, while being honest that the evaluation did not prove anything. If the change carries cost or complexity and this is the sole justification, no: the evidence does not support it.
The lasting fix is a bigger set. At 200 cases you cannot resolve differences below several percentage points. If the team needs to detect a two-point change, that requires substantially more cases — and knowing that in advance stops months of arguing about noise.
Key takeaways
- Without evaluation you are shipping on impressions; it is what makes AI systems engineerable.
- Build the set from real traffic, cover the tail deliberately, and turn every incident into a case.
- Prefer deterministic checks where possible; structured outputs widen where that is possible.
- Model-as-judge makes open-ended evaluation practical — with explicit criteria, small scales, reasoning before scoring, and validation against humans.
- Groundedness is the key RAG metric: narrower and far more checkable than general correctness.
- Agents need trajectory evaluation — tool choice, wasted steps, cost, policy compliance — not just final answers.
- Hold out cases you do not develop against, or you overfit your prompts to the test.
- Gate deploys on evaluation, report per-case regressions, and sample production continuously.
Further reading
- Lianmin Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023) — including position and verbosity biases.
- Percy Liang et al., "Holistic Evaluation of Language Models (HELM)" (TMLR, 2023).
- Chin-Yew Lin, "ROUGE: A Package for Automatic Evaluation of Summaries" (ACL workshop, 2004) — and its well-known limitations.
- Shahul Es et al., "RAGAS" (EACL 2024) — faithfulness and answer-relevance scoring.
- Ronny Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020) — sample size and how to avoid shipping noise.