Evaluating a model means checking an output. Evaluating an agent means checking a process that took twenty steps, called eight tools, changed real state, and will do something different next time you run it.
Everything that makes output evaluation tractable — one input, one output, deterministic scoring — is gone. This page is about what replaces it.
Why the final answer is not enough
Score only the endpoint and you accept results you should reject.
The right answer by a terrible path. An agent that solves the task in forty steps, having deleted and recreated a file three times, scores identically to one that took four. It will not survive a slightly harder variant.
Damage the score cannot see. It answered correctly and dropped a table on the way. Endpoint scoring is blind to side effects.
Luck. Non-determinism means one success proves very little. Run it again.
All-or-nothing. A four-of-five-steps run scores zero, which tells you nothing about whether you are close.
So evaluate two things: did it succeed, and how did it get there.
Trajectory metrics
The trajectory is the sequence of observations, decisions and actions. Useful things to measure over it:
| Metric | Question | Signals |
|---|---|---|
| Task success | Did the end state match the goal? | The headline number |
| Step efficiency | Steps taken versus a reasonable path | Flailing, wasted budget |
| Tool selection | Right tool at each decision? | Bad descriptions or too many tools |
| Redundant actions | Repeated or self-cancelling work | Loop and memory problems |
| Recovery rate | Failures recovered from versus abandoned | Error-message quality |
| Side effects | Anything changed outside the goal | Safety and blast radius |
Recovery rate deserves attention. Agents will encounter failures; what separates a good system from a bad one is usually what happens after the first failed action. It is also one of the most improvable numbers, because it responds directly to better error messages rather than a better model.
Grade the end state, not the exact path. There are many legitimate routes to a goal, and scoring against one blessed sequence punishes valid strategies. Where you need to constrain the path, assert checkpoints — "the tests were run before the change was committed" — rather than a full script.
Measure reliability, not accuracy
Diagram, as text
- Case from production history leads to Run N times
- Run N times leads to Tests pass?
- Tests pass? leads to Failure: record mode (no)
- Tests pass? leads to Diff scope clean? (yes)
- Diff scope clean? leads to Failure: record mode (no)
- Diff scope clean? leads to Pass (yes)
- Pass leads to Report distribution, cost, iterations
- Failure: record mode leads to Report distribution, cost, iterations
The single most important shift when moving from model evaluation to agent evaluation.
A model's score on a fixed input is roughly stable. An agent's is not: the same task, the same prompt, run five times, can produce five different trajectories and a mix of outcomes. A single pass tells you the task is possible, not that it is reliable.
So run each task several times and report the distribution. The distinction that matters:
- At least one success in k runs flatters the agent. It answers "can this ever work?"
- Every run succeeding out of k is the number that predicts production behaviour. It answers "can I depend on this?"
The gap between them is usually large and is exactly the reliability problem. An agent that succeeds on most single runs may succeed on very few five-for-five runs — and users experience something closer to the second figure, because they do not get to retry silently.
Building an evaluation set
Use real tasks. Failures from production, support tickets, actual work. Tasks invented alongside the agent encode the same assumptions the agent has and will flatter it.
Make the environment deterministic. Fixed fixtures, seeded data, a reset between runs, sandboxed side effects. Without reset, run three is polluted by run two. This is the largest engineering cost in agent evaluation and the one most often skipped.
Start small. Twenty representative tasks catch more than most teams expect, because early effect sizes are large. Grow it as failures teach you what is missing.
Hold out. Tasks you tuned against stop measuring anything. Keep a set you only look at before shipping.
Judging with a model
Some qualities need judgement — was the reasoning sound, was the summary faithful. A model can grade these, with care.
Give it a rubric, not "is this good?". Named criteria and a scale produce far more consistent scores.
Keep the judge independent. A model grading its own family's output shares its blind spots and rates it generously. Different family where you can; deterministic checks where possible.
Calibrate against humans. Score a sample by hand and compare. If judge and human disagree often, the judge is producing numbers, not measurements.
Prefer deterministic checks. A test run beats a judge on every axis — cheaper, faster, uncorrelated with the agent. Use a judge only where no such check exists.
In production
Offline sets do not cover real-world distribution. Watch signals that need no labelling:
Intervention rate — how often a human corrects, rejects or takes over. The best single health metric, and it moves before your evaluations do.
Abandonment — users who stop mid-task.
Rework — accepted output that gets immediately edited.
Step and cost distributions — a rising tail means new failure modes.
Trace enough to reconstruct a failure: context, tool calls, results, decisions. A failure you cannot replay is a failure you cannot fix.
Check your understanding
Common mistakes
Scoring only the final answer. Blind to path quality, side effects and luck.
One run per task. Measures possibility, not reliability.
Evaluating on the tasks you built the agent for. They encode the same assumptions.
No environment reset. Runs contaminate each other and results stop meaning anything.
Scoring against one blessed path. Punishes valid alternative strategies.
A judge from the same family as the agent. Shared blind spots, generous scores.
Only inspecting failures. Reward hacking hides in the passes.
Treating a model upgrade as routine. It is the highest-risk change you can make.
Reporting a mean with no variance. For a non-deterministic system that is barely a measurement.
Practice
Design the evaluation for a coding agent. State the metrics, how cases are built, and what a regression means.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
For coding there is a deterministic oracle available, which changes the design.
Show a strong answer
Take advantage of the oracle: the tests. Unlike most agent tasks, correctness here is checkable without judgement, so build everything around that.
Metrics:
- Success rate over N runs per case (N = 5 or 10), not a single pass. Report the distribution, not the mean alone.
- Tests pass — the primary signal, deterministic.
- No unrelated files modified. An agent that fixes the bug and reformats twelve other files has not succeeded, and diff scope catches it cheaply.
- Cost and iteration count per success, because an agent that succeeds in 40 iterations is different from one that succeeds in 4.
- Failure mode when it fails — gave up, hit the cap, produced a wrong fix that passes no tests, or produced a fix that passes tests but is wrong.
How cases are built:
- From real history. Take merged pull requests that fixed a bug, revert the fix, and keep the test that was added. The known-good outcome comes free, and the distribution is genuinely yours rather than a benchmark's.
- Include cases the agent should refuse or escalate — an ambiguous issue, a task requiring a product decision. An agent that confidently guesses on these is failing, and a set with no such cases cannot detect it.
- Grow from production failures, every one becoming a case.
What a regression means, and this needs deciding in advance:
- A previously-passing case failing consistently — a genuine regression, block the change.
- A case moving from 9/10 to 7/10 — a reliability regression, which is real and easy to dismiss. Treat it as a failure, or you will accumulate flakiness invisibly.
- A case moving from 1/10 to 0/10 — barely meaningful; it was never working.
The thing to hold firm on: run the suite on every prompt, tool or model change. Agent behaviour is emergent, and a tool-description edit can change success rates on tasks that appear unrelated to it.
Your agent scores 85% on the evaluation set but users report it 'often gets things wrong'. Explain how both can be true.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The set and the traffic may be different distributions.
Show a strong answer
Several explanations, and they can compound.
- The set does not represent real traffic. Cases were written by the team, who imagine well-formed requests. Real users write ambiguous, terse or compound requests. If the set is easier than production, 85% on it and a poor user experience are entirely consistent — and this is the most common cause.
- The failures are concentrated where it matters. 15% failure spread evenly is annoying; 15% concentrated on the most common request type means most users hit it. Score by traffic-weighted frequency, not by case count, or a rare-but-easy case counts the same as the one everyone uses.
- Success is defined too generously. If a case passes when the answer is approximately right, users applying a stricter standard will disagree. Check what a passing case actually looks like — reading twenty passes is often more revealing than reading the failures.
- Multi-turn reality versus single-turn evaluation. If each case is one exchange but users have long conversations, per-turn success of 85% compounds: five turns at 85% is 44% of conversations with at least one failure. This is usually the largest factor and the least obvious, because the per-turn number looks respectable.
- "Often" may mean something specific. A user who hits the same failure three times in a row perceives it as constant, even at a low overall rate. Look for whether failures cluster per user or per input type.
What I would do: sample real conversations, score them with the same rubric, and compare the distribution against the evaluation set. The gap between the two numbers is the finding — and closing it usually means rebuilding the set from production traffic rather than improving the agent.
Key takeaways
- Evaluate the process and the outcome — the endpoint alone hides bad paths, side effects and luck.
- Track success, efficiency, tool selection, redundancy, recovery and side effects; recovery rate is the most improvable.
- Grade the end state with checkpoints, not one blessed sequence of steps.
- Reliability, not accuracy. Run tasks repeatedly; the every-run-succeeds rate is what users feel.
- Agents reward-hack plausibly — protect the objective and inspect trajectories of passing runs.
- Deterministic environments with reset between runs are the real cost of agent evaluation.
- Prefer deterministic checks to model judges; keep judges independent and calibrated.
- In production, watch intervention rate, abandonment and rework, and trace enough to replay.
- Sample in CI, run the full set before release, and always after a model upgrade.
Further reading
- Carlos Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (ICLR 2024).
- Shunyu Yao et al., "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (arXiv, 2024) — reliability over repeated runs.
- Mark Chen et al., "Evaluating Large Language Models Trained on Code" (arXiv, 2021) — pass@k.
- Lianmin Zheng et al., "Judging LLM-as-a-Judge" (NeurIPS 2023).
- Ronny Kohavi et al., Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).