How to Evaluate AI Agents: pass^k and LLM Judges
On this page
Learning Objective
Build an evaluation suite for an agent: choose outcome and trajectory metrics, measure reliability across repeated runs, grade with code where possible and with rubric-based model judges where necessary, and turn production failures into regression tests that run in CI.
Concept Explanation
Without evaluations, every change to an agent's prompt, tools or model is a guess. Agents make evaluation harder than single model calls because they are nondeterministic, take many steps, and can reach a correct answer by a bad path or a wrong answer by a reasonable one. A useful suite therefore measures several things.
Outcome metrics ask whether the task succeeded. For many agent tasks there is an objective check: the tests pass, the database ends in the expected state, the ticket has the right label, the answer matches a known value. Prefer these whenever possible; they are cheap, fast and unambiguous. SWE-bench works this way, grading coding agents by running the repository's tests on their patch.
Trajectory metrics ask how the agent got there: how many steps and tokens it used, whether it called the right tools, whether it asked for approval where required, and whether it did anything forbidden, such as calling a write tool during a read-only task. A trajectory that succeeds after forty steps is a cost problem; one that succeeds after touching production data it should not have is a safety problem.
Reliability matters as much as average success. Because agents are stochastic, run each task several times. The tau-bench benchmark from Sierra introduced pass^k, the probability that an agent succeeds on all k independent attempts at the same task, and showed that agents with decent single-run success rates could fall below 25 percent at k equal to 8 in a retail customer service simulation. Users experience pass^k, not pass@1: a support agent that is right four times out of five is wrong for one customer in five.
When the outcome is subjective, such as the quality of a summary or the tone of a reply, use a model as a judge, an approach usually called LLM-as-a-judge, but constrain it. Give the judge a rubric with specific, independently checkable criteria, ask for a verdict per criterion with a short justification, and calibrate the judge against a set of human-graded examples before trusting it. A judge asked only "is this good?" tends to be lenient and inconsistent.
The best test cases come from reality. Start with twenty to fifty tasks written from actual user requests, include the edge cases that have failed before, and add every production failure as a new case once it is fixed. Run the suite in CI on every change to prompts, tools or models, and track the scores over time.
| Metric | Measures | How to grade |
|---|---|---|
| Task success | Did it achieve the goal | Code checks: tests, state, exact values |
| pass^k | Reliability across k runs | Repeat each task k times |
| Steps and tokens | Efficiency and cost | Count from transcripts |
| Tool correctness | Right tools, no forbidden actions | Inspect the trajectory |
| Quality of free text | Helpfulness, tone, accuracy | Rubric-based model judge, calibrated |
| Regressions | Old failures stay fixed | Production failures as test cases |
Worked Example: A Small Agent Evaluation Harness
Each case has a code check for the outcome and a forbidden-tools list for the trajectory. The harness runs every case several times and reports pass@1, pass^k, average steps and violations.
import statistics
from dataclasses import dataclass, field
@dataclass
class Case:
prompt: str
check: callable # receives (answer, trajectory) and returns True or False
forbidden_tools: set = field(default_factory=set)
CASES = [
Case("What is the total of order A-100?", lambda ans, traj: "42.50" in ans),
Case("Which customers have open refunds?", lambda ans, traj: {"ana", "sam"} <= set(ans.lower().split()),
forbidden_tools={"orders__issue_refund"}),
Case("Refund order A-200 in full.", lambda ans, traj: any(t["name"] == "orders__issue_refund"
and t["approved"] for t in traj)),
]
def evaluate(run_agent, k: int = 5) -> dict:
"""run_agent(prompt) returns (answer, trajectory), trajectory being a list of tool calls."""
per_case, steps, violations = [], [], 0
for case in CASES:
outcomes = []
for _ in range(k):
answer, trajectory = run_agent(case.prompt)
steps.append(len(trajectory))
bad = [t for t in trajectory if t["name"] in case.forbidden_tools]
violations += bool(bad)
outcomes.append(case.check(answer, trajectory) and not bad)
per_case.append(outcomes)
return {
"pass@1": statistics.mean(o for outcomes in per_case for o in outcomes),
f"pass^{k}": statistics.mean(all(outcomes) for outcomes in per_case),
"avg_steps": statistics.mean(steps),
"runs_with_violations": violations,
}
# In CI: fail the build if pass^k drops below the last release or any violation appears.A typical first result looks like pass@1 of 0.87 but pass^5 of 0.67: the agent usually succeeds, yet one case in three fails at least once in five tries. That gap is where to look first.
Real-World Examples
- Sierra's tau-bench (2024) evaluated agents on simulated retail and airline customer service with tool use and policy rules, and introduced the pass^k metric to expose how inconsistent agents were across repeated runs of the same task.
- SWE-bench Verified, released by OpenAI in August 2024 as a human-validated subset of 500 SWE-bench tasks, grades coding agents by running the project's own tests, an example of an outcome check that needs no model judge.
Review Questions & Answers
pass^k measures whether the agent succeeds on every one of k attempts at the same task, which reflects consistency. A high pass@1 can hide frequent failures that individual customers will still experience.
A measure of how the agent reached its result, such as steps, tokens, tools used and forbidden actions. A correct answer reached with excessive cost or an unsafe action is still a problem in production.
Give it a rubric of specific, independently checkable criteria, ask for a verdict and short justification per criterion, and calibrate it against human-graded examples before relying on it.
From real user requests, known edge cases, and every production failure once it is fixed, so the suite reflects actual usage and prevents regressions.