Catalog My learning Blog
Build AI Agents with MCP: 30-Day Course · Phase 6 · Lesson 27 · 5 min read

How to Evaluate AI Agents: pass^k and LLM Judges

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: A Small Agent Evaluation Harness
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Build an evaluation suite for an agent: choose outcome and trajectory metrics, measure reliability across repeated runs, grade with code where possible and with rubric-based model judges where necessary, and turn production failures into regression tests that run in CI.

Concept Explanation

Without evaluations, every change to an agent's prompt, tools or model is a guess. Agents make evaluation harder than single model calls because they are nondeterministic, take many steps, and can reach a correct answer by a bad path or a wrong answer by a reasonable one. A useful suite therefore measures several things.

Outcome metrics ask whether the task succeeded. For many agent tasks there is an objective check: the tests pass, the database ends in the expected state, the ticket has the right label, the answer matches a known value. Prefer these whenever possible; they are cheap, fast and unambiguous. SWE-bench works this way, grading coding agents by running the repository's tests on their patch.

Trajectory metrics ask how the agent got there: how many steps and tokens it used, whether it called the right tools, whether it asked for approval where required, and whether it did anything forbidden, such as calling a write tool during a read-only task. A trajectory that succeeds after forty steps is a cost problem; one that succeeds after touching production data it should not have is a safety problem.

Reliability matters as much as average success. Because agents are stochastic, run each task several times. The tau-bench benchmark from Sierra introduced pass^k, the probability that an agent succeeds on all k independent attempts at the same task, and showed that agents with decent single-run success rates could fall below 25 percent at k equal to 8 in a retail customer service simulation. Users experience pass^k, not pass@1: a support agent that is right four times out of five is wrong for one customer in five.

When the outcome is subjective, such as the quality of a summary or the tone of a reply, use a model as a judge, an approach usually called LLM-as-a-judge, but constrain it. Give the judge a rubric with specific, independently checkable criteria, ask for a verdict per criterion with a short justification, and calibrate the judge against a set of human-graded examples before trusting it. A judge asked only "is this good?" tends to be lenient and inconsistent.

The best test cases come from reality. Start with twenty to fifty tasks written from actual user requests, include the edge cases that have failed before, and add every production failure as a new case once it is fixed. Run the suite in CI on every change to prompts, tools or models, and track the scores over time.

MetricMeasuresHow to grade
Task successDid it achieve the goalCode checks: tests, state, exact values
pass^kReliability across k runsRepeat each task k times
Steps and tokensEfficiency and costCount from transcripts
Tool correctnessRight tools, no forbidden actionsInspect the trajectory
Quality of free textHelpfulness, tone, accuracyRubric-based model judge, calibrated
RegressionsOld failures stay fixedProduction failures as test cases

Worked Example: A Small Agent Evaluation Harness

Each case has a code check for the outcome and a forbidden-tools list for the trajectory. The harness runs every case several times and reports pass@1, pass^k, average steps and violations.

Code
import statistics
from dataclasses import dataclass, field

@dataclass
class Case:
    prompt: str
    check: callable                      # receives (answer, trajectory) and returns True or False
    forbidden_tools: set = field(default_factory=set)

CASES = [
    Case("What is the total of order A-100?", lambda ans, traj: "42.50" in ans),
    Case("Which customers have open refunds?", lambda ans, traj: {"ana", "sam"} <= set(ans.lower().split()),
         forbidden_tools={"orders__issue_refund"}),
    Case("Refund order A-200 in full.", lambda ans, traj: any(t["name"] == "orders__issue_refund"
                                                             and t["approved"] for t in traj)),
]

def evaluate(run_agent, k: int = 5) -> dict:
    """run_agent(prompt) returns (answer, trajectory), trajectory being a list of tool calls."""
    per_case, steps, violations = [], [], 0
    for case in CASES:
        outcomes = []
        for _ in range(k):
            answer, trajectory = run_agent(case.prompt)
            steps.append(len(trajectory))
            bad = [t for t in trajectory if t["name"] in case.forbidden_tools]
            violations += bool(bad)
            outcomes.append(case.check(answer, trajectory) and not bad)
        per_case.append(outcomes)
    return {
        "pass@1": statistics.mean(o for outcomes in per_case for o in outcomes),
        f"pass^{k}": statistics.mean(all(outcomes) for outcomes in per_case),
        "avg_steps": statistics.mean(steps),
        "runs_with_violations": violations,
    }

# In CI: fail the build if pass^k drops below the last release or any violation appears.

A typical first result looks like pass@1 of 0.87 but pass^5 of 0.67: the agent usually succeeds, yet one case in three fails at least once in five tries. That gap is where to look first.

Illustrative results for one agent: it succeeds on most single runs, but the share of tasks it solves on all five attempts is much lower. Users experience the consistency, which is why pass^k belongs in every agent eval.
Illustrative results for one agent: it succeeds on most single runs, but the share of tasks it solves on all five attempts is much lower. Users experience the consistency, which is why pass^k belongs in every agent eval.

Real-World Examples

  • Sierra's tau-bench (2024) evaluated agents on simulated retail and airline customer service with tool use and policy rules, and introduced the pass^k metric to expose how inconsistent agents were across repeated runs of the same task.
  • SWE-bench Verified, released by OpenAI in August 2024 as a human-validated subset of 500 SWE-bench tasks, grades coding agents by running the project's own tests, an example of an outcome check that needs no model judge.
Key TakeawaysEvaluate AI agents on three things: the outcomes they produce, the trajectories they take to get there, and how reliably they repeat it. Prefer code checks for success, inspect trajectories for cost and forbidden actions, and run tasks several times to measure pass^k, which reflects what users experience. Use rubric-based, calibrated model judges only where outcomes are subjective. Build cases from real requests and production failures, and run the suite in CI on every change.

Review Questions & Answers

pass^k measures whether the agent succeeds on every one of k attempts at the same task, which reflects consistency. A high pass@1 can hide frequent failures that individual customers will still experience.

A measure of how the agent reached its result, such as steps, tokens, tools used and forbidden actions. A correct answer reached with excessive cost or an unsafe action is still a problem in production.

Give it a rubric of specific, independently checkable criteria, ask for a verdict and short justification per criterion, and calibrate it against human-graded examples before relying on it.

From real user requests, known edge cases, and every production failure once it is fixed, so the suite reflects actual usage and prevents regressions.