AI Agent Evals: Outcomes, End State and Trajectory
On this page
Evaluating agents: the agent said done
DocsBot gains a new ability. With the customer's permission it can change their account: add a user seat, switch billing to annual, resend an invoice. A customer writes "please add Priya as an admin and send me the updated invoice". DocsBot replies "All set! Priya is now an admin and the invoice is on its way."
Did it happen? The reply can't tell you. Maybe it added Priya as a regular member. Maybe it sent the old invoice. Maybe it called nothing at all and just said it did. For agents, the message is a claim about work. You have to check the work.
Grading the end state
The most reliable agent eval compares the state of the world after the run with the state it should be in. This is how τ-bench (tau-bench), a 2024 benchmark from Sierra for customer-service agents, grades its tasks: it checks the final database against the expected one, regardless of the conversation that got there.
For DocsBot, each test case sets up a sandbox account, runs the conversation, then inspects the account:
# outcome_check.py: compare the sandbox account after the run with the expected changes
def check_outcome(before: dict, after: dict, expected: dict) -> dict:
problems = []
for user, role in expected.get("roles", {}).items():
if after["users"].get(user) != role:
problems.append(f"{user}: expected role {role}, found {after['users'].get(user)}")
unexpected = set(after["users"]) - set(before["users"]) - set(expected.get("roles", {}))
if unexpected:
problems.append(f"unexpected new users: {sorted(unexpected)}")
if expected.get("invoice_sent") and not after["outbox"]:
problems.append("no invoice sent")
if after["billing_cycle"] != expected.get("billing_cycle", before["billing_cycle"]):
problems.append("billing cycle changed")
return {"pass": not problems, "problems": problems}
before = {"users": {"sam": "admin"}, "outbox": [], "billing_cycle": "monthly"}
after = {"users": {"sam": "admin", "priya": "member"}, "outbox": ["INV-2291.pdf"], "billing_cycle": "monthly"}
print(check_outcome(before, after, {"roles": {"priya": "admin"}, "invoice_sent": True})){'pass': False, 'problems': ['priya: expected role admin, found member']}The check also looks for things that shouldn't have changed, like the billing cycle. Side effects are a classic agent failure, and a check that only confirms the requested change will miss them.
Grading the agent's path
Outcome checks say whether the task succeeded. Trajectory checks look at how: which tools were called, in what order, with which arguments, and how many steps it took. Use them to:
- Explain failures. The outcome failed because the agent called
add_userwithrole="member". That's a prompt or tool description issue. - Catch bad paths to good outcomes. The agent succeeded but looked up the customer's account four times, or tried a delete before an update. It costs money and will eventually go wrong.
- Enforce rules. "Confirm with the customer before any billing change" is a property of the path, not the end state.
Day 18 builds trajectory graders. Keep them separate from outcome grades in the report, because they answer different questions.
Grading the final message
Finally, grade the final reply as you would any answer: is it true to what happened? An agent that adds Priya as a member and says she's an admin has two failures, the wrong action and a false report. The second one erodes trust faster.
A simple code check works here: compare claims in the reply with the actual state, or use a judge that receives both.
Making agent tests repeatable
Agent evals need an environment that resets: a sandbox account per test, fake email outbox, frozen clock. Tests that touch shared staging data will interfere with each other and produce flaky results that nobody trusts.
Check yourself
No. Check the outbox in the end state: the call may have failed, sent the wrong invoice, or gone to the wrong address.
Agents often cause side effects, such as changing a setting while fixing another. An outcome check that only confirms the requested change won't see them.
Fail the trajectory rule. The outcome was right, but the path broke a policy that matters for other cases. Report both grades separately.
Reset the sandbox environment to a fixed starting state before each run, mock or record external services, and run each scenario several times to see flakiness.
That the agent's reply accurately reports what happened, including failures, without claiming actions that didn't happen or hiding ones that did.