Grading AI Agent Tool Calls and Trajectories
On this page
What an agent trace already tells you
Every DocsBot agent run produces a list of tool calls, each with a name and arguments. That list is the trajectory. Yesterday you graded the end state; today you grade the trajectory, which explains failures and enforces rules the end state can't show.
The trap is being too strict. If a test demands the exact sequence get_account, add_user, send_invoice, it fails a run that also called get_invoice to check the amount first. That's not wrong, just different. Good trajectory tests say what must happen, what must not, and leave the rest free.
Four kinds of trajectory expectations
| Expectation | Example for "add Priya as admin, send invoice" |
|---|---|
| Required calls, with argument rules | add_user with email ending in @customer.example and role == "admin" |
| Order constraints | confirm_with_customer before any billing change |
| Forbidden calls | no delete_user, no change_plan |
| Budget | at most 6 tool calls in total |
Arguments need flexible matching too. The agent might pass the email in any case, or include optional fields. Match what matters and ignore the rest.
A small trajectory matcher in Python
# trajectory.py: flexible checks on an agent's tool calls
Call = dict # {"name": str, "args": dict}
def matches(call: Call, name: str, **arg_rules) -> bool:
if call["name"] != name:
return False
for key, rule in arg_rules.items():
value = call["args"].get(key)
ok = rule(value) if callable(rule) else value == rule
if not ok:
return False
return True
def check(calls: list[Call], required=(), before=(), forbidden=(), max_calls=None) -> list[str]:
problems = []
for name, rules in required:
if not any(matches(c, name, **rules) for c in calls):
problems.append(f"missing {name} matching {sorted(rules)}")
for first, then in before:
idx = {c["name"]: i for i, c in reversed(list(enumerate(calls)))} # first index per tool
if then in idx and (first not in idx or idx[first] > idx[then]):
problems.append(f"{then} called before {first}")
problems += [f"forbidden call {c['name']}" for c in calls if c["name"] in forbidden]
if max_calls is not None and len(calls) > max_calls:
problems.append(f"{len(calls)} calls, budget {max_calls}")
return problems
calls = [
{"name": "get_account", "args": {"account_id": "acc-17"}},
{"name": "add_user", "args": {"email": "Priya@customer.example", "role": "member"}},
{"name": "send_invoice", "args": {"invoice_id": "INV-2291"}},
]
print(check(
calls,
required=[("add_user", {"email": lambda e: e.lower() == "priya@customer.example", "role": "admin"}),
("send_invoice", {})],
before=[("confirm_with_customer", "send_invoice")],
forbidden={"delete_user", "change_plan"},
max_calls=6,
))["missing add_user matching ['email', 'role']", 'send_invoice called before confirm_with_customer']Two problems, both specific: the role argument was wrong, and the invoice went out without confirmation. The email's capital P didn't fail anything, because the rule normalizes it.
Evaluating tool selection as classification
For single-step decisions, "which tool should handle this message?" is a classification problem. Build a dataset of messages with the expected tool, or "no tool", and use the confusion matrix from Day 16. It's cheap to run and catches tool description problems quickly, such as search_docs and get_account being confused.
When to use an LLM judge on trajectories
Some trajectory qualities don't reduce to rules: did the agent recover sensibly after a tool error? Was a clarifying question necessary? For those, give an LLM judge the full trace and one binary criterion, and align it with human labels like any other judge.
Check yourself
It's over-specified. Require the calls that must happen and the orders that matter, and let harmless extra steps pass, perhaps within a call budget.
An order constraint on the trajectory: the confirmation call must appear before any billing tool call, checked across every test case that touches billing.
Treat tool selection as classification: a dataset of messages with expected tools, scored with a confusion matrix and per-tool precision and recall.
Required calls, forbidden calls, ordering constraints such as confirm before charge, and limits such as a maximum number of steps. Most tests need only a few of these.
For judgments code can't make, like whether the agent's search queries were sensible, after code checks have covered the hard rules.