Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 4 · Lesson 18 · 4 min read

Grading AI Agent Tool Calls and Trajectories

On this page
  1. What an agent trace already tells you
  2. Four kinds of trajectory expectations
  3. A small trajectory matcher in Python
  4. Evaluating tool selection as classification
  5. When to use an LLM judge on trajectories
  6. Check yourself

What an agent trace already tells you

Every DocsBot agent run produces a list of tool calls, each with a name and arguments. That list is the trajectory. Yesterday you graded the end state; today you grade the trajectory, which explains failures and enforces rules the end state can't show.

The trap is being too strict. If a test demands the exact sequence get_account, add_user, send_invoice, it fails a run that also called get_invoice to check the amount first. That's not wrong, just different. Good trajectory tests say what must happen, what must not, and leave the rest free.

Four kinds of trajectory expectations

ExpectationExample for "add Priya as admin, send invoice"
Required calls, with argument rulesadd_user with email ending in @customer.example and role == "admin"
Order constraintsconfirm_with_customer before any billing change
Forbidden callsno delete_user, no change_plan
Budgetat most 6 tool calls in total

Arguments need flexible matching too. The agent might pass the email in any case, or include optional fields. Match what matters and ignore the rest.

A small trajectory matcher in Python

Python
# trajectory.py: flexible checks on an agent's tool calls
Call = dict  # {"name": str, "args": dict}

def matches(call: Call, name: str, **arg_rules) -> bool:
    if call["name"] != name:
        return False
    for key, rule in arg_rules.items():
        value = call["args"].get(key)
        ok = rule(value) if callable(rule) else value == rule
        if not ok:
            return False
    return True

def check(calls: list[Call], required=(), before=(), forbidden=(), max_calls=None) -> list[str]:
    problems = []
    for name, rules in required:
        if not any(matches(c, name, **rules) for c in calls):
            problems.append(f"missing {name} matching {sorted(rules)}")
    for first, then in before:
        idx = {c["name"]: i for i, c in reversed(list(enumerate(calls)))}   # first index per tool
        if then in idx and (first not in idx or idx[first] > idx[then]):
            problems.append(f"{then} called before {first}")
    problems += [f"forbidden call {c['name']}" for c in calls if c["name"] in forbidden]
    if max_calls is not None and len(calls) > max_calls:
        problems.append(f"{len(calls)} calls, budget {max_calls}")
    return problems

calls = [
    {"name": "get_account", "args": {"account_id": "acc-17"}},
    {"name": "add_user", "args": {"email": "Priya@customer.example", "role": "member"}},
    {"name": "send_invoice", "args": {"invoice_id": "INV-2291"}},
]
print(check(
    calls,
    required=[("add_user", {"email": lambda e: e.lower() == "priya@customer.example", "role": "admin"}),
              ("send_invoice", {})],
    before=[("confirm_with_customer", "send_invoice")],
    forbidden={"delete_user", "change_plan"},
    max_calls=6,
))
Code
["missing add_user matching ['email', 'role']", 'send_invoice called before confirm_with_customer']

Two problems, both specific: the role argument was wrong, and the invoice went out without confirmation. The email's capital P didn't fail anything, because the rule normalizes it.

Evaluating tool selection as classification

For single-step decisions, "which tool should handle this message?" is a classification problem. Build a dataset of messages with the expected tool, or "no tool", and use the confusion matrix from Day 16. It's cheap to run and catches tool description problems quickly, such as search_docs and get_account being confused.

When to use an LLM judge on trajectories

Some trajectory qualities don't reduce to rules: did the agent recover sensibly after a tool error? Was a clarifying question necessary? For those, give an LLM judge the full trace and one binary criterion, and align it with human labels like any other judge.

Check yourself

It's over-specified. Require the calls that must happen and the orders that matter, and let harmless extra steps pass, perhaps within a call budget.

An order constraint on the trajectory: the confirmation call must appear before any billing tool call, checked across every test case that touches billing.

Treat tool selection as classification: a dataset of messages with expected tools, scored with a confusion matrix and per-tool precision and recall.

Required calls, forbidden calls, ordering constraints such as confirm before charge, and limits such as a maximum number of steps. Most tests need only a few of these.

For judgments code can't make, like whether the agent's search queries were sensible, after code checks have covered the hard rules.