LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 3 · Lesson 15 · 4 min read
Evaluating LLM Structured Extraction Field by Field
On this page
Evaluating structured extraction: the handoff ticket
When DocsBot hands a conversation to a human, it fills a ticket so the agent doesn't have to reread the chat:
{"product_area": "billing", "plan": "pro", "urgency": "high", "invoice_ids": ["INV-2291", "INV-2304"], "summary": "Charged twice for September after upgrading."}The support team complains that tickets are "often wrong". That's not actionable. Is it the urgency? Missing invoice ids? To find out, you need a dataset of conversations with the correct ticket, written by a support agent, and a grader that scores each field.
Why whole-object exact match fails
Compare the predicted ticket with the reference as one object and almost everything fails: the summary is never word-for-word identical. Even without the summary, one wrong field fails the whole ticket, and you're back to not knowing which field is the problem.
Grade fields separately, and pick the comparison per field type:
| Field type | Comparison | Example |
|---|---|---|
| Enum | Exact match | product_area, plan, urgency |
| Identifier list | Set precision and recall after normalizing | invoice_ids |
| Number or date | Parse, then compare values | refund amount, renewal date |
| Free text | LLM judge on meaning | summary |
A field-by-field grader for JSON output
# extraction_eval.py: field-level scoring for extracted tickets
from collections import defaultdict
ENUM_FIELDS = ["product_area", "plan", "urgency"]
def norm_id(x: str) -> str:
return x.strip().upper().replace(" ", "")
def score_ticket(pred: dict, ref: dict) -> dict:
scores = {f: pred.get(f) == ref.get(f) for f in ENUM_FIELDS}
p, r = {norm_id(i) for i in pred.get("invoice_ids", [])}, {norm_id(i) for i in ref.get("invoice_ids", [])}
scores["invoice_tp"], scores["invoice_fp"], scores["invoice_fn"] = len(p & r), len(p - r), len(r - p)
return scores
def report(pairs: list[tuple[dict, dict]]) -> None:
totals = defaultdict(int)
for pred, ref in pairs:
for k, v