Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 3 · Lesson 15 · 4 min read

Evaluating LLM Structured Extraction Field by Field

On this page
  1. Evaluating structured extraction: the handoff ticket
  2. Why whole-object exact match fails
  3. A field-by-field grader for JSON output
  4. Precision vs recall for extracted lists
  5. Grading fields that should be absent
  6. Check yourself

Evaluating structured extraction: the handoff ticket

When DocsBot hands a conversation to a human, it fills a ticket so the agent doesn't have to reread the chat:

JSON
{"product_area": "billing", "plan": "pro", "urgency": "high", "invoice_ids": ["INV-2291", "INV-2304"], "summary": "Charged twice for September after upgrading."}

The support team complains that tickets are "often wrong". That's not actionable. Is it the urgency? Missing invoice ids? To find out, you need a dataset of conversations with the correct ticket, written by a support agent, and a grader that scores each field.

Why whole-object exact match fails

Compare the predicted ticket with the reference as one object and almost everything fails: the summary is never word-for-word identical. Even without the summary, one wrong field fails the whole ticket, and you're back to not knowing which field is the problem.

Grade fields separately, and pick the comparison per field type:

Field typeComparisonExample
EnumExact matchproduct_area, plan, urgency
Identifier listSet precision and recall after normalizinginvoice_ids
Number or dateParse, then compare valuesrefund amount, renewal date
Free textLLM judge on meaningsummary

A field-by-field grader for JSON output

Python
# extraction_eval.py: field-level scoring for extracted tickets
from collections import defaultdict

ENUM_FIELDS = ["product_area", "plan", "urgency"]

def norm_id(x: str) -> str:
    return x.strip().upper().replace(" ", "")

def score_ticket(pred: dict, ref: dict) -> dict:
    scores = {f: pred.get(f) == ref.get(f) for f in ENUM_FIELDS}
    p, r = {norm_id(i) for i in pred.get("invoice_ids", [])}, {norm_id(i) for i in ref.get("invoice_ids", [])}
    scores["invoice_tp"], scores["invoice_fp"], scores["invoice_fn"] = len(p & r), len(p - r), len(r - p)
    return scores

def report(pairs: list[tuple[dict, dict]]) -> None:
    totals = defaultdict(int)
    for pred, ref in pairs:
        for k, v