LLM Error Analysis: Reading Traces and Open Coding
On this page
Why you shouldn't start with a metric
The tempting first step is to install an eval library and turn on its built-in metrics: helpfulness, coherence, toxicity. You get a dashboard full of numbers by lunch. Then someone asks what to fix, and nobody can say.
Generic metrics measure what the library author thought might go wrong. Your app fails in its own ways. DocsBot might cite the right page but quote an old price, or hand off to a human for questions it could answer. No off-the-shelf score names those problems. You find them by reading.
Which traces to read in error analysis
A trace is the full record of one request: the user input, the retrieved documents, any tool calls and their results, and the final answer. If you log with Langfuse, LangSmith or OpenTelemetry you already have them. If not, log them now: inputs, retrieved document ids, tool calls and outputs, as JSON lines.
Pull a sample of about 100 traces. Mix them on purpose:
- random traffic, so you see what's common,
- traces with a thumbs-down or a support ticket attached,
- long conversations and ones that ended in a handoff.
If the feature isn't live yet, have teammates use it for an afternoon, or generate realistic inputs (Day 5) and run them through.
Open coding: write notes, not labels
Go through traces one at a time. For each, write a short note about the first thing that went wrong, in plain words. If nothing went wrong, mark it as a pass and move on. This is called open coding, a term from qualitative research: you describe what you see before deciding on categories.
# review.py: sample traces into a sheet you can annotate
import csv, json, random
random.seed(7) # same sample every time you rerun
traces = [json.loads(line) for line in open("traces.jsonl")]
flagged = [t for t in traces if t.get("feedback") == "down"]
rest = [t for t in traces if t.get("feedback") != "down"]
sample = flagged[:30] + random.sample(rest, min(70, len(rest)))
with open("review.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["trace_id", "question", "answer", "docs", "pass", "note"])
for t in sample:
w.writerow([t["id"], t["question"], t["answer"], " ".join(t["doc_ids"]), "", ""])
print(f"{len(sample)} traces written to review.csv")Open review.csv in a spreadsheet and fill in the last two columns. Notes from a DocsBot session might read:
- "Said reset links last 24 hours; docs say 1 hour."
- "Retrieved the 2023 pricing page, not the current one."
- "Customer asked about SSO; bot handed off but SSO page exists."
- "Answer fine but no citation."
Write the note about the first failure only. Later failures are often caused by the first one, and counting them twice skews your picture.
Who should do the reading: the domain expert
The person who knows what a good answer looks like. For DocsBot that's someone from support or the docs team, working with the engineer. Hamel Husain and Shreya Shankar, who teach this method, call this person the benevolent dictator: one domain expert whose judgment defines "good", so the team doesn't argue about taste in every review.
Expect about 30 seconds to a few minutes per trace. Stop when new traces stop showing new kinds of problems. Researchers call this theoretical saturation. In practice it often comes somewhere around 50 to 100 traces for a focused feature.
Why error analysis beats brainstorming
You could sit in a meeting and list everything that might go wrong. That list will be long, and it will miss the problems users hit. Reading traces grounds the list in reality, and it shows how often each problem occurs. Tomorrow you'll turn these notes into a short list of failure modes with counts.
Check yourself
One, about the wrong price. It's the first failure in the trace, and the wrong follow-up is a consequence of it. Noting only the first failure keeps your counts honest, since fixing the price would likely fix both.
A generic metric won't name DocsBot's actual failures, so it won't tell you what to fix. Read 100 traces first, then build metrics for the failures you find.
When new traces stop revealing new kinds of failures, which is called saturation. For a focused feature that often happens somewhere around 50 to 100 traces. Keep a sample of fresh traces in your routine afterwards, since new failures appear as the product changes.
They know what a correct answer looks like, like whether a refund policy was stated right, which engineers often can't judge. Their notes define what quality means for the product.
A short, specific description of what went wrong in that trace, in plain words, such as "quoted the Pro price for an Enterprise customer". You don't pick categories yet; those come from grouping notes later.