Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 1 · Lesson 3 · 4 min read

Axial Coding: Turning Notes into LLM Failure Modes

On this page
  1. From notes to failure modes: axial coding
  2. Grouping notes into failure modes
  3. Counting failure modes
  4. Deciding which failure modes need an eval
  5. Keeping the failure taxonomy up to date
  6. Check yourself

From notes to failure modes: axial coding

After yesterday you have a spreadsheet with around 100 notes like "quoted an old price" and "no citation". Individually they're anecdotes. Grouped and counted, they become a list of what's wrong with DocsBot and how often. That grouping step is called axial coding: you find the categories that tie the notes together.

Grouping notes into failure modes

Read through the notes and cluster similar ones. Give each cluster a name that describes the failure, not its cause. Aim for roughly 5 to 10 modes. Fewer usually means categories too broad to act on; many more usually means you're splitting hairs.

DocsBot's notes might cluster like this:

Failure modeExample notesCount
Stale or wrong fact"old price", "24 hours instead of 1 hour"14
Wrong page retrieved"2023 pricing page", "admin guide for wrong plan"11
Unneeded handoff"handed off SSO question, page exists"8
Missing citation"answer fine, no link"7
Unsupported claim"says API supports webhooks, docs don't mention"5
Ignores plan"explained Enterprise feature to Starter user"4

You can ask an LLM to propose clusters from the notes to save time. Review its proposal yourself, though: it doesn't know which distinctions matter to Acme, and merging "stale fact" with "unsupported claim" would hide two different fixes.

Counting failure modes

Add a mode column to the review sheet, assign each failing trace one mode, and count:

Python
# modes.py: count failure modes from the annotated review sheet
import csv
from collections import Counter

rows = list(csv.DictReader(open("review.csv")))
failed = [r for r in rows if r["pass"].strip().lower() == "n"]
counts = Counter(r["mode"] for r in failed)

print(f"{len(failed)} of {len(rows)} traces failed ({len(failed) / len(rows):.0%})")
for mode, n in counts.most_common():
    print(f"  {mode:<22} {n:>3}  {n / len(rows):.0%} of traffic")

The output is your first real eval result, done by hand: an overall failure rate and a ranked list of problems.

Deciding which failure modes need an eval

Not every failure mode needs an automated eval. Sort them into three groups:

  1. Fix it now, no eval needed. "Missing citation" might be a bug in the prompt template. Fix it, then add a cheap code check (Day 6) so it doesn't come back.
  2. Needs an automated eval. "Unsupported claim" depends on judgment and will come back as prompts and models change. These get code graders or LLM judges (Days 6 to 11).
  3. Watch, don't build yet. Rare modes with low impact. Note them and revisit in the next review.

Weigh frequency against harm. "Ignores plan" happens in 4 percent of traces, but telling a Starter customer about an Enterprise feature creates support tickets and angry customers. It may deserve an eval before "missing citation" does.

Failure modes ranked by frequency, with the action each one gets.
Failure modes ranked by frequency, with the action each one gets.

Keeping the failure taxonomy up to date

Failure modes change. A new feature, a model upgrade or new users bring new failures. Researchers studying LLM-assisted grading found that people refine their criteria as they see more outputs, an effect they call criteria drift. Plan for it: repeat error analysis on a fresh sample after every significant change, and at least monthly once the feature is live.

NoteKeep the notes, not just the counts. When a judge disagrees with a human on Day 9, the original note is often what explains why.

Check yourself

"Bad answer" is too broad to act on. Split it until each mode points to a specific kind of fix, such as retrieval, prompt or tool logic.

Probably not. Whether a citation is present is checkable with code, and the cause is often a simple bug. Fix it and add a code assertion to catch regressions.

New features, models, prompts and users bring new failure modes that your current evals don't measure, so a green dashboard can hide them. Only reading fresh traces finds them; schedule it, for example after every major change or monthly.

It's specific enough that two people would label the same trace the same way, like "cites a page that doesn't support the claim", and it points toward a fix. Vague names like "bad answer" don't.

By how often they happen and how much harm they cause. A frequent minor issue and a rare but serious one, like leaking another customer's data, can both deserve one.