Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 6 · Lesson 30 · 4 min read

Capstone: Eval-Driven Improvement of an LLM App

On this page
  1. Capstone: an eval-driven improvement cycle
  2. From traces to a plan
  3. Building the eval suite
  4. Measuring one change
  5. Leaving the evals running in production
  6. What a strong capstone includes
  7. Check yourself

Capstone: an eval-driven improvement cycle

DocsBot is live, and support says it "gets things wrong sometimes". Your task is to turn that vague complaint into a measured improvement, and leave the team with an eval system that keeps working after you're gone. Try it before reading the walkthrough.

Deliverables:

  1. An error analysis: 100 reviewed traces, failure modes with counts, and a decision for each mode (fix, eval, watch).
  2. A versioned dataset with dev and test splits, tags and origins.
  3. Graders: code checks for everything code can decide, and at least one LLM judge with measured TPR and TNR on a held-out labeled set.
  4. One improvement, chosen from the data, with a paired comparison before and after, including the examples it broke.
  5. Automation: a CI gate on pull requests and an online sampling plan with slices and alerts.
  6. A one-page report a product manager can read.

From traces to a plan

Sample traces, read them, and code them (Days 2 and 3). Suppose the counts come out like this:

Failure modeShare of trafficDecision
Wrong page retrieved11%Eval retrieval separately, fix first
Stale or wrong fact9%Groundedness and correctness judges
Unneeded handoff6%Route confusion matrix, track handoff recall
Missing citation4%Template bug: fix, add a code check

Retrieval leads, and on Day 13's grid most wrong answers fall in "missed and wrong". That makes retrieval the first change to try, before any prompt work.

Building the eval suite

  • Dataset (Days 4, 5 and 14): real traces for each failure mode, generated questions from docs for retrieval coverage, hand-written handoff and decline cases. Hash-based dev and test split.
  • Code graders (Day 6): citation present and valid, plan rule, valid tool JSON, routing label exact match.
  • Retrieval metrics (Day 12): recall@5 and MRR, sliced by doc section.
  • Judges (Days 8 to 10): groundedness and correctness, aligned against 150 expert labels with a train, dev and test split, with TPR and TNR on test reported next to every judge score.
  • Runner (Days 22 to 25): any of Inspect AI, promptfoo, DeepEval or Langfuse. Pick the one that fits how your team works; the dataset and graders matter far more than the tool.

Measuring one change

Say the change is hybrid retrieval: keyword search plus embeddings, with page titles added to each chunk. Run the suite on the old and new versions, three repeats each, and report like this:

MeasureBeforeAfterPaired difference (95% CI)
Recall@50.780.91+0.13 (+0.08 to +0.18)
Groundedness (judge, corrected)0.860.90+0.04 (−0.01 to +0.09)
Handoff recall0.940.95+0.01 (−0.03 to +0.05)
Examples newly broken9read and categorized

The numbers above show the format; yours come from your own runs. Notice the honest reading: retrieval clearly improved, groundedness probably improved but the interval includes zero, and handoffs are unchanged. The nine broken examples get read, and any real regressions get their own fix or an accepted-risk note.

Leaving the evals running in production

  • The CI gate from Day 26 with hard gates, rate regression rules and the newly broken list.
  • The online sampler from Day 27, judges on a few percent of traffic, alerts per slice.
  • A weekly 30-minute review of flagged traces, feeding the dataset (Day 28 guidelines, measured kappa).
  • A note in the report on what to re-check when the model is upgraded (Day 29).

What a strong capstone includes

  • Failure modes that came from reading traces, not from a metrics library's default list.
  • Judges with reported TPR and TNR on held-out labels, not just a prompt.
  • Every comparison paired, with an interval and the list of broken examples.
  • Hard gates for anything that must never happen, separate from rates.
  • A loop from production back to the dataset, with someone named to run it.

Check yourself

That the change probably helps but the evidence isn't conclusive at this sample size, plus the fixed and broken counts. Then either grow the dataset or ship with monitoring, depending on the risk.

The analysis showed most wrong answers had no relevant page in context. No prompt can fix an answer the model had no information for.

Without TPR and TNR on held-out human labels, judge scores can be off by many points in either direction. Alignment is what makes the numbers in the report trustworthy.

The failure modes found, the eval suite with grader alignment, the change made, the measured effect with uncertainty and per-slice results, and the monitoring left in place.

Quality drifts as users ask new kinds of questions, documents change and model providers update models. Online evals, slice monitoring and feedback loops catch those regressions after launch, when offline evals alone would still look green.