Capstone: Eval-Driven Improvement of an LLM App
On this page
Capstone: an eval-driven improvement cycle
DocsBot is live, and support says it "gets things wrong sometimes". Your task is to turn that vague complaint into a measured improvement, and leave the team with an eval system that keeps working after you're gone. Try it before reading the walkthrough.
Deliverables:
- An error analysis: 100 reviewed traces, failure modes with counts, and a decision for each mode (fix, eval, watch).
- A versioned dataset with dev and test splits, tags and origins.
- Graders: code checks for everything code can decide, and at least one LLM judge with measured TPR and TNR on a held-out labeled set.
- One improvement, chosen from the data, with a paired comparison before and after, including the examples it broke.
- Automation: a CI gate on pull requests and an online sampling plan with slices and alerts.
- A one-page report a product manager can read.
From traces to a plan
Sample traces, read them, and code them (Days 2 and 3). Suppose the counts come out like this:
| Failure mode | Share of traffic | Decision |
|---|---|---|
| Wrong page retrieved | 11% | Eval retrieval separately, fix first |
| Stale or wrong fact | 9% | Groundedness and correctness judges |
| Unneeded handoff | 6% | Route confusion matrix, track handoff recall |
| Missing citation | 4% | Template bug: fix, add a code check |
Retrieval leads, and on Day 13's grid most wrong answers fall in "missed and wrong". That makes retrieval the first change to try, before any prompt work.
Building the eval suite
- Dataset (Days 4, 5 and 14): real traces for each failure mode, generated questions from docs for retrieval coverage, hand-written handoff and decline cases. Hash-based dev and test split.
- Code graders (Day 6): citation present and valid, plan rule, valid tool JSON, routing label exact match.
- Retrieval metrics (Day 12): recall@5 and MRR, sliced by doc section.
- Judges (Days 8 to 10): groundedness and correctness, aligned against 150 expert labels with a train, dev and test split, with TPR and TNR on test reported next to every judge score.
- Runner (Days 22 to 25): any of Inspect AI, promptfoo, DeepEval or Langfuse. Pick the one that fits how your team works; the dataset and graders matter far more than the tool.
Measuring one change
Say the change is hybrid retrieval: keyword search plus embeddings, with page titles added to each chunk. Run the suite on the old and new versions, three repeats each, and report like this:
| Measure | Before | After | Paired difference (95% CI) |
|---|---|---|---|
| Recall@5 | 0.78 | 0.91 | +0.13 (+0.08 to +0.18) |
| Groundedness (judge, corrected) | 0.86 | 0.90 | +0.04 (−0.01 to +0.09) |
| Handoff recall | 0.94 | 0.95 | +0.01 (−0.03 to +0.05) |
| Examples newly broken | 9 | read and categorized |
The numbers above show the format; yours come from your own runs. Notice the honest reading: retrieval clearly improved, groundedness probably improved but the interval includes zero, and handoffs are unchanged. The nine broken examples get read, and any real regressions get their own fix or an accepted-risk note.
Leaving the evals running in production
- The CI gate from Day 26 with hard gates, rate regression rules and the newly broken list.
- The online sampler from Day 27, judges on a few percent of traffic, alerts per slice.
- A weekly 30-minute review of flagged traces, feeding the dataset (Day 28 guidelines, measured kappa).
- A note in the report on what to re-check when the model is upgraded (Day 29).
What a strong capstone includes
- Failure modes that came from reading traces, not from a metrics library's default list.
- Judges with reported TPR and TNR on held-out labels, not just a prompt.
- Every comparison paired, with an interval and the list of broken examples.
- Hard gates for anything that must never happen, separate from rates.
- A loop from production back to the dataset, with someone named to run it.
Check yourself
That the change probably helps but the evidence isn't conclusive at this sample size, plus the fixed and broken counts. Then either grow the dataset or ship with monitoring, depending on the risk.
The analysis showed most wrong answers had no relevant page in context. No prompt can fix an answer the model had no information for.
Without TPR and TNR on held-out human labels, judge scores can be off by many points in either direction. Alignment is what makes the numbers in the report trustworthy.
The failure modes found, the eval suite with grader alignment, the change made, the measured effect with uncertainty and per-slice results, and the monitoring left in place.
Quality drifts as users ask new kinds of questions, documents change and model providers update models. Online evals, slice monitoring and feedback loops catch those regressions after launch, when offline evals alone would still look green.