Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 6 · Lesson 27 · 4 min read

Online LLM Evals: Monitoring Quality in Production

On this page
  1. Why offline evals aren't enough: online monitoring
  2. Which evals to run on live traffic
  3. Watching slices, not just the average
  4. Why user feedback is a biased signal
  5. Feeding production failures back into the dataset
  6. Check yourself

Why offline evals aren't enough: online monitoring

DocsBot's gate is green. Then Acme launches a new integrations page, and customers start asking about it in ways your dataset never imagined. Retrieval misses, DocsBot hands off half of those questions, and nobody notices for two weeks because no eval contains them.

Offline evals check a fixed set of inputs. Online evals score the real traffic as it arrives. You need both: offline to test changes before they ship, online to see what's happening to users now.

Which evals to run on live traffic

Cost decides the layout:

LayerRuns onExamples
Code checksEvery requestCitation present and valid, plan rule, tool JSON valid, response length
Logged signalsEvery requestRoute chosen (answer, handoff, decline), retrieval scores, latency, tokens
User feedbackWhere givenThumbs up or down, follow-up "that's wrong", handoff after an answer
LLM judgesA sample, such as 2 to 5 percentGroundedness, answer relevance, handoff appropriateness
Human reviewA smaller sample plus flagged tracesError analysis as on Day 2

The judges are the same aligned prompts you use offline. Running them asynchronously, after the response is sent, keeps them out of the user's latency.

Python
# online_sample.py: decide which live traces get judged, and keep it reproducible
import hashlib

def should_judge(trace_id: str, route: str, flagged: bool, rate: float = 0.03) -> bool:
    if flagged:                                   # thumbs-down, failed code check, unusual route
        return True
    if route == "handoff":
        rate = 0.10                               # over-sample a decision we care about
    bucket = int(hashlib.sha256(trace_id.encode()).hexdigest(), 16) % 10_000
    return bucket < rate * 10_000

print(should_judge("t-1001", "answer", False), should_judge("t-1002", "answer", True))

Hashing the trace id means the same trace always gets the same decision, which helps when you rerun scoring later. Always judging flagged traces gives you rich failure data; just remember those aren't a random sample when you compute rates.

Watching slices, not just the average

An average groundedness of 94 percent can hide a new topic at 60 percent. Track each rate by:

  • topic or doc section, from retrieval,
  • customer plan,
  • route,
  • app version and model version.

Then alert on changes, not on absolute levels: "handoff rate for integrations questions doubled this week" is actionable. Compare against the previous week's same slice, and require enough traffic in the slice before alerting, or small slices will page you on noise.

Live traffic gets cheap checks everywhere and judges on a sample; failures flow back into error analysis and the offline dataset.
Live traffic gets cheap checks everywhere and judges on a sample; failures flow back into error analysis and the offline dataset.

Why user feedback is a biased signal

Thumbs-down rates are useful and misleading in equal measure. Few users click at all, and those who do are often the angriest or the most delighted. A good answer to a question the user didn't like hearing ("SSO isn't on your plan") gets a thumbs-down too.

Treat feedback as a way to find traces worth reading, not as a quality score. Implicit signals help: a handoff right after an answer, a user rephrasing the same question, a support ticket opened within an hour.

Feeding production failures back into the dataset

Online evals are only worth it if failures flow back:

  1. Weekly, read a sample of flagged and judged-failing traces, as in Day 2.
  2. Add representative ones to the dataset with labels, linking the source trace (Day 25).
  3. When a new failure mode appears, write a check or judge for it, offline and online.

That's how the dataset keeps looking like real traffic, and how the CI gate keeps protecting what users care about.

Check yourself

Judge calls cost money and time. A few percent of traffic gives enough data to track the rate per slice, while code checks cover every request.

Not necessarily. Users dislike correct "that's not on your plan" answers, so thumbs-down can rise while accuracy improves. Read the thumbs-down traces and run your quality judges before concluding anything.

Integrations is a small slice, so even a doubling barely moves the overall average. Track and alert on rates per slice, by topic, plan or language, so local problems surface.

Review flagged traces, add representative failures as new examples with labels, and rerun evals, so every fixed problem stays covered.

Code checks such as valid format, citations present, response length, latency and routing decisions. They cost almost nothing, so they can cover all traffic while expensive LLM judges run on a sample.