Online LLM Evals: Monitoring Quality in Production
On this page
Why offline evals aren't enough: online monitoring
DocsBot's gate is green. Then Acme launches a new integrations page, and customers start asking about it in ways your dataset never imagined. Retrieval misses, DocsBot hands off half of those questions, and nobody notices for two weeks because no eval contains them.
Offline evals check a fixed set of inputs. Online evals score the real traffic as it arrives. You need both: offline to test changes before they ship, online to see what's happening to users now.
Which evals to run on live traffic
Cost decides the layout:
| Layer | Runs on | Examples |
|---|---|---|
| Code checks | Every request | Citation present and valid, plan rule, tool JSON valid, response length |
| Logged signals | Every request | Route chosen (answer, handoff, decline), retrieval scores, latency, tokens |
| User feedback | Where given | Thumbs up or down, follow-up "that's wrong", handoff after an answer |
| LLM judges | A sample, such as 2 to 5 percent | Groundedness, answer relevance, handoff appropriateness |
| Human review | A smaller sample plus flagged traces | Error analysis as on Day 2 |
The judges are the same aligned prompts you use offline. Running them asynchronously, after the response is sent, keeps them out of the user's latency.
# online_sample.py: decide which live traces get judged, and keep it reproducible
import hashlib
def should_judge(trace_id: str, route: str, flagged: bool, rate: float = 0.03) -> bool:
if flagged: # thumbs-down, failed code check, unusual route
return True
if route == "handoff":
rate = 0.10 # over-sample a decision we care about
bucket = int(hashlib.sha256(trace_id.encode()).hexdigest(), 16) % 10_000
return bucket < rate * 10_000
print(should_judge("t-1001", "answer", False), should_judge("t-1002", "answer", True))Hashing the trace id means the same trace always gets the same decision, which helps when you rerun scoring later. Always judging flagged traces gives you rich failure data; just remember those aren't a random sample when you compute rates.
Watching slices, not just the average
An average groundedness of 94 percent can hide a new topic at 60 percent. Track each rate by:
- topic or doc section, from retrieval,
- customer plan,
- route,
- app version and model version.
Then alert on changes, not on absolute levels: "handoff rate for integrations questions doubled this week" is actionable. Compare against the previous week's same slice, and require enough traffic in the slice before alerting, or small slices will page you on noise.
Why user feedback is a biased signal
Thumbs-down rates are useful and misleading in equal measure. Few users click at all, and those who do are often the angriest or the most delighted. A good answer to a question the user didn't like hearing ("SSO isn't on your plan") gets a thumbs-down too.
Treat feedback as a way to find traces worth reading, not as a quality score. Implicit signals help: a handoff right after an answer, a user rephrasing the same question, a support ticket opened within an hour.
Feeding production failures back into the dataset
Online evals are only worth it if failures flow back:
- Weekly, read a sample of flagged and judged-failing traces, as in Day 2.
- Add representative ones to the dataset with labels, linking the source trace (Day 25).
- When a new failure mode appears, write a check or judge for it, offline and online.
That's how the dataset keeps looking like real traffic, and how the CI gate keeps protecting what users care about.
Check yourself
Judge calls cost money and time. A few percent of traffic gives enough data to track the rate per slice, while code checks cover every request.
Not necessarily. Users dislike correct "that's not on your plan" answers, so thumbs-down can rise while accuracy improves. Read the thumbs-down traces and run your quality judges before concluding anything.
Integrations is a small slice, so even a doubling barely moves the overall average. Track and alert on rates per slice, by topic, plan or language, so local problems surface.
Review flagged traces, add representative failures as new examples with labels, and rerun evals, so every fixed problem stays covered.
Code checks such as valid format, citations present, response length, latency and routing decisions. They cost almost nothing, so they can cover all traffic while expensive LLM judges run on a sample.