LLM Evals: How to Evaluate LLM Applications
On this page
Learning Objective
Design an evaluation strategy for a non-deterministic system, distinguish the four tiers of evaluation from deterministic assertions through model-graded scoring to human review and online experiments, and explain where LLM-as-judge is trustworthy and where it is not.
Concept Explanation
Evals are the test suite of an LLM system, and they exist because the ordinary one cannot work. An assertion comparing output to an expected string fails on a correct answer phrased differently and passes on a wrong answer phrased identically, so the only viable approach is to score outputs against criteria and track the distribution of scores over a fixed dataset. Once that reframing is accepted, everything familiar about testing returns: evals run in continuous integration, they gate releases, a regression is a specific numeric drop on a specific subset, and the dataset itself becomes a durable and appreciating asset that outlives every model and prompt version you will try.
The evaluation dataset is the part teams skip and the part that determines whether any of this works. It should be built from real traffic rather than invented examples, because synthetic questions systematically under-represent the messiness that causes actual failures. It should be stratified by the categories your traffic contains and by difficulty, so a score can be reported per stratum rather than as an average that hides where a regression landed, exactly as in the quantization lesson. It must include the adversarial and degenerate cases: questions with no answer in the corpus, ambiguous questions, questions that should be refused, prompt injection attempts, and inputs in unexpected languages or formats. And every production failure that gets reported should be added to it permanently, which is the mechanism that makes the suite compound in value rather than decay.
Evaluation methods form four tiers of increasing cost and decreasing objectivity, and a mature system uses all four. The first tier is deterministic checks, which are underrated because they are unglamorous: does the output parse as valid JSON against the schema, are all cited identifiers real, is the answer within the length bound, does it avoid a forbidden pattern, did it return the expected refusal token for an unanswerable question. These are cheap, fast, completely objective, and catch a surprising share of real regressions. The second tier is reference-based metrics comparing output to a known-good answer, which work for tasks with a constrained correct answer such as extraction, classification, and retrieval, where recall at k and ranking metrics live. Those two tiers should be exhausted before reaching for anything fuzzier.
The third tier is model-graded evaluation, where a strong model scores an output against an explicit rubric. It is the only practical way to evaluate open-ended generation at volume, and the research is genuinely encouraging: the MT-Bench work in 2023 reported that a strong model acting as a judge agreed with human preferences at a rate above eighty percent, comparable to the agreement rate between two human raters. It is also the tier where care is most required, because judges have documented biases. They prefer longer answers, they favor the first option presented in a pairwise comparison, they can prefer text from the same model family, and they will rate a fluent wrong answer above a terse correct one. The mitigations are concrete: use narrow rubrics with explicit criteria rather than asking for a general quality score, prefer binary or small discrete scales over a one-to-ten continuum nobody applies consistently, randomize position in pairwise comparisons, and periodically validate the judge against human labels on a sample to confirm it still agrees with the humans it is standing in for.
The fourth tier is human evaluation and online experimentation, which remain the ground truth. Sampled human review of production traffic catches failure categories no automated check anticipated and produces the labels that validate the judge. Online A/B testing measures what actually matters - task completion, deflection, escalation rate, thumbs-down rate, retry rate, time to resolution - which may move in the opposite direction from your offline score, and when it does, the online result is the one that counts. The habit that ties all four together is that no prompt change, model change, retrieval parameter change, or routing change ships without running the suite, and that stage-level metrics are reported alongside end-to-end ones so a regression can be attributed rather than merely observed.
| Tier | Method | Cost | Objectivity | Good For |
|---|---|---|---|---|
| 1 | Deterministic checks: schema, citation validity, length, refusal token | Negligible | Complete | Format and contract regressions; run on every request if you like |
| 2 | Reference-based: exact or fuzzy match, recall@k, ranking metrics | Low | High | Extraction, classification, and retrieval stages |
| 3 | Model-graded against a rubric | Moderate | Good, with known biases | Open-ended generation, groundedness, tone at volume |
| 4 | Human review and online A/B tests | High | Ground truth | Validating the judge and measuring real product outcomes |
Worked Example: A Layered Eval Suite That Gates a Release
The suite below runs in continuous integration, reports per stratum rather than as one number, and fails the build on a regression in any stratum rather than on a movement in the average.
# eval/suite.py - runs on every prompt, model, retrieval, or routing change.
DATASET = load('eval/golden_v11.jsonl') # 340 cases from REAL traffic, stratified:
# grounded_qa 120 | extraction 60 | refusal_expected 45 | ambiguous 30
# injection_attempt 35 | non_english 30 | multi_turn 20
def evaluate(system, dataset):
results = []
for case in dataset:
out = system.answer(case.question, case.context)
# TIER 1 - deterministic. Cheap, objective, catches real regressions.
checks = {
'schema_valid': schema.validates(out),
'citations_real': set(out.citations) <= {c.id for c in case.context},
'within_length': out.token_count <= 800,
'refused_when_it_should': (out.status == 'insufficient_context')
== case.expect_refusal,
}
# TIER 2 - reference-based, where a correct answer is constrained.
if case.stratum in ('extraction', 'grounded_qa'):
checks['recall_at_50'] = case.gold_chunk_id in out.retrieved_ids
# TIER 3 - model-graded, narrow rubrics, BINARY not 1-10.
if case.stratum == 'grounded_qa':
checks['grounded'] = judge.binary(
'Is every factual claim in the answer supported by the context?',
answer=out.text, context=case.context)
checks['answers_question'] = judge.binary(
'Does the answer address the question that was asked?',
answer=out.text, question=case.question)
results.append((case.stratum, checks))
return aggregate_by_stratum(results)
# Gate on PER-STRATUM regression, never on the average.
baseline, current = load_baseline(), evaluate(system, DATASET)
for stratum, metrics in current.items():
for name, value in metrics.items():
if value < baseline[stratum][name] - 0.02:
fail(f'REGRESSION {stratum}.{name}: {baseline[stratum][name]:.3f} -> {value:.3f}')
# TIER 4 runs continuously, not in CI:
# - 2% of production traffic sampled for human review
# - those labels re-validate the judge monthly (does it still agree?)
# - every reported production failure is appended to golden_v*.jsonl foreverReal-World Examples
- The "Judging LLM-as-a-Judge" work behind MT-Bench, published in 2023, reported that a strong model used as an automated judge agreed with human preference labels at a rate above eighty percent, which is roughly the level at which two independent human raters agree with each other. That result is what made model-graded evaluation credible at volume, and the same paper documented the position, verbosity, and self-preference biases that make rubric design and position randomization necessary rather than optional.
- LMSYS Chatbot Arena evaluates models through anonymized pairwise human comparisons aggregated into an Elo-style rating across millions of votes, deliberately avoiding fixed benchmark datasets. Its design is a direct response to benchmark contamination, where evaluation sets leak into training data and inflate scores, which is the same reason a production eval set should be built from your own private traffic rather than assembled from public datasets.
Review Questions & Answers
Because the failure is in the nature of the comparison, not in its strictness. An exact assertion fails on a correct answer that is phrased differently and passes on a wrong answer that happens to match, and loosening it to a substring or fuzzy match makes both problems worse rather than better. The only workable reframing is to score outputs against explicit criteria and track the distribution of those scores across a fixed dataset, which turns testing from a binary pass or fail on each case into a statistical comparison of a system version against a baseline.
Because they are free, instantaneous, completely objective, and catch a large share of real regressions. Whether the output parses against the schema, whether every cited identifier corresponds to a chunk that was actually supplied, whether the length bound was respected, whether the expected refusal was returned for an unanswerable question, and whether the correct chunk was retrieved are all checkable without any judgment at all. Model-graded evaluation costs money and latency on every case, introduces its own biases, and requires periodic validation against human labels, so it should be reserved for the open-ended qualities that genuinely cannot be checked any other way.
Judges tend to prefer longer and more fluent answers over terse correct ones, to favor whichever option appears first in a pairwise comparison, and in some cases to prefer output from their own model family. The practical mitigations are to use narrow rubrics asking a specific checkable question rather than requesting a general quality rating, to prefer binary or small discrete scales because a one-to-ten scale is not applied consistently, to randomize the order of options in pairwise comparisons and average across both orderings, and to periodically re-validate the judge against human labels on a sample so drift in agreement is detected rather than assumed away.
Because regressions are almost never uniform. A change to the prompt, the retrieval configuration, the model, or the routing policy typically improves some categories while degrading others, and an average dominated by the largest and easiest stratum will barely move while a small, high-value category collapses. This is the same effect seen when evaluating quantized models, where reasoning and non-English performance fall sharply behind an almost unchanged average. Gating on per-stratum regression catches the change; gating on the average ships it.