Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 6 · Lesson 28 · 4 min read

Human Review for LLM Evals: Guidelines and Cohen's Kappa

On this page
  1. Human labels: whose correct is it?
  2. Writing labeling guidelines
  3. Measuring reviewer agreement with Cohen's kappa
  4. Fix disagreements, don't average them
  5. Making the review queue fast
  6. Check yourself

Human labels: whose correct is it?

On Day 2 one domain expert defined what a good DocsBot answer is. That works for the first hundred traces. Now you need labels every week: for judge alignment, new dataset items and online review. One person can't do all of it, and when two people label, they disagree more than anyone expects.

If the labels are inconsistent, everything built on them is too. Judges aligned to noisy labels learn the noise, and pass rates move when the reviewer changes.

Writing labeling guidelines

A labeling guideline turns the expert's judgment into rules others can apply. For each criterion, include:

  • A one-sentence definition, the same one the judge prompt uses.
  • Pass and fail examples, especially borderline ones, with the reason.
  • Decision rules for common gray areas. "Paraphrased numbers count as supported if the value is identical. A correct answer with a broken link fails the citation check but passes groundedness."
  • What to do when unsure: mark it, add a note, and move on. Don't guess.

Keep the guideline in the repository next to the judge prompts. When the guideline changes, the judge prompt usually should too.

Measuring reviewer agreement with Cohen's kappa

Have two reviewers label the same 50 to 100 traces independently. Raw agreement looks reassuring but is inflated by chance: if 90 percent of answers pass, two reviewers who label randomly at that rate still agree about 82 percent of the time. Cohen's kappa corrects for that:

κ = (observed agreement − chance agreement) / (1 − chance agreement)

Python
# kappa.py: agreement between two reviewers, corrected for chance
from collections import Counter

def cohen_kappa(a: list[str], b: list[str]) -> float:
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    chance = sum(ca[label] * cb[label] for label in set(a) | set(b)) / (n * n)
    return (observed - chance) / (1 - chance)

# 100 traces: both pass 80, both fail 8, reviewer A fails 7 that B passes, B fails 5 that A passes
a = ["pass"] * 80 + ["fail"] * 8 + ["fail"] * 7 + ["pass"] * 5
b = ["pass"] * 80 + ["fail"] * 8 + ["pass"] * 7 + ["fail"] * 5
print(f"raw agreement {sum(x == y for x, y in zip(a, b)) / len(a):.2f}, kappa {cohen_kappa(a, b):.2f}")
Code
raw agreement 0.88, kappa 0.50

88 percent agreement sounds fine. A kappa of 0.50 says the reviewers agree only moderately once chance is removed, and nearly all of their disagreement is on the failures, the labels that matter most. Common rules of thumb treat kappa above roughly 0.6 as substantial and above 0.8 as strong, but the right bar depends on the decision you'll make with the labels.

Fix disagreements, don't average them

Low kappa is a guideline problem, not a math problem. Sit the reviewers down with the disagreements:

  • Most come from an unwritten rule one reviewer applies. Add it to the guideline.
  • Some reveal that the criterion mixes two things. Split it into two labels.
  • A few are mistakes. Correct them.

Then label a fresh batch and measure again. Two or three rounds usually bring agreement up sharply.

Making the review queue fast

Reviewers label more and better when the tool fits the task. Langfuse, Braintrust and similar platforms offer annotation queues, or a simple internal page works. Whatever you use:

  • Show the full trace: question, retrieved pages, tool calls, answer.
  • Put the criterion and its guideline on screen.
  • Use one-key pass and fail, plus an optional note.
  • Feed it from the online sampler and flagged traces (Day 27).

Check yourself

Check kappa first. If most answers pass, high raw agreement can happen by chance, and the reviewers may still disagree on most failures.

Review the disagreements together, find the unwritten rules or mixed criteria behind them, update the guideline, and measure again on a fresh batch.

They define the same criterion, one for humans and one for the judge. Keeping them together makes it obvious when one changes and the other needs to follow, so human labels and judge verdicts keep measuring the same thing.

Agreement that would happen by chance. When most labels are pass, two reviewers agree often even when guessing, so raw agreement looks better than it is.

Show everything the reviewer needs on one screen, use clear guidelines with examples, ask binary questions, and make keyboard shortcuts for verdicts.