Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 2 · Lesson 10 · 5 min read

Correcting LLM Judge Error: Rogan-Gladen and Bootstrap

On this page
  1. Can you trust an LLM judge's pass rate?
  2. Correcting the pass rate for judge error
  3. Adding a bootstrap confidence interval
  4. When the correction matters
  5. Check yourself

Can you trust an LLM judge's pass rate?

You ran the aligned groundedness judge on 1,000 DocsBot answers and it passed 850 of them. But on the labeled test split, the judge's TPR was 0.95 and its TNR was 0.80. It misses one in five ungrounded answers, so 85 percent is probably too optimistic.

You don't have to guess by how much. If you know how often the judge errs in each direction, you can work backwards to the pass rate a human would have found.

Correcting the pass rate for judge error

Call the true pass rate θ (theta): the rate you'd get if a human expert graded all 1,000 answers. The judge passes a true pass with probability TPR, and wrongly passes a true failure with probability 1 − TNR. So the pass rate you observe is:

observed = θ × TPR + (1 − θ) × (1 − TNR)

Solve for θ:

θ = (observed + TNR − 1) / (TPR + TNR − 1)

This is known in epidemiology as the Rogan-Gladen estimator, used to correct disease prevalence for an imperfect test. It's the same problem: a noisy test applied to many cases.

For DocsBot: (0.85 + 0.80 − 1) / (0.95 + 0.80 − 1) = 0.65 / 0.75 ≈ 0.87.

Here the correction moves the estimate up slightly, because the judge's strictness and leniency partly cancel. With other numbers the correction can move several points either way. That's exactly why you compute it instead of assuming.

Adding a bootstrap confidence interval

TPR and TNR were measured on maybe 80 labeled examples, so they're uncertain too. A bootstrap captures that: resample the labeled test set and the judged outputs many times, recompute the estimate each time, and take the middle 95 percent of results as the interval.

Python
# corrected.py: judge pass rate corrected for judge error, with a bootstrap interval
import random

def rates(labeled):
    passes = [j for h, j in labeled if h == "pass"]
    fails = [j for h, j in labeled if h == "fail"]
    tpr = sum(j == "pass" for j in passes) / len(passes)
    tnr = sum(j == "fail" for j in fails) / len(fails)
    return tpr, tnr

def corrected(observed, tpr, tnr):
    if tpr + tnr <= 1:                       # judge no better than chance: can't correct
        return float("nan")
    return min(1.0, max(0.0, (observed + tnr - 1) / (tpr + tnr - 1)))

def estimate(labeled, judged, n_boot=2000, seed=0):
    rng = random.Random(seed)
    tpr, tnr = rates(labeled)
    point = corrected(sum(j == "pass" for j in judged) / len(judged), tpr, tnr)
    samples = []
    for _ in range(n_boot):
        lab = [rng.choice(labeled) for _ in labeled]
        jud = [rng.choice(judged) for _ in judged]
        try:
            value = corrected(sum(j == "pass" for j in jud) / len(jud), *rates(lab))
        except ZeroDivisionError:            # a resample with no passes or no fails
            continue
        if value == value:                   # skip NaN from resamples where the judge is at chance
            samples.append(value)
    samples.sort()
    return point, samples[int(0.025 * len(samples))], samples[int(0.975 * len(samples))]

# test split: 60 human passes (judge passed 57), 20 human fails (judge failed 16)
labeled = ([("pass", "pass")] * 57 + [("pass", "fail")] * 3
           + [("fail", "fail")] * 16 + [("fail", "pass")] * 4)
judged = ["pass"] * 850 + ["fail"] * 150    # judge verdicts on 1,000 unlabeled answers
point, low, high = estimate(labeled, judged)
print(f"raw 0.850, corrected {point:.3f}, 95% interval {low:.3f} to {high:.3f}")

Running it gives:

Code
raw 0.850, corrected 0.867, 95% interval 0.786 to 0.954

The interval is wide, and most of the width comes from having only 20 labeled failures. That's useful information: to narrow it, label more failures, not more passes.

When the correction matters

  • Reporting a number to others. A corrected rate with an interval is honest. A raw judge rate is a claim about the judge as much as about DocsBot.
  • Comparing versions. If both versions are graded by the same judge, the judge's bias affects both, and comparisons are less distorted than absolute numbers. Day 21 covers comparisons.
  • Judges near chance. If TPR + TNR is close to 1, the judge carries almost no information and the correction explodes. Fix the judge instead.

Check yourself

Upward. The judge passes 45 percent of true failures, so it reports more passes than a human would find. The correction formula uses TPR and TNR to estimate the true pass rate from the judge's raw rate.

Label more outputs in the class with fewer labels, usually failures. The uncertainty in TNR from a handful of failures dominates the width.

TPR + TNR equals 1, so the judge's verdicts are independent of the truth. The formula divides by zero because there's no signal to recover.

When the judge's TPR or TNR is clearly below 1 and you report absolute pass rates or compare versions whose failure rates differ. Small judge errors can then swing conclusions.

Uncertainty from both the evaluated examples and the human-labeled set used to estimate TPR and TNR, by resampling both many times and recomputing the corrected rate.