Correcting LLM Judge Error: Rogan-Gladen and Bootstrap
On this page
Can you trust an LLM judge's pass rate?
You ran the aligned groundedness judge on 1,000 DocsBot answers and it passed 850 of them. But on the labeled test split, the judge's TPR was 0.95 and its TNR was 0.80. It misses one in five ungrounded answers, so 85 percent is probably too optimistic.
You don't have to guess by how much. If you know how often the judge errs in each direction, you can work backwards to the pass rate a human would have found.
Correcting the pass rate for judge error
Call the true pass rate θ (theta): the rate you'd get if a human expert graded all 1,000 answers. The judge passes a true pass with probability TPR, and wrongly passes a true failure with probability 1 − TNR. So the pass rate you observe is:
observed = θ × TPR + (1 − θ) × (1 − TNR)
Solve for θ:
θ = (observed + TNR − 1) / (TPR + TNR − 1)
This is known in epidemiology as the Rogan-Gladen estimator, used to correct disease prevalence for an imperfect test. It's the same problem: a noisy test applied to many cases.
For DocsBot: (0.85 + 0.80 − 1) / (0.95 + 0.80 − 1) = 0.65 / 0.75 ≈ 0.87.
Here the correction moves the estimate up slightly, because the judge's strictness and leniency partly cancel. With other numbers the correction can move several points either way. That's exactly why you compute it instead of assuming.
Adding a bootstrap confidence interval
TPR and TNR were measured on maybe 80 labeled examples, so they're uncertain too. A bootstrap captures that: resample the labeled test set and the judged outputs many times, recompute the estimate each time, and take the middle 95 percent of results as the interval.
# corrected.py: judge pass rate corrected for judge error, with a bootstrap interval
import random
def rates(labeled):
passes = [j for h, j in labeled if h == "pass"]
fails = [j for h, j in labeled if h == "fail"]
tpr = sum(j == "pass" for j in passes) / len(passes)
tnr = sum(j == "fail" for j in fails) / len(fails)
return tpr, tnr
def corrected(observed, tpr, tnr):
if tpr + tnr <= 1: # judge no better than chance: can't correct
return float("nan")
return min(1.0, max(0.0, (observed + tnr - 1) / (tpr + tnr - 1)))
def estimate(labeled, judged, n_boot=2000, seed=0):
rng = random.Random(seed)
tpr, tnr = rates(labeled)
point = corrected(sum(j == "pass" for j in judged) / len(judged), tpr, tnr)
samples = []
for _ in range(n_boot):
lab = [rng.choice(labeled) for _ in labeled]
jud = [rng.choice(judged) for _ in judged]
try:
value = corrected(sum(j == "pass" for j in jud) / len(jud), *rates(lab))
except ZeroDivisionError: # a resample with no passes or no fails
continue
if value == value: # skip NaN from resamples where the judge is at chance
samples.append(value)
samples.sort()
return point, samples[int(0.025 * len(samples))], samples[int(0.975 * len(samples))]
# test split: 60 human passes (judge passed 57), 20 human fails (judge failed 16)
labeled = ([("pass", "pass")] * 57 + [("pass", "fail")] * 3
+ [("fail", "fail")] * 16 + [("fail", "pass")] * 4)
judged = ["pass"] * 850 + ["fail"] * 150 # judge verdicts on 1,000 unlabeled answers
point, low, high = estimate(labeled, judged)
print(f"raw 0.850, corrected {point:.3f}, 95% interval {low:.3f} to {high:.3f}")Running it gives:
raw 0.850, corrected 0.867, 95% interval 0.786 to 0.954The interval is wide, and most of the width comes from having only 20 labeled failures. That's useful information: to narrow it, label more failures, not more passes.
When the correction matters
- Reporting a number to others. A corrected rate with an interval is honest. A raw judge rate is a claim about the judge as much as about DocsBot.
- Comparing versions. If both versions are graded by the same judge, the judge's bias affects both, and comparisons are less distorted than absolute numbers. Day 21 covers comparisons.
- Judges near chance. If TPR + TNR is close to 1, the judge carries almost no information and the correction explodes. Fix the judge instead.
Check yourself
Upward. The judge passes 45 percent of true failures, so it reports more passes than a human would find. The correction formula uses TPR and TNR to estimate the true pass rate from the judge's raw rate.
Label more outputs in the class with fewer labels, usually failures. The uncertainty in TNR from a handful of failures dominates the width.
TPR + TNR equals 1, so the judge's verdicts are independent of the truth. The formula divides by zero because there's no signal to recover.
When the judge's TPR or TNR is clearly below 1 and you report absolute pass rates or compare versions whose failure rates differ. Small judge errors can then swing conclusions.
Uncertainty from both the evaluated examples and the human-labeled set used to estimate TPR and TNR, by resampling both many times and recomputing the corrected rate.