LLM Eval Statistics: Error Bars and Paired Comparison
On this page
Is 82 percent really better than 79?
The new prompt scores 82 percent on 150 examples; the old one scored 79. The team wants to ship. But run the old prompt again and it might score 81. Is 3 points a real improvement, or noise from which examples happened to be in the set and how the model sampled today?
Statistics answers that question cheaply. Anthropic's 2024 research note "A statistical approach to model evaluations" makes the case for the same few habits shown here: report standard errors, compare with paired differences, and plan sample sizes.
Standard error and confidence intervals for a pass rate
Treat each example as a coin flip that passes with probability p. With n examples and an observed pass rate p̂, the standard error is:
SE = √(p̂ × (1 − p̂) / n)
A 95 percent confidence interval is roughly p̂ ± 1.96 × SE. For 82 percent on 150 examples, SE ≈ 3.1 points, so the interval runs from about 76 to 88 percent. The old prompt's 79 sits comfortably inside it.
That doesn't settle the comparison, though. Both versions ran on the same examples, and you can use that.
Paired comparison of two versions
Most of the variation in pass rates comes from examples: some are easy, some hard. When both versions see the same examples, those differences cancel out if you look at the per-example difference instead of comparing two separate rates.
# compare.py: paired comparison of two versions on the same examples
import math
def summary(scores: list[float]) -> tuple[float, float]:
n = len(scores)
mean = sum(scores) / n
var = sum((s - mean) ** 2 for s in scores) / (n - 1)
return mean, math.sqrt(var / n)
def paired(old: list[int], new: list[int]) -> None:
diffs = [b - a for a, b in zip(old, new)]
mean, se = summary(diffs)
print(f"old {sum(old) / len(old):.3f} new {sum(new) / len(new):.3f}")
print(f"difference {mean:+.3f} 95% CI {mean - 1.96 * se:+.3f} to {mean + 1.96 * se:+.3f}")
print(f"changed: {sum(d == 1 for d in diffs)} fixed, {sum(d == -1 for d in diffs)} broken")
# 150 examples: 109 pass in both, 17 fail in both, 14 fixed by the new prompt, 10 broken by it
old = [1] * 109 + [0] * 17 + [0] * 14 + [1] * 10
new = [1] * 109 + [0] * 17 + [1] * 14 + [0] * 10
paired(old, new)old 0.793 new 0.820
difference +0.027 95% CI -0.037 to +0.091
changed: 14 fixed, 10 brokenThe interval includes zero, so this run doesn't show that the new prompt is better. The last line matters as much as the interval: the new prompt fixed 14 examples and broke 10. Read those 10 before shipping. A version that's better on average can still break something important.
Three more sources of eval noise
- Repeated sampling. The same example can pass or fail on reruns (Day 20). Running each example several times and averaging per example shrinks this part of the noise.
- Clustered examples. If your dataset has ten questions generated from the same doc page, they aren't independent. Standard errors computed as if they were come out too small. Compute the SE over clusters, such as pages or scenarios, when examples come in groups.
- Judge error. If a judge grades the outputs, its mistakes add noise and bias (Day 10).
How many eval examples you need to detect a difference
Turn it around: to detect a difference of δ with reasonable confidence, you need roughly n ≈ 16 × p(1 − p) / δ² examples per version, for two independent samples at 80 percent power. "80 percent power" means that if the improvement is real, an experiment this size detects it 4 times out of 5; the other time, it looks like noise. For a pass rate near 80 percent and a 3 point difference, that's about 16 × 0.16 / 0.0009, or roughly 2,800 examples.
Pairing helps a lot, often cutting that by a large factor, because only the examples that change outcome carry information. Still, the lesson holds: with 150 examples, only differences of around 10 points or more show up reliably. Either grow the dataset, or accept that small improvements need more evidence than one run.
Check yourself
The paired difference with its confidence interval, and the counts of examples fixed and broken. With 100 examples, a 2 point difference will almost certainly be within noise.
Both versions saw the same examples, so example difficulty cancels out in the differences. That gives a much tighter interval than comparing two independent rates.
Questions from the same page are correlated, since they share content and failure causes, so you effectively have fewer independent observations than 300. Compute standard errors over page clusters, which gives a wider and more honest interval.
Read the 10 broken examples. Even if the average barely moved, one of them may be a failure you can't accept, and the interval says the improvement isn't proven yet.
Usually far more than people expect, often over a thousand for independent comparisons. Paired comparisons on the same examples need fewer, which is why they're preferred.