pass@k vs pass^k: AI Agent Reliability Explained
On this page
Why one run per test hides flakiness
You run DocsBot's 40 agent scenarios once and 34 pass: 85 percent. Run them again and a different 33 pass. Some scenarios pass every time, some never, and a handful flip. A single run can't tell those apart, and the flippers are exactly the ones that will annoy real customers.
So run each scenario several times, typically 3 to 10, and look at the pattern per scenario.
pass@k vs pass^k
Say a scenario ran n times and passed c times.
- pass@k is the chance that at least one of k attempts succeeds. It comes from code generation, where you can try several solutions and keep one that passes the tests. The unbiased estimate, from the 2021 Codex paper by Chen and colleagues, is 1 − C(n−c, k) / C(n, k).
- pass^k (read "pass hat k") is the chance that all k attempts succeed. It was introduced by τ-bench in 2024 to measure agent consistency. The estimate is C(c, k) / C(n, k).
Here C(a, b) means "a choose b", the number of ways to pick b items from a.
For a customer-facing agent, pass^k is the honest number. Each customer gets one attempt. If the same request succeeds 7 times in 10, one customer in three, roughly, has a bad experience, and you can't pick the good run for them. The τ-bench authors reported that even strong models scored well below their single-run success rate on pass^8 in the retail domain, which is the kind of gap a single run hides.
Computing pass@k and pass^k
# reliability.py: pass@k and pass^k from repeated runs
from math import comb
def pass_at_k(n: int, c: int, k: int) -> float:
return 1.0 - comb(n - c, k) / comb(n, k)
def pass_hat_k(n: int, c: int, k: int) -> float:
return comb(c, k) / comb(n, k)
# scenario -> number of passes out of 8 runs
results = {"add-seat": 8, "switch-annual": 7, "resend-invoice": 5, "refund-dispute": 2, "sso-question": 8}
n = 8
for k in (1, 4, 8):
at = sum(pass_at_k(n, c, k) for c in results.values()) / len(results)
hat = sum(pass_hat_k(n, c, k) for c in results.values()) / len(results)
print(f"k={k}: pass@k {at:.2f} pass^k {hat:.2f}")k=1: pass@k 0.75 pass^k 0.75
k=4: pass@k 0.96 pass^k 0.51
k=8: pass@k 1.00 pass^k 0.40At k = 1 both equal the plain pass rate. As k grows, pass@k climbs toward "everything works sometimes" and pass^k falls toward "only the rock-solid scenarios count". The gap between them is a direct measure of inconsistency.
What to do with flaky scenarios
Sort scenarios into three groups by their pass count:
- Always pass: good. Keep them as regression tests.
- Never pass: a capability gap. These need design work, not luck.
- Sometimes pass: the most informative group. Read a passing and a failing trace side by side. The difference usually points to an ambiguous tool description, a missing instruction, or a step where the model guesses.
Lowering temperature can reduce variation, but it rarely fixes the cause, and some models and settings don't allow it. Fix the ambiguity instead.
Balancing cost and repeated runs
Repeated runs multiply cost. A practical pattern: run every scenario 3 times in pull request checks, and 8 to 10 times nightly or before releases. Track pass^k for the scenarios where a failure hurts customers, and plain pass rate for the rest.
Check yourself
It can do almost every task sometimes, but does fewer than half of them reliably. For customers who get one attempt, it's inconsistent.
In code generation you can generate several candidates and keep one that passes tests. A support customer gets one response, so every attempt has to succeed.
Compare a passing and a failing trace to find where they diverge, often an ambiguous instruction or tool description, and fix that rather than relying on lower temperature.
pass@k is the chance that at least one of k runs succeeds. pass^k is the chance that all k runs succeed, which measures how reliably the agent works every time.
Run many repetitions on a small set of important or flaky scenarios, and fewer on the rest. Increase repetitions before major releases.