LLM Eval Regression Gates in CI with GitHub Actions
On this page
Running evals in CI as regression gates
The suite exists. It runs when someone remembers. Two prompt tweaks and a model upgrade later, citation rates have slipped from 97 to 88 percent, and nobody can say which change did it.
The fix is the same as for tests: run evals automatically on every change that can affect behavior, and fail the build when something important gets worse. The design question is what "worse" means when scores are noisy.
Three kinds of CI eval checks
| Check | Rule | DocsBot examples |
|---|---|---|
| Hard gate | Must pass on every example | No other customer's data in output, valid tool JSON, handoff on legal threats |
| Rate regression | Paired difference against baseline must not be clearly negative | Groundedness, citation rate, handoff recall |
| Newly broken list | Report only, for review | Examples that passed on main and fail on this branch |
Hard gates are simple. Rates need the statistics from Day 21, or you'll either block good changes on noise or wave through real regressions.
An eval gate script
The suite writes one result per example and check. The gate compares the branch with the latest results from main:
# gate.py: compare this branch's eval results with main's and decide pass or fail
import json, math, sys
HARD = {"no_data_leak", "valid_tool_json", "legal_handoff"}
RATES = {"grounded": 0.02, "has_citation": 0.02} # tolerated drop before we block
def load(path): # {"example_id": {"check": 0 or 1}}
return json.load(open(path))
def gate(base: dict, head: dict) -> list[str]:
failures = []
for ex_id, checks in head.items():
for check in HARD & checks.keys():
if checks[check] == 0:
failures.append(f"HARD {check} failed on {ex_id}")
for check, tolerance in RATES.items():
ids = [i for i in head if check in head[i] and check in base.get(i, {})]
diffs = [head[i][check] - base[i][check] for i in ids]
n = len(diffs)
mean = sum(diffs) / n
sd = math.sqrt(sum((d - mean) ** 2 for d in diffs) / (n - 1)) if n > 1 else 0.0
upper = mean + 1.96 * sd / math.sqrt(n)
broken = [i for i in ids if base[i][check] == 1 and head[i][check] == 0]
print(f"{check}: {mean:+.3f} (95% upper bound {upper:+.3f}), newly broken: {broken[:10]}")
if upper < -tolerance:
failures.append(f"RATE {check} dropped {mean:+.3f}, clearly beyond -{tolerance}")
return failures
if __name__ == "__main__":
problems = gate(load(sys.argv[1]), load(sys.argv[2]))
print("\n".join(problems) or "eval gate passed")
sys.exit(1 if problems else 0)The rate rule blocks only when even the optimistic end of the interval shows a drop larger than the tolerance. A 1 point dip on 150 examples passes, because it's within noise. A 10 point drop fails. Everything in between passes the gate but shows up in the newly broken list for a human to read.
Wiring the eval gate into GitHub Actions
# .github/workflows/evals.yml
name: evals
on:
pull_request:
paths: ["docsbot/**", "prompts/**", "evals/**"]
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install -r evals/requirements.txt
- run: python evals/run_suite.py --split dev --out head.json
env: { ANTHROPIC_API_KEY: "${{ secrets.ANTHROPIC_API_KEY }}" }
- run: gh run download --name eval-baseline --dir baseline
env: { GH_TOKEN: "${{ github.token }}" }
- run: python evals/gate.py baseline/head.json head.jsonA separate workflow on pushes to main runs the same suite and uploads head.json as the eval-baseline artifact, so every PR compares against the current main. The paths filter keeps evals off changes that can't affect DocsBot's behavior, which saves time and money.
Practical limits of CI eval gates
- Cost and time. Run the dev split with code checks and one or two judges on every PR. Run the full suite, repeats and the held-out test split nightly.
- Don't tune against the gate. If people tweak prompts until the gate goes green, the dev set slowly stops representing reality. Refresh it from production regularly (Day 27).
- Model changes count. A provider's new model version is a change, too. Pin model versions and route upgrades through the same gate.
Check yourself
No. The interval includes no drop at all, so the change is within noise. The newly broken list still shows which examples to review.
A single failure is a serious privacy incident with legal consequences. It isn't acceptable at any rate, so any failure must block the merge, regardless of how well everything else scores.
Small drops within tolerance accumulate, and the baseline moves with them. Track absolute scores over time and compare against a fixed reference release periodically, not only the latest main.
Hard rules that must never fail, regression checks comparing a rate against the main branch with uncertainty, and informational metrics that are reported but don't block.
Cost and time limit dataset size, so small regressions go undetected, and judge noise causes occasional false blocks. Back them up with fuller nightly runs.