Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 5 · Lesson 26 · 4 min read

LLM Eval Regression Gates in CI with GitHub Actions

On this page
  1. Running evals in CI as regression gates
  2. Three kinds of CI eval checks
  3. An eval gate script
  4. Wiring the eval gate into GitHub Actions
  5. Practical limits of CI eval gates
  6. Check yourself

Running evals in CI as regression gates

The suite exists. It runs when someone remembers. Two prompt tweaks and a model upgrade later, citation rates have slipped from 97 to 88 percent, and nobody can say which change did it.

The fix is the same as for tests: run evals automatically on every change that can affect behavior, and fail the build when something important gets worse. The design question is what "worse" means when scores are noisy.

Three kinds of CI eval checks

CheckRuleDocsBot examples
Hard gateMust pass on every exampleNo other customer's data in output, valid tool JSON, handoff on legal threats
Rate regressionPaired difference against baseline must not be clearly negativeGroundedness, citation rate, handoff recall
Newly broken listReport only, for reviewExamples that passed on main and fail on this branch

Hard gates are simple. Rates need the statistics from Day 21, or you'll either block good changes on noise or wave through real regressions.

Each pull request runs the suite, compares with main's stored results, and fails on hard-gate failures or clear regressions.
Each pull request runs the suite, compares with main's stored results, and fails on hard-gate failures or clear regressions.

An eval gate script

The suite writes one result per example and check. The gate compares the branch with the latest results from main:

Python
# gate.py: compare this branch's eval results with main's and decide pass or fail
import json, math, sys

HARD = {"no_data_leak", "valid_tool_json", "legal_handoff"}
RATES = {"grounded": 0.02, "has_citation": 0.02}     # tolerated drop before we block

def load(path):  # {"example_id": {"check": 0 or 1}}
    return json.load(open(path))

def gate(base: dict, head: dict) -> list[str]:
    failures = []
    for ex_id, checks in head.items():
        for check in HARD & checks.keys():
            if checks[check] == 0:
                failures.append(f"HARD {check} failed on {ex_id}")
    for check, tolerance in RATES.items():
        ids = [i for i in head if check in head[i] and check in base.get(i, {})]
        diffs = [head[i][check] - base[i][check] for i in ids]
        n = len(diffs)
        mean = sum(diffs) / n
        sd = math.sqrt(sum((d - mean) ** 2 for d in diffs) / (n - 1)) if n > 1 else 0.0
        upper = mean + 1.96 * sd / math.sqrt(n)
        broken = [i for i in ids if base[i][check] == 1 and head[i][check] == 0]
        print(f"{check}: {mean:+.3f} (95% upper bound {upper:+.3f}), newly broken: {broken[:10]}")
        if upper < -tolerance:
            failures.append(f"RATE {check} dropped {mean:+.3f}, clearly beyond -{tolerance}")
    return failures

if __name__ == "__main__":
    problems = gate(load(sys.argv[1]), load(sys.argv[2]))
    print("\n".join(problems) or "eval gate passed")
    sys.exit(1 if problems else 0)

The rate rule blocks only when even the optimistic end of the interval shows a drop larger than the tolerance. A 1 point dip on 150 examples passes, because it's within noise. A 10 point drop fails. Everything in between passes the gate but shows up in the newly broken list for a human to read.

Wiring the eval gate into GitHub Actions

YAML
# .github/workflows/evals.yml
name: evals
on:
  pull_request:
    paths: ["docsbot/**", "prompts/**", "evals/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r evals/requirements.txt
      - run: python evals/run_suite.py --split dev --out head.json
        env: { ANTHROPIC_API_KEY: "${{ secrets.ANTHROPIC_API_KEY }}" }
      - run: gh run download --name eval-baseline --dir baseline
        env: { GH_TOKEN: "${{ github.token }}" }
      - run: python evals/gate.py baseline/head.json head.json

A separate workflow on pushes to main runs the same suite and uploads head.json as the eval-baseline artifact, so every PR compares against the current main. The paths filter keeps evals off changes that can't affect DocsBot's behavior, which saves time and money.

Practical limits of CI eval gates

  • Cost and time. Run the dev split with code checks and one or two judges on every PR. Run the full suite, repeats and the held-out test split nightly.
  • Don't tune against the gate. If people tweak prompts until the gate goes green, the dev set slowly stops representing reality. Refresh it from production regularly (Day 27).
  • Model changes count. A provider's new model version is a change, too. Pin model versions and route upgrades through the same gate.

Check yourself

No. The interval includes no drop at all, so the change is within noise. The newly broken list still shows which examples to review.

A single failure is a serious privacy incident with legal consequences. It isn't acceptable at any rate, so any failure must block the merge, regardless of how well everything else scores.

Small drops within tolerance accumulate, and the baseline moves with them. Track absolute scores over time and compare against a fixed reference release periodically, not only the latest main.

Hard rules that must never fail, regression checks comparing a rate against the main branch with uncertainty, and informational metrics that are reported but don't block.

Cost and time limit dataset size, so small regressions go undetected, and judge noise causes occasional false blocks. Back them up with fuller nightly runs.