Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 5 · Lesson 24 · 4 min read

DeepEval Tutorial: LLM Evals as Pytest Tests

On this page
  1. DeepEval: evals in pytest
  2. DocsBot evals as parametrized tests
  3. What DeepEval's G-Eval metric does
  4. The catch with pass/fail test framing
  5. Check yourself

DeepEval: evals in pytest

If your team runs pytest on every change, the lowest-friction place for evals is right next to the unit tests. DeepEval is an open-source Python library built for that. Each example becomes a test case, metrics become assertions, and failures show up in the same report developers already read.

It also ships many ready-made metrics: G-Eval for custom criteria, faithfulness and answer relevancy for RAG, tool correctness for agents. They're convenient. As with every library metric, read how each one is computed before you trust its numbers.

DocsBot evals as parametrized tests

Python
# test_docsbot.py: DocsBot evals as pytest tests with DeepEval
import json
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.models import AnthropicModel
from deepeval.test_case import LLMTestCase, SingleTurnParams

from docsbot import answer_with_context          # your app: returns (answer, retrieved_pages)

judge_model = AnthropicModel(model="claude-opus-5-5")

grounded = GEval(
    name="Grounded",
    criteria=("Every factual claim about Acme's product in the actual output is stated in or "
              "directly implied by the retrieval context. One unsupported claim fails."),
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.RETRIEVAL_CONTEXT],
    model=judge_model,
    strict_mode=True,                             # binary: score is 1 or 0
)

examples = [json.loads(line) for line in open("docsbot_dev.jsonl")]

@pytest.mark.parametrize("ex", examples, ids=[e["id"] for e in examples])
def test_docsbot(ex):
    answer, pages = answer_with_context(ex["input"])
    case = LLMTestCase(input=ex["input"], actual_output=answer,
                       expected_output=ex.get("expected"), retrieval_context=pages)
    assert "https://docs.acme.example/" in answer, "no citation"     # code check first
    assert_test(case, [grounded])

Run it with plain pytest, or with DeepEval's runner for a richer report:

Code
pip install deepeval anthropic
pytest test_docsbot.py
deepeval test run test_docsbot.py

Each example becomes its own test with the example id in its name, so a failure reads like test_docsbot[pw-reset-01] FAILED: Grounded (score 0.0), with the judge's reason underneath.

What DeepEval's G-Eval metric does

G-Eval comes from a 2023 paper on using LLMs to grade generated text. DeepEval's version takes your criteria, has the model write evaluation steps from them, then scores the test case against those steps. By default it returns a score between 0 and 1 and passes above a threshold.

That default is a scale score, which this course advised against on Day 8. strict_mode=True makes it binary. Better still, write the criteria like a judge prompt: one failure mode, with clear pass and fail conditions. And align it with human labels, exactly as on Day 9. G-Eval is a convenient way to run a judge, not a substitute for checking one.

The catch with pass/fail test framing

Pytest thinks in pass or fail per test. An eval suite thinks in rates. If 3 of 120 examples fail the groundedness check, pytest reports a red build, even though 97.5 percent might be your best result ever.

Two ways to handle it:

  • Hard gates as tests. Use pytest assertions for things that must never fail: no leaked customer data, valid JSON, citation present. A single failure should block a merge.
  • Rates as reports. Run judge-based metrics across the dataset, compute the pass rate with its error bar, and compare with the baseline. Day 26 turns that into a CI gate. DeepEval's evaluate() function runs metrics over a list of test cases without pytest for this purpose.
TipDeepEval can send results to its hosted platform, Confident AI. It's optional. Everything in this lesson runs locally and writes to your terminal.

Check yourself

It passes the default threshold of 0.5, but the number is a scale score that's hard to interpret. Use strict mode for a binary verdict and define pass and fail in the criteria.

Keep must-never-fail checks as hard test assertions, and treat judged qualities like tone as rates compared with a baseline, rather than per-example test failures.

It's a cheap, deterministic code check. If it fails, there's no need to spend a judge call, and the failure message is unambiguous.

It asks a model to grade an output against criteria, often generating evaluation steps first, and turns the result into a score between 0 and 1 with a threshold for passing.

LLM quality is a rate, so a few failures are expected. Failing the whole suite on any failure trains people to ignore or rerun it. Gate on rates and hard rules instead.