Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 5 · Lesson 23 · 4 min read

promptfoo Tutorial: Compare Prompts and Models

On this page
  1. promptfoo: a test matrix in one file
  2. The promptfoo config explained
  3. Running and comparing promptfoo results
  4. Keeping llm-rubric assertions honest
  5. Check yourself

promptfoo: a test matrix in one file

You have two candidate system prompts for DocsBot and want to know which is better across your test cases, maybe on two models. That's a grid: prompts × models × tests. promptfoo, an open-source CLI, is built around exactly that grid. You describe it in YAML, run one command, and get a table with every cell graded.

It suits teams who want evals without writing a framework, including people who don't write Python. The AI security course used promptfoo's red teaming features; here you use its core eval features.

The promptfoo config explained

YAML
# promptfooconfig.yaml
description: DocsBot prompt comparison
prompts:
  - id: file://prompts/v1.txt
    label: v1
  - id: file://prompts/v2.txt
    label: v2
providers:
  - anthropic:messages:claude-opus-5-5
defaultTest:
  options:
    provider: anthropic:messages:claude-opus-5-5     # model that grades llm-rubric assertions
  assert:
    - type: regex
      value: 'https://docs\.acme\.example/[a-z0-9\-/]+'
      metric: has_citation
    - type: javascript
      value: output.split(/\s+/).length <= 120
      metric: concise
tests:
  - vars:
      question: How long is a password reset link valid?
      plan: pro
    assert:
      - type: icontains
        value: 1 hour
  - vars:
      question: Can I set up SSO on the Starter plan?
      plan: starter
    assert:
      - type: not-icontains
        value: configure sso
      - type: llm-rubric
        value: Says SSO needs the Enterprise plan and does not give setup steps.

The prompt files use {{question}} and {{plan}} as variables, filled from each test's vars. Assertions in defaultTest run on every test; each test adds its own. The metric names group results so the summary shows a has_citation rate and a concise rate per prompt.

Assertion types cover the grader kinds from this course:

AssertionGrader kind
contains, icontains, regex, is-jsonCode checks (Day 6)
javascript, pythonCustom code graders
similarEmbedding similarity (Day 7, with its caveats)
llm-rubricLLM judge with your criterion (Day 8)

Running and comparing promptfoo results

Shell
npx promptfoo@latest eval                 # runs the full matrix
npx promptfoo@latest eval -o results.json # also writes machine-readable results
npx promptfoo@latest view                 # opens the side-by-side viewer

The terminal table shows each test as a row and each prompt as a column, with pass or fail in every cell. The viewer lets you filter to failures and read outputs side by side, which is often where you see why v2 is better: it links pages more consistently, or it stops over-explaining.

For tests at scale, keep cases out of the main file with tests: file://tests/docsbot_dev.yaml, or load a CSV where columns become variables. That way the dataset stays the single source from Day 4.

Keeping llm-rubric assertions honest

llm-rubric makes it easy to add a judge in one line. That's both its strength and its risk. A one-line rubric is a vague judge prompt, and it hasn't been aligned with anyone. For checks that gate decisions, write the rubric with the same care as Day 8: one criterion, clear pass and fail conditions. Spot-check its verdicts against a human on a sample, as on Day 9.

Also pin the grading model with defaultTest.options.provider. Otherwise grading depends on promptfoo's default grader, which may not be the model you validated.

Notenpx promptfoo@latest is convenient for trying things. In CI, pin a version, as the security course did with promptfoo@0.124, so an upgrade doesn't change grading behavior between runs.

Check yourself

3 × 2 × 50 = 300 outputs, each graded by the assertions that apply to its test. promptfoo shows them as a matrix, so you can see which prompt and model combination fails which tests.

It's a vague judge with no criterion. Rewrite it as one specific pass or fail condition and check its verdicts against human labels.

Reference it from tests as an external file, such as a YAML, JSON or CSV file whose fields become variables in the prompt. The dataset then stays the single source of truth, versioned alongside other evals.

A rubric like "answer is good" gives the grading model no criterion, so verdicts are inconsistent. A specific rule, like "states that SSO requires Enterprise", can be checked.

For quickly comparing prompts and models side by side on a shared test set, with simple assertions, especially during prompt development.