promptfoo Tutorial: Compare Prompts and Models
On this page
promptfoo: a test matrix in one file
You have two candidate system prompts for DocsBot and want to know which is better across your test cases, maybe on two models. That's a grid: prompts × models × tests. promptfoo, an open-source CLI, is built around exactly that grid. You describe it in YAML, run one command, and get a table with every cell graded.
It suits teams who want evals without writing a framework, including people who don't write Python. The AI security course used promptfoo's red teaming features; here you use its core eval features.
The promptfoo config explained
# promptfooconfig.yaml
description: DocsBot prompt comparison
prompts:
- id: file://prompts/v1.txt
label: v1
- id: file://prompts/v2.txt
label: v2
providers:
- anthropic:messages:claude-opus-5-5
defaultTest:
options:
provider: anthropic:messages:claude-opus-5-5 # model that grades llm-rubric assertions
assert:
- type: regex
value: 'https://docs\.acme\.example/[a-z0-9\-/]+'
metric: has_citation
- type: javascript
value: output.split(/\s+/).length <= 120
metric: concise
tests:
- vars:
question: How long is a password reset link valid?
plan: pro
assert:
- type: icontains
value: 1 hour
- vars:
question: Can I set up SSO on the Starter plan?
plan: starter
assert:
- type: not-icontains
value: configure sso
- type: llm-rubric
value: Says SSO needs the Enterprise plan and does not give setup steps.The prompt files use {{question}} and {{plan}} as variables, filled from each test's vars. Assertions in defaultTest run on every test; each test adds its own. The metric names group results so the summary shows a has_citation rate and a concise rate per prompt.
Assertion types cover the grader kinds from this course:
| Assertion | Grader kind |
|---|---|
contains, icontains, regex, is-json | Code checks (Day 6) |
javascript, python | Custom code graders |
similar | Embedding similarity (Day 7, with its caveats) |
llm-rubric | LLM judge with your criterion (Day 8) |
Running and comparing promptfoo results
npx promptfoo@latest eval # runs the full matrix
npx promptfoo@latest eval -o results.json # also writes machine-readable results
npx promptfoo@latest view # opens the side-by-side viewerThe terminal table shows each test as a row and each prompt as a column, with pass or fail in every cell. The viewer lets you filter to failures and read outputs side by side, which is often where you see why v2 is better: it links pages more consistently, or it stops over-explaining.
For tests at scale, keep cases out of the main file with tests: file://tests/docsbot_dev.yaml, or load a CSV where columns become variables. That way the dataset stays the single source from Day 4.
Keeping llm-rubric assertions honest
llm-rubric makes it easy to add a judge in one line. That's both its strength and its risk. A one-line rubric is a vague judge prompt, and it hasn't been aligned with anyone. For checks that gate decisions, write the rubric with the same care as Day 8: one criterion, clear pass and fail conditions. Spot-check its verdicts against a human on a sample, as on Day 9.
Also pin the grading model with defaultTest.options.provider. Otherwise grading depends on promptfoo's default grader, which may not be the model you validated.
npx promptfoo@latest is convenient for trying things. In CI, pin a version, as the security course did with promptfoo@0.124, so an upgrade doesn't change grading behavior between runs.Check yourself
3 × 2 × 50 = 300 outputs, each graded by the assertions that apply to its test. promptfoo shows them as a matrix, so you can see which prompt and model combination fails which tests.
It's a vague judge with no criterion. Rewrite it as one specific pass or fail condition and check its verdicts against human labels.
Reference it from tests as an external file, such as a YAML, JSON or CSV file whose fields become variables in the prompt. The dataset then stays the single source of truth, versioned alongside other evals.
A rubric like "answer is good" gives the grading model no criterion, so verdicts are inconsistent. A specific rule, like "states that SSO requires Enterprise", can be checked.
For quickly comparing prompts and models side by side on a shared test set, with simple assertions, especially during prompt development.