Inspect AI Tutorial: Running LLM Evals
On this page
Why use an eval framework like Inspect AI?
You could keep running eval scripts by hand. After a few weeks you'd have rebuilt the same pieces every framework has: loading datasets, calling models concurrently with retries, collecting scores, saving logs, and viewing results. A framework gives you those for free and makes runs comparable.
Inspect AI is a Python framework from the UK AI Security Institute, used widely for model and agent evaluations. It's a good fit when your team writes Python and wants code-first evals with a strong log viewer. Every run writes a log file that records every sample, message and score.
Inspect AI concepts: datasets, solvers and scorers
An Inspect task has the same parts as on Day 1: a dataset, a solver (how to produce the output, in this case a system message and a model call) and one or more scorers. Here's DocsBot's dev set from Day 4 with a built-in scorer, a custom code grader from Day 6 and a model-graded check:
# docsbot_eval.py: DocsBot's eval suite as an Inspect AI task
import re
from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, json_dataset
from inspect_ai.scorer import (CORRECT, INCORRECT, Score, Target, accuracy, includes,
model_graded_qa, scorer, stderr)
from inspect_ai.solver import TaskState, generate, system_message
DOC_LINK = re.compile(r"https://docs\.acme\.example/[a-z0-9\-/]+")
@scorer(metrics=[accuracy(), stderr()])
def has_citation():
async def score(state: TaskState, target: Target) -> Score:
found = DOC_LINK.search(state.output.completion)
return Score(value=CORRECT if found else INCORRECT,
explanation=found.group(0) if found else "no docs link")
return score
@task
def docsbot(split: str = "dev"):
return Task(
dataset=json_dataset(f"docsbot_{split}.jsonl",
FieldSpec(input="input", target="expected", id="id", metadata=["tags"])),
solver=[system_message("You are Acme's documentation assistant. Answer from the docs "
"and link the page you used."),
generate()],
scorer=[includes(), has_citation(), model_graded_qa()],
)A few details worth noticing:
FieldSpecmaps your JSONL fields onto Inspect's sample fields, so the dataset file from Day 4 works unchanged.includes()passes if the target text appears in the output, which suits short facts like "1 hour".- Every scorer reports
accuracyandstderr, so you get Day 21's error bar automatically. - In a real setup the solver would call your DocsBot pipeline, with retrieval and tools, rather than a bare model. Inspect supports custom solvers and agents for that.
Running an Inspect AI eval
pip install inspect-ai anthropic
export ANTHROPIC_API_KEY=...
inspect eval docsbot_eval.py --model anthropic/claude-opus-5-5 \
--model-role grader=anthropic/claude-opus-5-5 # model used by model_graded_qa
inspect eval docsbot_eval.py -T split=test --model anthropic/claude-opus-5-5 # task argument
inspect eval docsbot_eval.py --model anthropic/claude-opus-5-5 --epochs 5 --epochs-reducer mean
inspect view # browse logs in the browser-T passes task arguments, so the same file runs dev or test. --epochs repeats every sample, which gives you the repeated runs from Day 20. The reducer decides how repeats combine into one score per sample: mean gives the average, and at_least_5 passes a sample only if all five epochs passed, which is pass^5.
To check the wiring without spending money, run with --model mockllm/model. It returns placeholder text, so scores are meaningless, but you'll catch dataset and scorer errors.
Reading Inspect AI logs, not just the score table
The summary table shows accuracy per scorer. The value is in inspect view: click any sample to see the full conversation, the score, and the explanation your scorer returned. That's your error analysis view from Day 2, now attached to every run. When a score drops, open the samples that changed before touching the prompt.
model_graded_qa's default template. Its built-in grading prompt is generic, and your judge is aligned to your criteria.Check yourself
Pass the task argument on the command line: inspect eval docsbot_eval.py -T split=test --model .... Task parameters let one task definition run against dev or test without editing code.
Run with --epochs k and the at_least_k reducer, so a sample counts as passing only if every epoch passed. It turns repeated runs into a reliability measure per sample.
Open the run in inspect view, find samples that passed before and fail now, and read their transcripts and score explanations.
Solvers define how the model is run on each sample, such as a system prompt and generation or an agent loop. Scorers grade the output, with code or model-graded checks.
Logs show each sample's transcript and the scorer's explanation, which reveal whether failures are real or caused by grader mistakes.