How to Build an LLM Eval Dataset from Real Traffic
On this page
Why the dataset is the eval
Graders get the attention, but the dataset decides what an eval can see. If none of your examples asks about billing, no grader will notice that billing answers broke. A good dataset looks like real traffic, over-samples the failure modes you found, and stays fixed between runs so scores are comparable.
What one eval example contains
Store examples as JSON Lines, one per line, in the repository next to the code:
{"id": "pw-reset-01", "input": "How long is a password reset link valid?", "expected": "1 hour", "source_doc": "account/reset-password", "plan": "starter", "tags": ["account", "stale_fact"], "origin": "prod-2026-09"}
{"id": "sso-04", "input": "Can we use Okta for login?", "expected_behavior": "answer from the SSO page, no handoff", "source_doc": "security/sso", "plan": "enterprise", "tags": ["security", "unneeded_handoff"], "origin": "prod-2026-09"}
{"id": "refund-legal-02", "input": "I want to sue you over the outage", "expected_behavior": "hand off to a human", "plan": "pro", "tags": ["handoff_required"], "origin": "written"}The fields that matter:
- input: what the user sends, plus any context the app needs, such as the customer's plan.
- expected or expected_behavior: a reference answer when one exists, otherwise a short description of what a good response does. Many LLM tasks have no single correct text, so a behavior is often the honest target.
- tags: the failure modes and product areas this example tests. You'll slice results by them.
- origin: where it came from, so you know how much of the set is real traffic.
Where eval examples come from
| Source | Strength | Watch out for |
|---|---|---|
| Production traces | Real wording and real distribution | Personal data: redact before committing |
| Failures from error analysis | Target known weaknesses | Over-represent past bugs if used alone |
| Hand-written by experts | Cover rare but important cases | Experts write cleaner questions than users |
| Synthetic generation (Day 5) | Fill gaps fast | Can drift from how real users write |
Start with the failing and passing traces you already reviewed. They come with notes, and the notes often become the expected behavior.
How many eval examples you need
Fewer than you think to start, more than you think to make decisions. Around 50 to 100 well-chosen examples is enough to catch big regressions and to begin. To tell a 3 point improvement from noise you need a few hundred, and Day 21 shows how to calculate it. Grow the set when a comparison comes back inconclusive.
Coverage matters more than size. Twenty examples per important failure mode beats 500 random questions where the interesting cases appear twice.
Splitting dev and test sets, and locking the test set
You'll tune prompts against this dataset, and later you'll tune LLM judges against human labels. Anything you tune against stops being a fair test. Keep two parts:
- dev: the set you look at and iterate on daily,
- test: a held-out slice, around 20 to 30 percent, that you run only before a release or a big decision.
# split.py: deterministic dev/test split that stays stable as the file grows
import hashlib, json
def bucket(example_id: str) -> str:
h = int(hashlib.sha256(example_id.encode()).hexdigest(), 16)
return "test" if h % 100 < 25 else "dev"
examples = [json.loads(l) for l in open("docsbot_eval.jsonl")]
for name in ("dev", "test"):
with open(f"docsbot_{name}.jsonl", "w") as f:
for ex in examples:
if bucket(ex["id"]) == name:
f.write(json.dumps(ex) + "\n")Hashing the id instead of shuffling means an example never moves between splits when you add new ones.
Versioning an eval dataset like code
Commit the dataset, review changes in pull requests, and record the dataset version in every eval report. When a score changes, you need to know whether the app or the dataset changed. Tools such as Langfuse datasets (Day 25) add hosted versioning, but a JSONL file in git works well for a long time.
Check yourself
A short description of the expected behavior, such as "answers from the SSO page with a citation, no handoff". Graders can check behavior even when the exact text varies.
A random shuffle reassigns examples on every run, so test examples leak into dev. Hashing keeps each example in the same split forever, even as the file grows.
Coverage matters more than size. Tag examples by area and failure mode, and make sure each important slice has enough examples to show a change.
An id, the input exactly as the app would receive it, any context such as the user's plan, the expected result or criteria for grading, and tags like topic or difficulty for slicing results.
If you tune prompts while looking at test results, the test set stops measuring how the app does on unseen inputs. Tune on dev, and check test only for final decisions.