Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 1 · Lesson 5 · 4 min read

Synthetic Test Data for LLM Evals with Dimensions

On this page
  1. Why asking a model for 100 test questions doesn't work
  2. Step 1: choose dimensions
  3. Step 2: pick dimension tuples on purpose
  4. Step 3: generate a question for each tuple
  5. Checking synthetic test data
  6. Check yourself

Why asking a model for 100 test questions doesn't work

You need more examples about plan limits, and production has only a few. The obvious move is to ask a model: "Write 100 customer questions about Acme." You get 100 polite, well-formed questions that look alike, use the product's own vocabulary, and miss the cases you needed. Real customers write "cant add ppl to my team??" and mention their plan only when it's relevant.

The fix is to control the variety yourself and let the model handle only the wording.

Step 1: choose dimensions

A dimension is one way inputs vary that affects whether DocsBot succeeds. Take them from your failure modes. For DocsBot:

DimensionValues
Topicbilling, account access, SSO, data export, integrations
Planstarter, pro, enterprise
Personanew admin, developer, frustrated end user
Answerable from docsyes, partly, no

Each combination of values, such as (SSO, starter, frustrated end user, no), is a tuple. That tuple describes a test case precisely: a Starter user asking about SSO, which Starter doesn't include, so the right behavior is a clear "not on your plan" rather than a setup guide.

Step 2: pick dimension tuples on purpose

There are 5 × 3 × 3 × 3 = 135 combinations. You don't need all of them. Sample to cover each value, then add the combinations your error analysis says are risky:

Code
# tuples.py: choose dimension combinations to generate questions for
import itertools, random

DIMENSIONS = {
    "topic": ["billing", "account access", "sso", "data export", "integrations"],
    "plan": ["starter", "pro", "enterprise"],
    "persona": ["new admin", "developer", "frustrated end user"],
    "answerable": ["yes", "partly", "no"],
}
random.seed(11)
all_tuples = [dict(zip(DIMENSIONS, combo)) for combo in itertools.product(*DIMENSIONS.values())]
chosen = random.sample(all_tuples, 30)
# known risk from error analysis: plan-gated features asked by lower plans
chosen += [t for t in all_tuples if t["plan"] == "starter" and t["topic"] == "sso" and t not in chosen]
print(len(chosen), "tuples")

Review the list before generating. It's quick, and it's where you catch combinations that make no sense.

Step 3: generate a question for each tuple

Now the model writes the wording, one tuple at a time, with instructions to sound like a real user. Structured output keeps the result parseable:

Python
# generate.py: write realistic questions for each tuple
import anthropic, json
from pydantic import BaseModel

client = anthropic.Anthropic()

class Questions(BaseModel):
    questions: list[str]

def write_questions(t: dict, n: int = 3) -> list[str]:
    prompt = (
        f"Write {n} different messages a {t['persona']} on Acme's {t['plan']} plan might send "
        f"to support about {t['topic']}. Whether the docs can answer it: {t['answerable']}. "
        "Write like real users: vary length, skip greetings sometimes, allow typos and "
        "vague wording. Don't use Acme's feature names unless a user would know them."
    )
    response = client.messages.parse(
        model="claude-opus-5-5", max_tokens=1000,
        messages=[{"role": "user", "content": prompt}],
        output_format=Questions,
    )
    return response.parsed_output.questions

with open("synthetic.jsonl", "w") as f:
    for i, t in enumerate(chosen):                     # from tuples.py
        for j, q in enumerate(write_questions(t)):
            f.write(json.dumps({"id": f"syn-{i}-{j}", "input": q, "plan": t["plan"],
                                "tags": [t["topic"], f"answerable_{t['answerable']}"],
                                "origin": "synthetic"}) + "\n")

Generating from tuples gives you two things a single big prompt can't: control over coverage, and tags for free. Every synthetic example knows which slice it belongs to.

Checking synthetic test data

Synthetic data needs the same review as anything else:

  • Read a sample. Do these sound like your users? Compare a few with real traces side by side.
  • Run them through the app, then do error analysis on the results. Synthetic inputs are only useful if they expose real behavior.
  • Label expected behavior yourself. The generator wrote questions, not ground truth. A domain expert decides what a good answer is.
  • Keep it a minority. Mark origin: synthetic and make sure real traffic still dominates the dataset once you have it.

Check yourself

One big prompt produces similar, generic questions and you can't control coverage. Tuples fix the variety you need and leave only the wording to the model.

The generator copied the tuple into the text. Tighten the prompt to describe the user without stating the dimensions, and filter out questions that echo them.

No. It generated inputs only. Expected behavior must come from the docs and a domain expert, or the eval measures agreement with another model.

Axes of variation that matter for your app, such as user persona, plan, topic and question phrasing. Combining values from each gives tuples that cover realistic, varied cases.

Synthetic questions fill coverage gaps, like rare plans or topics, but real traffic has phrasing and messiness no generator fully predicts. Together they make the dataset both broad and realistic.