Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 3 · Lesson 14 · 4 min read

Build a RAG Eval Set from Your Own Docs

On this page
  1. Why labeled RAG data is the bottleneck
  2. Generating RAG test questions from chunks
  3. Filtering synthetic RAG questions
  4. Mixing synthetic questions with real traffic
  5. Check yourself

Why labeled RAG data is the bottleneck

Days 12 and 13 need, for each question, the relevant pages and a reference answer. Real traffic gives you realistic questions but no labels; someone has to find the right page for each. For a docs site with 400 pages, you want broader coverage faster.

The trick is to run the process backwards. Start from a page, generate questions that page answers, and the label comes for free: the source page is relevant by construction.

Generating RAG test questions from chunks

Use the same chunks your retriever indexes, so the labels match what retrieval can return:

Python
# rag_questions.py: generate labeled questions from documentation chunks
import anthropic, json
from pydantic import BaseModel

client = anthropic.Anthropic()

class QA(BaseModel):
    question: str
    answer: str

class QASet(BaseModel):
    items: list[QA]

def questions_for(chunk: dict, n: int = 2) -> list[QA]:
    prompt = (
        f"Here is a section of Acme's product docs, titled '{chunk['title']}'.\n"
        f"<section>{chunk['text']}</section>\n\n"
        f"Write {n} questions a customer might ask that this section fully answers, with a short "
        "answer taken from the section. Write the questions the way a customer would, in their own "
        "words, not copying phrases from the section. Don't mention the section or the docs."
    )
    response = client.messages.parse(model="claude-opus-5-5", max_tokens=1000,
                                     messages=[{"role": "user", "content": prompt}],
                                     output_format=QASet)
    return response.parsed_output.items

def build(chunks: list[dict], path: str = "rag_eval.jsonl") -> None:
    with open(path, "w") as f:
        for chunk in chunks:
            for i, qa in enumerate(questions_for(chunk)):
                f.write(json.dumps({"id": f"{chunk['id']}-q{i}", "input": qa.question,
                                    "expected": qa.answer, "relevant": [chunk["page"]],
                                    "origin": "generated-from-docs"}) + "\n")

The instruction to avoid copying phrases matters. Generated questions tend to reuse the section's exact wording, which makes retrieval look far better than it is with real users.

Filtering synthetic RAG questions

Expect to throw away a good share. Three filters catch most of the bad ones:

  1. Too lexical. If most of the question's words appear in the chunk, keyword search finds it trivially. Drop or rewrite those.
  2. Not answerable from the chunk. Run a quick judge: "Can this question be fully answered from this section alone?" Questions that need other pages or general knowledge get relabeled or dropped.
  3. Ambiguous. "How do I change it?" Change what? A reader with no context can't tell.

The lexical filter is plain code:

Python
# overlap.py: flag questions that copy too much of their source chunk
import re

STOP = {"the", "a", "an", "to", "of", "in", "is", "do", "i", "how", "can", "my", "for", "and", "what", "on"}

def words(text: str) -> set[str]:
    return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}

def overlap(question: str, chunk_text: str) -> float:
    q = words(question)
    return len(q & words(chunk_text)) / len(q) if q else 0.0

chunk = "To export your workspace data, open Settings, choose Data, then Export. Exports are CSV files."
for q in ["How do I export workspace data from Settings?", "can i get all our stuff out as a spreadsheet"]:
    print(f"{overlap(q, chunk):.2f}  {q}")
Code
0.80  How do I export workspace data from Settings?
0.00  can i get all our stuff out as a spreadsheet

The first question would make any retriever look perfect. The second is the kind real users send and the kind your eval needs. A threshold around 0.6 to 0.7 is a reasonable start; tune it by reading what gets flagged.

Mixing synthetic questions with real traffic

Generated sets have known blind spots. Questions come one page at a time, so they rarely need two pages, and nobody asks about features that don't exist. Real users do both. Keep generated questions as one tagged slice, add multi-page and unanswerable questions by hand, and keep labeling a sample of production questions.

NoteRegenerate the slice when the docs change a lot. A question about a removed feature with a reference answer from the old page now tests the wrong thing.

Check yourself

Generated questions probably reuse the docs' wording, so retrieval is easy. Filter by lexical overlap and add real user questions, which use different vocabulary.

The relevance label then matches exactly what retrieval can return, so recall is measured correctly, and each question is answerable from one chunk the model would actually see. Questions from whole pages may need text split across chunks.

Questions that need information from several pages, and questions the docs can't answer at all, where the right behavior is to say so. Add those by hand or from production traffic so the eval covers them.

Generators produce questions that copy the chunk's wording, are unanswerable, or are trivially easy. Filtering removes them so the set doesn't inflate retrieval scores.

They reuse the exact words of the chunk they came from, which makes lexical and semantic matching easy. Real users phrase things differently.