Build a RAG Eval Set from Your Own Docs
On this page
Why labeled RAG data is the bottleneck
Days 12 and 13 need, for each question, the relevant pages and a reference answer. Real traffic gives you realistic questions but no labels; someone has to find the right page for each. For a docs site with 400 pages, you want broader coverage faster.
The trick is to run the process backwards. Start from a page, generate questions that page answers, and the label comes for free: the source page is relevant by construction.
Generating RAG test questions from chunks
Use the same chunks your retriever indexes, so the labels match what retrieval can return:
# rag_questions.py: generate labeled questions from documentation chunks
import anthropic, json
from pydantic import BaseModel
client = anthropic.Anthropic()
class QA(BaseModel):
question: str
answer: str
class QASet(BaseModel):
items: list[QA]
def questions_for(chunk: dict, n: int = 2) -> list[QA]:
prompt = (
f"Here is a section of Acme's product docs, titled '{chunk['title']}'.\n"
f"<section>{chunk['text']}</section>\n\n"
f"Write {n} questions a customer might ask that this section fully answers, with a short "
"answer taken from the section. Write the questions the way a customer would, in their own "
"words, not copying phrases from the section. Don't mention the section or the docs."
)
response = client.messages.parse(model="claude-opus-5-5", max_tokens=1000,
messages=[{"role": "user", "content": prompt}],
output_format=QASet)
return response.parsed_output.items
def build(chunks: list[dict], path: str = "rag_eval.jsonl") -> None:
with open(path, "w") as f:
for chunk in chunks:
for i, qa in enumerate(questions_for(chunk)):
f.write(json.dumps({"id": f"{chunk['id']}-q{i}", "input": qa.question,
"expected": qa.answer, "relevant": [chunk["page"]],
"origin": "generated-from-docs"}) + "\n")The instruction to avoid copying phrases matters. Generated questions tend to reuse the section's exact wording, which makes retrieval look far better than it is with real users.
Filtering synthetic RAG questions
Expect to throw away a good share. Three filters catch most of the bad ones:
- Too lexical. If most of the question's words appear in the chunk, keyword search finds it trivially. Drop or rewrite those.
- Not answerable from the chunk. Run a quick judge: "Can this question be fully answered from this section alone?" Questions that need other pages or general knowledge get relabeled or dropped.
- Ambiguous. "How do I change it?" Change what? A reader with no context can't tell.
The lexical filter is plain code:
# overlap.py: flag questions that copy too much of their source chunk
import re
STOP = {"the", "a", "an", "to", "of", "in", "is", "do", "i", "how", "can", "my", "for", "and", "what", "on"}
def words(text: str) -> set[str]:
return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}
def overlap(question: str, chunk_text: str) -> float:
q = words(question)
return len(q & words(chunk_text)) / len(q) if q else 0.0
chunk = "To export your workspace data, open Settings, choose Data, then Export. Exports are CSV files."
for q in ["How do I export workspace data from Settings?", "can i get all our stuff out as a spreadsheet"]:
print(f"{overlap(q, chunk):.2f} {q}")0.80 How do I export workspace data from Settings?
0.00 can i get all our stuff out as a spreadsheetThe first question would make any retriever look perfect. The second is the kind real users send and the kind your eval needs. A threshold around 0.6 to 0.7 is a reasonable start; tune it by reading what gets flagged.
Mixing synthetic questions with real traffic
Generated sets have known blind spots. Questions come one page at a time, so they rarely need two pages, and nobody asks about features that don't exist. Real users do both. Keep generated questions as one tagged slice, add multi-page and unanswerable questions by hand, and keep labeling a sample of production questions.
Check yourself
Generated questions probably reuse the docs' wording, so retrieval is easy. Filter by lexical overlap and add real user questions, which use different vocabulary.
The relevance label then matches exactly what retrieval can return, so recall is measured correctly, and each question is answerable from one chunk the model would actually see. Questions from whole pages may need text split across chunks.
Questions that need information from several pages, and questions the docs can't answer at all, where the right behavior is to say so. Add those by hand or from production traffic so the eval covers them.
Generators produce questions that copy the chunk's wording, are unanswerable, or are trivially easy. Filtering removes them so the set doesn't inflate retrieval scores.
They reuse the exact words of the chunk they came from, which makes lexical and semantic matching easy. Real users phrase things differently.