RAG Retrieval Metrics: Recall@k, MRR and nDCG
On this page
Why grade RAG retrieval separately
DocsBot quotes an old price. Was the current pricing page not retrieved, or was it retrieved and ignored? The fix is completely different: better search in one case, a better prompt in the other. If you only grade final answers, you can't tell.
So evaluate the two stages separately. Retrieval is the easier one to measure, because it has a clear right answer: for each question, which pages contain what's needed. Your dataset already records it in the source_doc field from Day 4.
Retrieval metrics: recall@k, precision@k, MRR and nDCG
For each question the retriever returns a ranked list of pages. Compare it with the relevant pages:
| Metric | Question it answers |
|---|---|
| Recall@k | Of the relevant pages, what share is in the top k? |
| Precision@k | Of the top k results, what share is relevant? |
| MRR (mean reciprocal rank) | How high is the first relevant page? 1 for rank 1, 1/2 for rank 2, and so on, averaged |
| nDCG@k | How good is the whole ordering, giving more credit to relevant pages near the top? |
nDCG sounds harder than it is: each relevant page earns points, and the points shrink the lower it sits in the list ("discounted"), then the total is divided by the best possible score so the result runs from 0 to 1 ("normalized").
For RAG, recall@k is the one to watch first, where k is the number of chunks you put in the prompt. If the right page isn't in the context, the model can't use it. Precision matters second: irrelevant chunks cost tokens and can distract the model.
Computing retrieval metrics in Python
# retrieval_metrics.py: score a retriever against known relevant pages
import math
def recall_at_k(ranked, relevant, k):
return len(set(ranked[:k]) & relevant) / len(relevant)
def precision_at_k(ranked, relevant, k):
return len(set(ranked[:k]) & relevant) / k
def reciprocal_rank(ranked, relevant):
for i, doc in enumerate(ranked, start=1):
if doc in relevant:
return 1 / i
return 0.0
def ndcg_at_k(ranked, relevant, k):
dcg = sum(1 / math.log2(i + 1) for i, doc in enumerate(ranked[:k], start=1) if doc in relevant)
ideal = sum(1 / math.log2(i + 1) for i in range(1, min(len(relevant), k) + 1))
return dcg / ideal
runs = [ # (retrieved ranking, relevant pages)
(["billing/plans", "billing/invoices", "account/users"], {"billing/plans"}),
(["account/users", "account/reset-password", "security/sso"], {"account/reset-password"}),
(["data/import", "integrations/zapier", "data/api"], {"data/export"}),
(["security/sso", "security/saml", "billing/plans"], {"security/sso", "security/saml"}),
]
k = 3
for name, fn in [("recall@3", lambda r, rel: recall_at_k(r, rel, k)),
("precision@3", lambda r, rel: precision_at_k(r, rel, k)),
("MRR", reciprocal_rank),
("nDCG@3", lambda r, rel: ndcg_at_k(r, rel, k))]:
print(f"{name:<12} {sum(fn(r, rel) for r, rel in runs) / len(runs):.2f}")recall@3 0.75
precision@3 0.33
MRR 0.62
nDCG@3 0.66The third question is the interesting one: the export page isn't retrieved at all. No prompt change can fix that answer.
Reading retrieval misses, not just averages
A recall of 0.75 tells you a quarter of questions start without the right page. Pull those questions and read them. On DocsBot you'd likely find patterns:
- Vocabulary mismatch. Users say "download my data"; the page says "export". Hybrid search with keywords plus embeddings, or query rewriting, helps.
- Stale duplicates. The 2023 pricing page outranks the current one. Fix the index, not the retriever.
- Chunking. The answer sits in a chunk that lacks the page title, so it doesn't match. Add titles or headings to each chunk.
Slice recall by tag, too. Retrieval often works well for common topics and fails for the long tail.
Labeling multiple relevant pages
Some questions need two pages, such as "Can Starter users set up SSO?", which needs both the plans page and the SSO page. Record all of them in the dataset. Recall then shows whether the retriever brought back everything the answer needs, which a single source page would hide.
Check yourself
k = 5. Recall at the number of chunks that reach the model tells you whether the answer was possible at all.
At generation: the right page is in the context but the model ignores or misreads it. Check faithfulness and answer correctness (Day 13).
It shows whether the relevant page is near the top. Models tend to use early context more reliably, and you may be able to send fewer chunks if the right one ranks first.
It rewards relevant results more when they rank higher and supports graded relevance, like very relevant versus somewhat relevant, so it measures ranking quality, not just presence.
Misses reveal causes, such as wrong chunking, vocabulary mismatch or missing documents, which an average can't tell you. Each cause has a different fix.