Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 3 · Lesson 13 · 4 min read

RAG Faithfulness, Answer Relevance and Correctness

On this page
  1. Evaluating RAG answers: three questions
  2. Measuring faithfulness claim by claim
  3. Answer relevance and correctness
  4. Combining retrieval and generation scores
  5. Check yourself

Evaluating RAG answers: three questions

DocsBot retrieved the right page and still answered badly. "Badly" can mean three different things:

  • Faithfulness (groundedness): is every claim supported by the retrieved context? An unfaithful answer adds or changes facts.
  • Answer relevance: does it address what the user asked? A faithful answer about the wrong thing is still useless.
  • Correctness: does it match the reference answer in your dataset? An answer can be faithful to a stale page and still be wrong.

Libraries such as RAGAS, DeepEval and TruLens ship metrics with these names. The names vary, and so do the definitions, so read how a metric is computed before trusting it. Building your own from the Day 8 judge pattern takes an afternoon and you'll know exactly what it measures.

Measuring faithfulness claim by claim

A single "is this grounded?" verdict works, as Day 8 showed. A finer version splits the answer into individual claims, checks each one against the context, and reports the share that's supported. That tells you *which* sentence was invented, and gives partial credit you can track over time.

Python
# faithfulness.py: split an answer into claims, check each against the retrieved context
import anthropic
from typing import Literal
from pydantic import BaseModel

client = anthropic.Anthropic()

class Claims(BaseModel):
    claims: list[str]

class ClaimCheck(BaseModel):
    reasoning: str
    supported: Literal["yes", "no"]

def ask(system: str, content: str, schema):
    response = client.messages.parse(model="claude-opus-5-5", max_tokens=1000, system=system,
                                     messages=[{"role": "user", "content": content}],
                                     output_format=schema)
    return response.parsed_output

def faithfulness(answer: str, context: str) -> dict:
    claims = ask("Split the answer into short, self-contained factual claims about the product. "
                 "Skip greetings, offers to help and questions.", answer, Claims).claims
    checks = [ask("Decide if the claim is stated in or directly implied by the context.",
                  f"<context>{context}</context>\n<claim>{c}</claim>", ClaimCheck) for c in claims]
    unsupported = [c for c, chk in zip(claims, checks) if chk.supported == "no"]
    score = 1.0 if not claims else 1 - len(unsupported) / len(claims)
    return {"score": score, "unsupported": unsupported, "n_claims": len(claims)}

One call to split plus one per claim adds up, so use it on your dataset runs and samples of production, not every request. Validate both steps against human labels: claim splitting can drop or merge claims, and that changes the score.

Answer relevance and correctness

Relevance is a straightforward pass/fail judge: given the question and the answer, does the answer address what was asked? Some libraries compute it by generating questions from the answer and measuring embedding similarity to the original. That's indirect; a direct judge aligned with human labels is easier to reason about.

Correctness compares against the reference from your dataset. For short facts, a judge prompt like "Does the answer state the same reset-link lifetime as the reference? Ignore wording" works well. It's the only one of the three that catches an answer faithful to an outdated page.

Combining retrieval and generation scores

Combine yesterday's retrieval result with today's answer grades to locate each failure:

Crossing retrieval hit or miss with answer pass or fail tells you which stage to fix.
Crossing retrieval hit or miss with answer pass or fail tells you which stage to fix.
  • Retrieved and correct: working as intended.
  • Retrieved but wrong or unfaithful: a generation problem. Look at the prompt, context order and model.
  • Missed but correct: the model answered from its own knowledge. Lucky today, risky tomorrow, and often unfaithful by definition.
  • Missed and wrong: fix retrieval first; generation never had a chance.

Count the cells across your dataset and you know where the next week of work should go.

Check yourself

The retrieved context itself is wrong or stale, for example an old pricing page. Faithfulness can't catch it; correctness against the reference does, and the fix is in the index.

Claim-level checks show exactly which statement was invented and give partial credit, which makes the score more informative and easier to debug.

No. In those cases the model never saw the right page, so no answer prompt can fix it. Improve retrieval first, then revisit generation for the failures that remain where the right context was retrieved.

Faithfulness checks whether every claim is supported by the retrieved context. Correctness checks whether the answer matches the truth, usually a reference. An answer can be faithful to wrong context.

Whether the answer actually addresses the question asked. A faithful, correct answer to a different question still fails the user.