Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 2 · Lesson 7 · 4 min read

BLEU, ROUGE, Token F1 and Embedding Similarity Explained

On this page
  1. Grading against a reference answer
  2. Exact match, token F1, BLEU, ROUGE and embedding similarity
  3. Computing token F1 by hand
  4. Why embedding similarity misses wrong facts
  5. Where reference metrics do work
  6. Check yourself

Grading against a reference answer

Some DocsBot examples come with a short correct answer: "1 hour", "Settings > Data > Export", "Pro and Enterprise". Comparing the output with that reference seems like the obvious grader. It works for some tasks and misleads on others, and the difference is worth understanding before you trust a number.

Exact match, token F1, BLEU, ROUGE and embedding similarity

MetricWhat it comparesGood fit
Exact matchWhole string after normalizing case and punctuationClassification labels, short factual answers, IDs
ContainsReference appears inside the outputShort facts inside longer answers
Token F1Overlap of words between output and referenceShort extractive answers
BLEU, ROUGEOverlap of word sequences (n-grams)Translation and summarization research, with care
Embedding similarityDistance between vector representationsRough semantic closeness, deduplication

BLEU comes from machine translation and focuses on precision of n-grams. ROUGE comes from summarization and focuses on recall. Both reward wording that matches, not meaning that matches.

Computing token F1 by hand

Token F1 treats the output and the reference as bags of words. Precision is the share of output words that appear in the reference. Recall is the share of reference words that appear in the output. F1 is their harmonic mean, a kind of average that stays low if either number is low, so an answer can't score well by being long (high recall, low precision) or by being one correct word (the reverse):

Python
# f1.py: SQuAD-style token F1 between an answer and a reference
import re, string
from collections import Counter

def normalize(text: str) -> list[str]:
    text = text.lower()
    text = "".join(ch for ch in text if ch not in string.punctuation)
    text = re.sub(r"\b(a|an|the)\b", " ", text)
    return text.split()

def token_f1(prediction: str, reference: str) -> float:
    pred, ref = normalize(prediction), normalize(reference)
    common = Counter(pred) & Counter(ref)
    overlap = sum(common.values())
    if overlap == 0:
        return 0.0
    precision, recall = overlap / len(pred), overlap / len(ref)
    return 2 * precision * recall / (precision + recall)

reference = "Reset links are valid for 1 hour."
for answer in ["Reset links are valid for 1 hour.",
               "Reset links are valid for 24 hours.",
               "You have 60 minutes to use the link."]:
    print(f"{token_f1(answer, reference):.2f}  {answer}")

The output:

Code
1.00  Reset links are valid for 1 hour.
0.71  Reset links are valid for 24 hours.
0.00  You have 60 minutes to use the link.

Read that again. The wrong answer, 24 hours, scores 0.71. The correct answer phrased differently, 60 minutes, scores zero. That's the core problem with overlap metrics on free text: they measure shared words, and the words that matter most are often few.

Why embedding similarity misses wrong facts

Embedding similarity would rate "60 minutes" close to "1 hour", which helps. But it also rates "valid for 24 hours" very close to "valid for 1 hour", because the sentences are about the same thing. Embeddings capture topic well and specific facts, numbers and negations poorly. "SSO is included in Pro" and "SSO is not included in Pro" sit close together.

Where reference metrics do work

Use them when the answer space is small and the wording is constrained:

  • Exact match for labels: routing decisions, intent classes, yes or no.
  • Contains for short facts you've normalized, such as a plan name or a setting path.
  • F1 for extractive tasks where the answer is a span copied from a document.
  • Embedding similarity to find near-duplicate examples in your dataset, not to grade correctness.

For DocsBot's open-ended answers, a reference is still valuable, but as input to an LLM judge: "Does the answer state the same reset-link lifetime as the reference?" That's tomorrow's lesson.

Check yourself

Exact match after normalizing case and whitespace. The output space is small and fixed, so overlap or similarity metrics add nothing, and per-label precision and recall show which intents get confused.

Not on that alone. ROUGE rewards shared wording, not faithfulness or usefulness. Check with a judge or human review for the qualities you care about.

Embeddings mostly capture topic, and both sentences are about the same topic with almost identical words. A single negation barely moves the vector.

For tasks with a short, well-defined correct output, such as classification labels, extracted ids or numbers, where exact or normalized matching is meaningful.

The overlap of words between the output and the reference, balancing precision and recall. It rewards shared words, not correct meaning.