BLEU, ROUGE, Token F1 and Embedding Similarity Explained
On this page
Grading against a reference answer
Some DocsBot examples come with a short correct answer: "1 hour", "Settings > Data > Export", "Pro and Enterprise". Comparing the output with that reference seems like the obvious grader. It works for some tasks and misleads on others, and the difference is worth understanding before you trust a number.
Exact match, token F1, BLEU, ROUGE and embedding similarity
| Metric | What it compares | Good fit |
|---|---|---|
| Exact match | Whole string after normalizing case and punctuation | Classification labels, short factual answers, IDs |
| Contains | Reference appears inside the output | Short facts inside longer answers |
| Token F1 | Overlap of words between output and reference | Short extractive answers |
| BLEU, ROUGE | Overlap of word sequences (n-grams) | Translation and summarization research, with care |
| Embedding similarity | Distance between vector representations | Rough semantic closeness, deduplication |
BLEU comes from machine translation and focuses on precision of n-grams. ROUGE comes from summarization and focuses on recall. Both reward wording that matches, not meaning that matches.
Computing token F1 by hand
Token F1 treats the output and the reference as bags of words. Precision is the share of output words that appear in the reference. Recall is the share of reference words that appear in the output. F1 is their harmonic mean, a kind of average that stays low if either number is low, so an answer can't score well by being long (high recall, low precision) or by being one correct word (the reverse):
# f1.py: SQuAD-style token F1 between an answer and a reference
import re, string
from collections import Counter
def normalize(text: str) -> list[str]:
text = text.lower()
text = "".join(ch for ch in text if ch not in string.punctuation)
text = re.sub(r"\b(a|an|the)\b", " ", text)
return text.split()
def token_f1(prediction: str, reference: str) -> float:
pred, ref = normalize(prediction), normalize(reference)
common = Counter(pred) & Counter(ref)
overlap = sum(common.values())
if overlap == 0:
return 0.0
precision, recall = overlap / len(pred), overlap / len(ref)
return 2 * precision * recall / (precision + recall)
reference = "Reset links are valid for 1 hour."
for answer in ["Reset links are valid for 1 hour.",
"Reset links are valid for 24 hours.",
"You have 60 minutes to use the link."]:
print(f"{token_f1(answer, reference):.2f} {answer}")The output:
1.00 Reset links are valid for 1 hour.
0.71 Reset links are valid for 24 hours.
0.00 You have 60 minutes to use the link.Read that again. The wrong answer, 24 hours, scores 0.71. The correct answer phrased differently, 60 minutes, scores zero. That's the core problem with overlap metrics on free text: they measure shared words, and the words that matter most are often few.
Why embedding similarity misses wrong facts
Embedding similarity would rate "60 minutes" close to "1 hour", which helps. But it also rates "valid for 24 hours" very close to "valid for 1 hour", because the sentences are about the same thing. Embeddings capture topic well and specific facts, numbers and negations poorly. "SSO is included in Pro" and "SSO is not included in Pro" sit close together.
Where reference metrics do work
Use them when the answer space is small and the wording is constrained:
- Exact match for labels: routing decisions, intent classes, yes or no.
- Contains for short facts you've normalized, such as a plan name or a setting path.
- F1 for extractive tasks where the answer is a span copied from a document.
- Embedding similarity to find near-duplicate examples in your dataset, not to grade correctness.
For DocsBot's open-ended answers, a reference is still valuable, but as input to an LLM judge: "Does the answer state the same reset-link lifetime as the reference?" That's tomorrow's lesson.
Check yourself
Exact match after normalizing case and whitespace. The output space is small and fixed, so overlap or similarity metrics add nothing, and per-label precision and recall show which intents get confused.
Not on that alone. ROUGE rewards shared wording, not faithfulness or usefulness. Check with a judge or human review for the qualities you care about.
Embeddings mostly capture topic, and both sentences are about the same topic with almost identical words. A single negation barely moves the vector.
For tasks with a short, well-defined correct output, such as classification labels, extracted ids or numbers, where exact or normalized matching is meaningful.
The overlap of words between the output and the reference, balancing precision and recall. It rewards shared words, not correct meaning.