Elasticsearch Relevance Scoring: TF-IDF vs BM25
On this page
Why relevance scoring matters
A search for database performance returned "Why Our Search Was Slow" first and "Network Performance Basics" last (Day 1). The order comes from a scoring formula. Knowing how it works lets you predict rankings and fix them when they're wrong.
TF-IDF ideas: term frequency, rarity and field length
Three intuitions, inherited from classic TF-IDF:
- Term frequency (TF): a document mentioning
databasethree times is probably more about databases than one mentioning it once. - Inverse document frequency (IDF): matching a rare term means more than matching a common one.
performancein 4 of 7 articles is less informative than a term in 1 of 7. - Field length: a match in a short title means more than one buried in a long body.
An everyday version: if a friend's message mentions "concert" five times, it's probably about a concert, but the fifth mention tells you less than the first. If a word appears in almost every message ("the", "hi"), it tells you nothing about any of them. And "concert" in a three-word subject line says more than the same word in a long email.
BM25, the default since Elasticsearch 5, refines TF-IDF in two ways: term frequency saturates (the tenth mention adds little), and length normalization is tunable.
Reading a real BM25 explain output
The _explain API shows exactly how one document scored. Article 3 for database performance, abridged:
1.3563 sum of:
0.7997 weight(body:database), computed as boost * idf * tf from:
2.2000 boost
0.8267 idf = log(1 + (N - n + 0.5) / (n + 0.5))
3 n, documents containing "database"
7 N, total documents with the field
0.4397 tf = freq / (freq + k1 * (1 - b + b * dl / avgdl))
1 freq, occurrences in this document
1.2 k1, term saturation parameter
0.75 b, length normalization parameter
15 dl, length of this field
13.86 avgdl, average field length
0.5566 weight(body:performance)
0.5754 idf (4 of 7 documents contain it)
0.4397 tfThe 2.2 labeled boost isn't one you set: it's k1 + 1 (1.2 + 1), a constant from the BM25 formula. The score is a sum over query terms. database contributes more than performance because it's rarer (IDF 0.83 versus 0.58). This document's body is slightly longer than average (15 versus 13.86 terms), which nudges TF down.
BM25 parameters k1 and b
- k1 (default 1.2): how quickly repeated terms stop helping. Higher means more reward for repetition.
- b (default 0.75): how much field length matters. 0 ignores length, 1 normalizes fully.
You can change them per field with a custom similarity, but rarely need to. Boosting and function scores (Day 10) are the usual tools.
Relevance scoring surprises
- Scores are relative, not absolute. A score of 4.7 means nothing alone, and scores from different queries can't be compared or used as a "confidence" threshold.
- IDF is per shard by default. With few documents spread across several shards, the same document can score differently depending on its shard. Large indices even this out; for tests, use one shard or
search_type=dfs_query_then_fetch. - Deleted documents still count in statistics until segments merge.
- Matching more terms usually wins, which is why articles containing both words outranked the one containing only
performance.
Check yourself
Fewer documents contained it, so its inverse document frequency was higher. Rare terms are better at telling documents apart, so BM25 gives them more weight than common ones.
Term frequency saturates: the first few occurrences of a term add a lot, and extra occurrences add less and less, controlled by k1. Classic TF-IDF keeps rewarding repetition, which keyword-stuffed documents exploit.
Not reliably; scores depend on the query and the index statistics, so they're only meaningful for ranking within one query.
How much field length normalizes the score. With the default 0.75, a match in a short field counts more than the same match in a long one.
Term statistics are computed per shard by default, so small indices with few documents per shard can show inconsistent scores.