Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 2 · Lesson 8 · 8 min read

Dense vs Sparse vs Hybrid Retrieval for RAG

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Hybrid Retrieval With Reciprocal Rank Fusion
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Compare dense vector retrieval and sparse lexical retrieval in terms of what each one can and cannot match, explain how BM25 scores a document, and describe how hybrid retrieval combines the two, including why score fusion methods such as Reciprocal Rank Fusion are preferred over naive score addition.

Concept Explanation

Sparse retrieval is the classical information retrieval approach, and BM25 is its dominant scoring function. The representation is a very high-dimensional vector with one dimension per vocabulary term, almost all of which are zero for any given document, hence sparse. BM25 scores a document against a query by summing over the query terms, weighting each by inverse document frequency so that rare terms count for much more than common ones, by a saturating function of how often the term appears in the document so that the tenth occurrence adds much less than the second, and by a length normalization so that long documents are not favored merely for containing more words. It has been the default scoring function in Lucene and therefore in Elasticsearch and OpenSearch for years, and it remains a genuinely strong baseline that newer methods do not always beat.

Dense retrieval replaces term matching with the embedding geometry from earlier in this course. The query and each document become fixed-size vectors, and retrieval means finding the nearest neighbors by cosine similarity. Its defining advantage is matching across vocabulary: a query about resetting a password matches a document about recovering login credentials with zero shared terms, which BM25 scores at essentially nothing. Its defining weakness is the mirror image. A rare exact token - an error code like ORA-01555, a part number, an unusual surname, an internal project codename - is compressed into a general semantic neighborhood rather than preserved as a distinct signal, so dense retrieval will happily rank a document about a similar error code above the one containing the exact string.

The two methods therefore fail on disjoint sets of queries, which is the entire argument for hybrid retrieval. Run both, combine the result lists, and you recover the exact-match reliability of lexical search and the vocabulary-bridging of embeddings in one candidate set. In practice hybrid retrieval is close to the default recommendation for production systems over heterogeneous corpora, because real query distributions contain both natural language questions and identifier lookups, often in the same session, and choosing one method means accepting known failures on a predictable slice of traffic.

Combining the two lists is less trivial than it looks, because BM25 scores and cosine similarities are not on the same scale and are not even on a stable scale between queries. A BM25 score has no upper bound and its magnitude depends on corpus statistics and query length; a cosine similarity is bounded but its useful range varies by embedding model. Adding them directly, or adding them after min-max normalization within each result list, makes the weighting an accident of the score distribution for that particular query. Reciprocal Rank Fusion sidesteps the problem by discarding the scores entirely and fusing on rank: each document receives a contribution of one divided by a constant plus its rank in each list, and the contributions are summed. Because it uses only ordinal position, it is scale-free, needs no per-query normalization, and is robust when one retriever performs poorly on a given query. It is the fusion method Elasticsearch and Azure AI Search expose for hybrid search, and it is the sensible default.

Two practical notes complete the picture. First, hybrid retrieval doubles your infrastructure surface unless your store handles both, which is a genuine argument for systems that support lexical and vector search in one engine rather than running a separate keyword store and vector database that must be kept consistent. Second, hybrid retrieval is a recall strategy, not a precision strategy. Its purpose is to make sure the right passage is somewhere in a candidate set of thirty to a hundred, not to put it first. Getting the right passage to position one is the job of the re-ranking stage in the next lesson, and the two are designed to work together: cast a wide, cheap, high-recall net, then spend expensive precision only on what the net caught.

Query TypeSparse (BM25)Dense (Embeddings)Why
"how do I reset my password"WeakStrongThe document says "recover login credentials" with no shared terms
"error ORA-01555"StrongWeakAn exact rare token is a decisive lexical signal but blurs in embedding space
"SKU 4471-B warranty"StrongWeakIdentifiers must match exactly; a near-miss SKU is a wrong answer
"why did the deployment stall"WeakStrongIntent is paraphrased across entirely different vocabulary
"Kubernetes ingress TLS termination"StrongStrongDomain jargon is both lexically exact and semantically meaningful

Worked Example: Hybrid Retrieval With Reciprocal Rank Fusion

The implementation is short, and the important detail is that fusion happens on rank rather than on score, so no normalization of incomparable score scales is required.

Python
def reciprocal_rank_fusion(result_lists, k=60):
    """Fuse ranked lists on ORDINAL POSITION only - scores are never compared."""
    scores = defaultdict(float)
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

def hybrid_search(query, tenant_id, top_k=50):
    # Both retrievers run in parallel; both apply the SAME metadata filter,
    # or the two lists describe different corpora and fusion is meaningless.
    bm25_hits, dense_hits = parallel(
        lambda: lexical.search(query, filter={'tenant': tenant_id}, size=top_k),
        lambda: vectors.search(embed(query), filter={'tenant': tenant_id}, size=top_k),
    )
    return reciprocal_rank_fusion([bm25_hits, dense_hits])[:top_k]

# Why not just add the scores?
#   bm25("error ORA-01555")  -> 31.4   (unbounded, corpus-dependent)
#   cosine("error ORA-01555") -> 0.71  (bounded, model-dependent)
# Adding these weights BM25 ~44x higher by accident of scale, and the ratio
# changes with every query. RRF has no such failure because rank 1 is rank 1
# in both lists regardless of what produced it.
A query fanned out to a BM25 lexical retriever and a dense vector retriever in parallel, each producing a ranked list, with Reciprocal Rank Fusion merging them on ordinal position into a single high-recall candidate set for the re-ranking stage.
A query fanned out to a BM25 lexical retriever and a dense vector retriever in parallel, each producing a ranked list, with Reciprocal Rank Fusion merging them on ordinal position into a single high-recall candidate set for the re-ranking stage.

Real-World Examples

  • BM25, derived from the probabilistic relevance framework developed by Stephen Robertson, Karen Sparck Jones and colleagues, became the default similarity scoring function in Apache Lucene as of version 6 and therefore in Elasticsearch from version 5 onward, replacing classic TF-IDF. Decades after its introduction it remains a baseline that newer neural retrieval methods are still measured against, and frequently do not beat outright on exact-term queries.
  • Reciprocal Rank Fusion, introduced in a 2009 paper by Cormack, Clarke and Buettcher, is the fusion method both Elasticsearch and Azure AI Search expose for combining keyword and vector result sets in hybrid search, chosen specifically because it requires no score normalization between retrievers whose scores live on incomparable scales.
Best PracticeDefault to hybrid retrieval for any corpus containing both natural-language content and identifiers, and fuse the result lists on rank using Reciprocal Rank Fusion rather than on raw or normalized scores. Treat the hybrid stage as a recall stage whose job is to get the right passage somewhere into a candidate set of thirty to a hundred, and leave the job of ordering that set to a dedicated re-ranking stage.

Review Questions & Answers

Inverse document frequency weights each query term by how rare it is across the corpus, so a distinctive term contributes far more than a common one. Term frequency saturation makes each additional occurrence of a term in a document contribute less than the previous one, so a document that repeats a word fifty times does not overwhelm one that uses it twice in a genuinely relevant passage. Length normalization discounts long documents so they are not favored simply for containing more words and therefore more chances to match. Together these correct the three ways a naive term-count score would be misled.

The two scores are not merely on different ranges, they have different and unstable distributions. BM25 is unbounded and its magnitude depends on corpus statistics and query length, so its raw values vary enormously between queries, while cosine similarity is bounded but its useful discriminating band depends on the embedding model. Normalizing within each result list makes the weight assigned to each retriever an accident of that particular query's score spread, so the effective blend ratio changes query to query in a way nobody chose. Rank-based fusion avoids the problem entirely because ordinal position carries the same meaning regardless of what produced it.

An embedding compresses its input into a few hundred or thousand dimensions optimized for topical and paraphrase proximity, which means a rare literal token like an error code or part number is represented as a point in a general semantic neighborhood rather than as an exact signal. Two similar-looking codes therefore land close together, and the exact match holds no special advantage. This is a property of the compression, not a deficiency of a particular model, so a better embedding model shifts the problem rather than removing it. Lexical retrieval scores exact token overlap directly and weights rare tokens heavily, which is precisely the signal dense retrieval discards, so running both recovers it.

Its purpose is to ensure the correct passage appears somewhere in a candidate set, by covering two disjoint failure modes with two retrievers, rather than to place that passage at position one. Fusion on rank produces a reasonable but not finely calibrated ordering, and neither retriever ever compares the query and document jointly. Precision is bought separately in the re-ranking stage, where a more expensive model scores each candidate against the query directly. The division of labor is deliberate: cheap and wide first so nothing relevant is lost, expensive and narrow second on the small surviving set.