Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 2 · Lesson 9 · 8 min read

Reranking in RAG Explained

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Sizing and Instrumenting a Two-Stage Pipeline
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain the architectural difference between a bi-encoder and a cross-encoder, why that difference forces a two-stage retrieve-then-rerank design, and how to size and budget a re-ranking stage against its latency and cost.

Concept Explanation

Vector search is fast because of a structural compromise: the query and the document are encoded independently. A bi-encoder embeds every document at ingest time, long before any query exists, and at query time it embeds the query once and compares. Because the document vectors are precomputed and the comparison is a distance calculation over an index, you can search millions of documents in milliseconds. The compromise is that the model never sees the query and the document together, so it cannot represent the relationship between them. It can only ask whether two independently-produced summaries happen to land near each other, which is a weaker question than whether this document answers this query.

A cross-encoder makes the opposite trade. It takes the query and one candidate document as a single concatenated input and runs a full transformer forward pass over the pair, with attention flowing between query tokens and document tokens throughout. That lets it model exactly what the bi-encoder cannot: which specific phrase in the document answers which part of the question, whether the document contradicts the query, whether a superficially similar passage is actually about something else. The resulting relevance scores are consistently and substantially more accurate. The cost is that nothing is precomputable, because the score exists only for a specific pair, so scoring a million documents means a million forward passes, and that is not a system anyone can build.

The two-stage architecture falls directly out of this. Stage one uses the cheap bi-encoder approach, plus BM25 in a hybrid setup, and optimizes purely for recall: cast a wide net, retrieve perhaps fifty to a hundred candidates, and accept that the ordering within that set is mediocre. Stage two runs the expensive cross-encoder over only those candidates and produces a genuinely well-ordered list, of which you keep the top three to ten for the prompt. The economics work because the expensive model runs on a hundred items rather than a million, and the quality works because the cheap stage only had to get the right document into the candidate set, not into first place. This is the same funnel shape that classical search engines and recommendation systems have used for years, and it transfers directly.

Sizing the two stages is the main tuning decision, and the two knobs interact. The first-stage k controls recall: too small and the right document is never in the candidate set, which no amount of re-ranking can repair, and that failure is invisible unless you measure recall at k separately from end-to-end answer quality. Too large and the re-ranker's latency and cost grow linearly with candidates. The second-stage k controls how much context reaches the model, and it interacts with everything from the second lesson of this course: more chunks means more tokens, more cost, and eventually worse answers due to position effects and distraction. A common starting point is retrieving fifty candidates and keeping five, then tuning both numbers against an eval set rather than leaving them at defaults.

There are cheaper points on the quality curve worth knowing. Late-interaction models such as ColBERT precompute per-token document representations and defer only a lightweight interaction to query time, landing between bi-encoders and cross-encoders in both quality and cost, at the price of a substantially larger index. Using a general-purpose LLM as a re-ranker by asking it to score relevance works and needs no specialized model, but it is usually slower and more expensive than a purpose-built cross-encoder of a few hundred million parameters, which is one of the cases where a small dedicated model decisively beats a large general one. And re-ranking is also the natural place to apply business logic that has nothing to do with semantics: boosting recent documents, demoting deprecated ones, enforcing source authority, and diversifying results so five chunks from the same section do not crowd out everything else.

PropertyBi-Encoder (Stage 1)Cross-Encoder (Stage 2)
EncodingQuery and document encoded separatelyQuery and document encoded jointly as one input
PrecomputationAll document vectors built at ingest timeNothing precomputable; every pair needs a forward pass
ScaleMillions of documents in milliseconds via an ANN indexTens to low hundreds of candidates per query
Relevance qualityGood recall, mediocre orderingSubstantially more accurate ordering
Role in the pipelineGet the right document into the candidate setGet the right document to position one

Worked Example: Sizing and Instrumenting a Two-Stage Pipeline

The instrumentation in this example matters as much as the ranking. Measuring stage-one recall separately from final answer quality is what tells you whether a bad answer was a retrieval failure or a generation failure, and without that split the two are indistinguishable from the outside.

Python
RETRIEVE_K = 50   # stage 1: recall. Too small = unrecoverable misses.
RERANK_K   = 5    # stage 2: precision + context budget from Lesson 2.

def retrieve_and_rank(query, tenant_id, trace):
    t0 = now()
    candidates = hybrid_search(query, tenant_id, top_k=RETRIEVE_K)   # ~40-80 ms
    trace.span('retrieve', now() - t0, n=len(candidates))

    t1 = now()
    # One forward pass PER PAIR - cost is linear in RETRIEVE_K, so this is
    # the knob that trades latency for recall headroom.
    scored = cross_encoder.score(query, [c.text for c in candidates])  # ~120-250 ms
    ranked = [c for _, c in sorted(zip(scored, candidates), reverse=True)]
    trace.span('rerank', now() - t1, n=len(candidates))

    # Business rules belong here, not in the embedding.
    ranked = demote(ranked, where=lambda c: c.status == 'deprecated', factor=0.5)
    ranked = boost(ranked, where=lambda c: c.updated_at > days_ago(90), factor=1.2)
    ranked = diversify(ranked, key=lambda c: c.parent_id, max_per_key=2)

    return ranked[:RERANK_K]

# The two metrics that must be tracked SEPARATELY on a labelled eval set:
#   recall@RETRIEVE_K  - was the right chunk in the candidate set at all?
#                        If this is low, re-ranking cannot help you.
#   ndcg@RERANK_K      - given it was there, did we order it to the top?
#                        If this is low, the re-ranker or its input text is wrong.
The retrieval funnel: millions of indexed chunks narrowed to roughly fifty candidates by a cheap bi-encoder and BM25 stage optimized for recall, then reordered by an expensive cross-encoder that reads query and candidate jointly, with only a handful surviving into the prompt.
The retrieval funnel: millions of indexed chunks narrowed to roughly fifty candidates by a cheap bi-encoder and BM25 stage optimized for recall, then reordered by an expensive cross-encoder that reads query and candidate jointly, with only a handful surviving into the prompt.

Real-World Examples

  • The Sentence-BERT paper by Reimers and Gurevych in 2019 made the tradeoff explicit and quantitative: it noted that finding the most similar pair in a collection of ten thousand sentences with a BERT cross-encoder required roughly fifty million inference computations and around sixty-five hours, while a bi-encoder reduced the equivalent search to seconds, which is precisely why cross-encoders are confined to a re-ranking stage over a small candidate set.
  • Anthropic reported in its 2024 Contextual Retrieval work that adding a re-ranking stage on top of contextual embeddings and contextual BM25 produced a further substantial reduction in the rate at which the correct passage failed to reach the top results, which is a concrete public data point that re-ranking is additive with retrieval-side improvements rather than a substitute for them.
Common PitfallMeasure recall at the first-stage k separately from final answer quality. If the correct chunk is not in the candidate set, no re-ranker, prompt change, or larger model can recover it, yet from the outside that failure looks identical to a generation failure and teams routinely spend weeks tuning the wrong stage. Track recall@k and ranking quality as two distinct numbers on a labelled evaluation set.

Review Questions & Answers

A cross-encoder produces a score only for a specific query-document pair, so nothing can be precomputed before the query arrives. Scoring a corpus of a million chunks would require a million transformer forward passes per query, which is computationally impossible at interactive latency and cost. A bi-encoder precomputes all document vectors at ingest time and reduces query-time work to one embedding plus an approximate nearest neighbor lookup. The two-stage design exists exactly to get the bi-encoder's scalability and the cross-encoder's accuracy by applying each where it is affordable.

The relationship between the query and the document. Because both are fed through the model as a single input, attention flows between query tokens and document tokens, letting the model identify which phrase answers which part of the question, notice that a passage contradicts rather than supports the query, and distinguish a superficially similar passage that is actually about a different subject. A bi-encoder only ever compares two independently produced summaries, so it can assess topical proximity but cannot assess whether this document answers this question.

By measuring recall at the first-stage k on a labelled evaluation set where the correct chunk for each question is known. If the correct chunk frequently does not appear among the retrieved candidates at all, the failure is in retrieval, and improving the prompt, the re-ranker, or the generation model cannot help. If the correct chunk is reliably in the candidate set but is not ranked into the final few, the re-ranker or the text being scored is the problem. If the correct chunk reaches the prompt and the answer is still wrong, the failure is in generation. Without splitting those metrics, all three look identical from the outside.

Because it is the first point in the pipeline where a small, complete, ordered candidate set exists and relevance has already been assessed, so non-semantic adjustments can be applied without distorting the semantic signal itself. Encoding recency or authority into the embedding is not possible in any controlled way, and applying it before retrieval would filter out documents you may still want. At the re-ranking stage you can multiply scores by a recency factor, demote documents flagged as deprecated, and cap how many chunks may come from the same parent section so a single long document does not consume the entire context budget.