Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 2 · Lesson 10 · 8 min read

RAG Pipeline Explained End to End

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: An Instrumented Query Path With a Relevance Floor
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Trace a complete RAG system through both its ingestion path and its query path, identify the specific failure mode attached to each stage, and describe how to instrument the pipeline so that a bad answer can be attributed to a single stage rather than debugged as an undifferentiated whole.

Concept Explanation

A production RAG system is two pipelines that share a data store, and conflating them is a persistent source of confusion. The ingestion pipeline runs asynchronously as documents arrive or change: load the source, parse it into text, split it into chunks, enrich each chunk with context and metadata, embed it, and write it to the index. The query pipeline runs synchronously per request: optionally transform the query, retrieve candidates, re-rank them, assemble the prompt within a budget, generate, and validate the output before returning it. The two have completely different latency requirements, failure semantics, and scaling characteristics, and treating ingestion as a background system with its own monitoring rather than as a setup step is one of the differences between a prototype and a product.

The ingestion path has three failure modes that are easy to overlook because they are silent. Parsing is the first and the most underestimated: extracting text from PDFs, scanned documents, slide decks, and HTML is genuinely hard, and a table that flattens into an unreadable run of numbers or a two-column layout that interleaves into nonsense produces chunks that are indexed, embedded, and permanently unretrievable, with no error anywhere. The second is staleness: without a change-detection and re-indexing path, the index drifts away from the source of truth, and the system confidently answers from a policy that was superseded months ago. The third is deletion, which is both a correctness and a compliance problem, because a document removed from the source system that remains in the vector index is still being served to users, and under data protection regimes that is a live obligation rather than a quality issue.

The query path has its own chain, and the important property is that errors compound rather than average. Query transformation is optional but often valuable: a follow-up question like "and what about the annual plan" is uninterpretable standing alone, so a conversational system must rewrite it against the conversation history into a self-contained query before retrieval, and forgetting this step makes multi-turn RAG mysteriously bad in a way that looks like a retrieval problem. Retrieval can miss, re-ranking can misorder, prompt assembly can truncate the one chunk that mattered, and generation can ignore what it was given. Because each stage feeds the next, a modest failure rate at each of five stages compounds into a substantial end-to-end failure rate, which is why stage-level instrumentation is not optional at any real scale.

That instrumentation is what makes the system debuggable. Every request should carry a trace id, and each stage should record its inputs, outputs, and timing: the rewritten query, the candidate chunk ids and scores from retrieval, the post-rerank order, the exact assembled prompt with its token count, the raw model output, and the result of output validation. With that trace, a user complaint resolves in minutes because you can see whether the right chunk was retrieved, whether it survived re-ranking, whether it made it into the prompt, and whether the model used it. Without it, every complaint is an open-ended investigation, and the team ends up changing the prompt because that is the only lever that feels adjustable.

Finally, a mature pipeline has a defined behavior for the case where retrieval finds nothing good. The default behavior of most implementations is to pass whatever the top-k returned into the prompt regardless of score, which means a question with no answer in the corpus still gets five irrelevant chunks and an answer synthesized from them. A relevance floor - a minimum re-ranker score below which a candidate is discarded - combined with an explicit path for the empty case, where the system returns an honest "I do not have information about that" rather than calling the model at all, eliminates a whole class of confident wrong answers and saves the token spend on requests that were never going to produce a good result.

StageFailure ModeDetection Signal
ParseTables and multi-column layouts flatten into unusable textSample extracted text at ingest; alert on chunks with anomalous character profiles
ChunkAnswer severed from its context or headingrecall@k on a labelled eval set drops after a chunking change
Embed and indexMixed embedding model versions in one indexAssert the model id on every write and filter on it at query time
Sync and deleteIndex serves superseded or deleted documentsReconciliation job comparing source and index counts and checksums
Query rewriteFollow-up questions retrieved as literal fragmentsLog the rewritten query; compare multi-turn versus single-turn eval scores
RetrieveCorrect chunk not in the candidate setrecall@RETRIEVE_K measured separately from answer quality
RerankCorrect chunk present but ordered out of the final setndcg@RERANK_K on the same eval set
AssembleBudget truncation drops the decisive chunkLog prompt token count and chunks_dropped per request
GenerateAnswer ignores or contradicts the supplied contextGroundedness or citation-validity scoring on sampled traffic

Worked Example: An Instrumented Query Path With a Relevance Floor

The code below is deliberately unremarkable except for two things: every stage emits a trace span with its intermediate values, and there is an explicit early return for the case where nothing retrieved is good enough to answer from.

Python
async def answer(question, history, user, trace):
    # 1. Rewrite - without this, "and for annual plans?" retrieves garbage.
    standalone = await rewrite_query(question, history)
    trace.set('query.original', question)
    trace.set('query.rewritten', standalone)

    # 2. Retrieve (hybrid, tenant-scoped at the index, never post-filtered).
    candidates = await hybrid_search(standalone, filters={'tenant': user.tenant_id}, top_k=50)
    trace.set('retrieve.ids', [c.id for c in candidates])

    # 3. Rerank, then apply a RELEVANCE FLOOR.
    ranked = await rerank(standalone, candidates)
    kept = [c for c in ranked if c.score >= RELEVANCE_FLOOR][:5]
    trace.set('rerank.scores', [c.score for c in ranked[:10]])
    trace.set('rerank.kept', [c.id for c in kept])

    # 4. The empty case is a FIRST-CLASS path, not an accident.
    if not kept:
        trace.set('outcome', 'no_relevant_context')
        metrics.increment('rag.no_context')       # a rising rate = a corpus gap
        return NoAnswer('I do not have information about that in the knowledge base.')

    # 5. Assemble within the budget from Lesson 2.
    prompt, dropped = build_prompt('grounded_answer', v=3, chunks=kept, question=standalone)
    trace.set('prompt.tokens', prompt.token_count)
    trace.set('prompt.chunks_dropped', dropped)

    # 6. Generate, then validate BEFORE returning.
    raw = await llm.complete(prompt, max_tokens=800, timeout=20.0)
    result = validate_schema_and_citations(raw, allowed_ids={c.id for c in kept})
    trace.set('generate.status', result.status)
    return result
The two pipelines of a RAG system side by side: an asynchronous ingestion path from source through parse, chunk, enrich, embed and index, and a synchronous query path through rewrite, retrieve, rerank, assemble, generate and validate, each stage annotated with its characteristic failure mode.
The two pipelines of a RAG system side by side: an asynchronous ingestion path from source through parse, chunk, enrich, embed and index, and a synchronous query path through rewrite, retrieve, rerank, assemble, generate and validate, each stage annotated with its characteristic failure mode.

Real-World Examples

  • LinkedIn engineers published an account in 2024 of building a generative AI product on top of retrieval, describing reaching roughly eighty percent of the intended experience within the first month and then spending an additional four months fighting for the remaining twenty percent, with a large share of that effort going into evaluation, per-stage quality attribution, and handling the long tail of failure cases rather than into the model itself.
  • Regulations such as the EU General Data Protection Regulation grant a right to erasure, which for a RAG system means a deletion in the source of truth must propagate to derived copies including chunk tables, vector indexes, and any caches. Systems that treated ingestion as a one-time bulk load with no deletion path have had to retrofit reconciliation jobs, which is considerably harder than designing the delete path in from the start.
Best PracticeInstrument every stage of the query path with a shared trace id recording the rewritten query, retrieved ids and scores, the post-rerank order, the assembled prompt token count, and the validation result. Errors compound across stages, so without per-stage attribution every user complaint becomes an open-ended investigation and teams default to changing the prompt because it is the only lever that feels adjustable.

Review Questions & Answers

Because it fails silently and permanently. Extracting text from PDFs, scanned documents, slide decks, and complex HTML regularly mangles tables into unreadable number runs and interleaves multi-column layouts into incoherent sentences, and the resulting text is then chunked, embedded, and indexed without any error being raised. The damaged chunks are either unretrievable or retrievable but useless, and because the failure happened three stages upstream it presents as a retrieval or generation problem. Detection requires actively sampling extracted text at ingest and alerting on anomalous output, since nothing in the pipeline will complain on its own.

Retrieval operates on a single query string with no awareness of conversation history, so a follow-up like "and what about the annual plan" is retrieved literally and matches nothing useful, because the subject of the question lives in the previous turn. Rewriting the question against the history into a self-contained query restores the missing subject before retrieval. Without it, the system works well on first questions and degrades sharply on follow-ups, which typically gets diagnosed as a retrieval quality problem and leads to embedding or index tuning that cannot possibly fix it.

Top-k always returns k results regardless of how poor they are, so a question with no answer in the corpus still produces five chunks and an answer synthesized from irrelevant material, delivered with the same confidence as a correct one. A relevance floor discards candidates scoring below a threshold, which means the k retrieved can legitimately be zero, and that turns "we have nothing relevant" into an explicit, detectable state. It enables an honest refusal path, avoids spending tokens on requests that could not have succeeded, and produces a metric whose rate directly indicates gaps between your corpus and what users are actually asking.

The stages are sequential dependencies rather than independent contributors, so a failure at any one stage propagates and cannot be recovered downstream. If parsing, chunking, retrieval, re-ranking, and assembly each succeed ninety-five percent of the time, the probability that all five succeed on a given request is roughly seventy-seven percent, so a system with no individually alarming stage still fails on nearly a quarter of requests. This compounding is why per-stage measurement matters more than a single end-to-end score: the aggregate number tells you something is wrong but not which of five places to look.