RAG Pipeline Explained End to End
On this page
Learning Objective
Trace a complete RAG system through both its ingestion path and its query path, identify the specific failure mode attached to each stage, and describe how to instrument the pipeline so that a bad answer can be attributed to a single stage rather than debugged as an undifferentiated whole.
Concept Explanation
A production RAG system is two pipelines that share a data store, and conflating them is a persistent source of confusion. The ingestion pipeline runs asynchronously as documents arrive or change: load the source, parse it into text, split it into chunks, enrich each chunk with context and metadata, embed it, and write it to the index. The query pipeline runs synchronously per request: optionally transform the query, retrieve candidates, re-rank them, assemble the prompt within a budget, generate, and validate the output before returning it. The two have completely different latency requirements, failure semantics, and scaling characteristics, and treating ingestion as a background system with its own monitoring rather than as a setup step is one of the differences between a prototype and a product.
The ingestion path has three failure modes that are easy to overlook because they are silent. Parsing is the first and the most underestimated: extracting text from PDFs, scanned documents, slide decks, and HTML is genuinely hard, and a table that flattens into an unreadable run of numbers or a two-column layout that interleaves into nonsense produces chunks that are indexed, embedded, and permanently unretrievable, with no error anywhere. The second is staleness: without a change-detection and re-indexing path, the index drifts away from the source of truth, and the system confidently answers from a policy that was superseded months ago. The third is deletion, which is both a correctness and a compliance problem, because a document removed from the source system that remains in the vector index is still being served to users, and under data protection regimes that is a live obligation rather than a quality issue.
The query path has its own chain, and the important property is that errors compound rather than average. Query transformation is optional but often valuable: a follow-up question like "and what about the annual plan" is uninterpretable standing alone, so a conversational system must rewrite it against the conversation history into a self-contained query before retrieval, and forgetting this step makes multi-turn RAG mysteriously bad in a way that looks like a retrieval problem. Retrieval can miss, re-ranking can misorder, prompt assembly can truncate the one chunk that mattered, and generation can ignore what it was given. Because each stage feeds the next, a modest failure rate at each of five stages compounds into a substantial end-to-end failure rate, which is why stage-level instrumentation is not optional at any real scale.
That instrumentation is what makes the system debuggable. Every request should carry a trace id, and each stage should record its inputs, outputs, and timing: the rewritten query, the candidate chunk ids and scores from retrieval, the post-rerank order, the exact assembled prompt with its token count, the raw model output, and the result of output validation. With that trace, a user complaint resolves in minutes because you can see whether the right chunk was retrieved, whether it survived re-ranking, whether it made it into the prompt, and whether the model used it. Without it, every complaint is an open-ended investigation, and the team ends up changing the prompt because that is the only lever that feels adjustable.
Finally, a mature pipeline has a defined behavior for the case where retrieval finds nothing good. The default behavior of most implementations is to pass whatever the top-k returned into the prompt regardless of score, which means a question with no answer in the corpus still gets five irrelevant chunks and an answer synthesized from them. A relevance floor - a minimum re-ranker score below which a candidate is discarded - combined with an explicit path for the empty case, where the system returns an honest "I do not have information about that" rather than calling the model at all, eliminates a whole class of confident wrong answers and saves the token spend on requests that were never going to produce a good result.
| Stage | Failure Mode | Detection Signal |
|---|---|---|
| Parse | Tables and multi-column layouts flatten into unusable text | Sample extracted text at ingest; alert on chunks with anomalous character profiles |
| Chunk | Answer severed from its context or heading | recall@k on a labelled eval set drops after a chunking change |
| Embed and index | Mixed embedding model versions in one index | Assert the model id on every write and filter on it at query time |
| Sync and delete | Index serves superseded or deleted documents | Reconciliation job comparing source and index counts and checksums |
| Query rewrite | Follow-up questions retrieved as literal fragments | Log the rewritten query; compare multi-turn versus single-turn eval scores |
| Retrieve | Correct chunk not in the candidate set | recall@RETRIEVE_K measured separately from answer quality |
| Rerank | Correct chunk present but ordered out of the final set | ndcg@RERANK_K on the same eval set |
| Assemble | Budget truncation drops the decisive chunk | Log prompt token count and chunks_dropped per request |
| Generate | Answer ignores or contradicts the supplied context | Groundedness or citation-validity scoring on sampled traffic |
Worked Example: An Instrumented Query Path With a Relevance Floor
The code below is deliberately unremarkable except for two things: every stage emits a trace span with its intermediate values, and there is an explicit early return for the case where nothing retrieved is good enough to answer from.
async def answer(question, history, user, trace):
# 1. Rewrite - without this, "and for annual plans?" retrieves garbage.
standalone = await rewrite_query(question, history)
trace.set('query.original', question)
trace.set('query.rewritten', standalone)
# 2. Retrieve (hybrid, tenant-scoped at the index, never post-filtered).
candidates = await hybrid_search(standalone, filters={'tenant': user.tenant_id}, top_k=50)
trace.set('retrieve.ids', [c.id for c in candidates])
# 3. Rerank, then apply a RELEVANCE FLOOR.
ranked = await rerank(standalone, candidates)
kept = [c for c in ranked if c.score >= RELEVANCE_FLOOR][:5]
trace.set('rerank.scores', [c.score for c in ranked[:10]])
trace.set('rerank.kept', [c.id for c in kept])
# 4. The empty case is a FIRST-CLASS path, not an accident.
if not kept:
trace.set('outcome', 'no_relevant_context')
metrics.increment('rag.no_context') # a rising rate = a corpus gap
return NoAnswer('I do not have information about that in the knowledge base.')
# 5. Assemble within the budget from Lesson 2.
prompt, dropped = build_prompt('grounded_answer', v=3, chunks=kept, question=standalone)
trace.set('prompt.tokens', prompt.token_count)
trace.set('prompt.chunks_dropped', dropped)
# 6. Generate, then validate BEFORE returning.
raw = await llm.complete(prompt, max_tokens=800, timeout=20.0)
result = validate_schema_and_citations(raw, allowed_ids={c.id for c in kept})
trace.set('generate.status', result.status)
return resultReal-World Examples
- LinkedIn engineers published an account in 2024 of building a generative AI product on top of retrieval, describing reaching roughly eighty percent of the intended experience within the first month and then spending an additional four months fighting for the remaining twenty percent, with a large share of that effort going into evaluation, per-stage quality attribution, and handling the long tail of failure cases rather than into the model itself.
- Regulations such as the EU General Data Protection Regulation grant a right to erasure, which for a RAG system means a deletion in the source of truth must propagate to derived copies including chunk tables, vector indexes, and any caches. Systems that treated ingestion as a one-time bulk load with no deletion path have had to retrofit reconciliation jobs, which is considerably harder than designing the delete path in from the start.
Review Questions & Answers
Because it fails silently and permanently. Extracting text from PDFs, scanned documents, slide decks, and complex HTML regularly mangles tables into unreadable number runs and interleaves multi-column layouts into incoherent sentences, and the resulting text is then chunked, embedded, and indexed without any error being raised. The damaged chunks are either unretrievable or retrievable but useless, and because the failure happened three stages upstream it presents as a retrieval or generation problem. Detection requires actively sampling extracted text at ingest and alerting on anomalous output, since nothing in the pipeline will complain on its own.
Retrieval operates on a single query string with no awareness of conversation history, so a follow-up like "and what about the annual plan" is retrieved literally and matches nothing useful, because the subject of the question lives in the previous turn. Rewriting the question against the history into a self-contained query restores the missing subject before retrieval. Without it, the system works well on first questions and degrades sharply on follow-ups, which typically gets diagnosed as a retrieval quality problem and leads to embedding or index tuning that cannot possibly fix it.
Top-k always returns k results regardless of how poor they are, so a question with no answer in the corpus still produces five chunks and an answer synthesized from irrelevant material, delivered with the same confidence as a correct one. A relevance floor discards candidates scoring below a threshold, which means the k retrieved can legitimately be zero, and that turns "we have nothing relevant" into an explicit, detectable state. It enables an honest refusal path, avoids spending tokens on requests that could not have succeeded, and produces a metric whose rate directly indicates gaps between your corpus and what users are actually asking.
The stages are sequential dependencies rather than independent contributors, so a failure at any one stage propagates and cannot be recovered downstream. If parsing, chunking, retrieval, re-ranking, and assembly each succeed ninety-five percent of the time, the probability that all five succeed on a given request is roughly seventy-seven percent, so a system with no individually alarming stage still fails on nearly a quarter of requests. This compounding is why per-stage measurement matters more than a single end-to-end score: the aggregate number tells you something is wrong but not which of five places to look.