Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 2 · Lesson 6 · 8 min read

What Is RAG and When Not to Use It

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Routing Four Questions to Four Different Mechanisms
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain what retrieval-augmented generation actually solves - knowledge cutoff, private data, attribution, and cost of keeping knowledge current - be precise about what it does not solve, and choose deliberately between RAG, fine-tuning, long-context prompting, and plain tool calls for a given problem.

Concept Explanation

A language model's parametric knowledge is whatever got compressed into its weights during training, which creates four distinct problems that RAG addresses. The model does not know anything after its training cutoff. It does not know your private data at all, because your wiki was not in the training corpus. It cannot tell you where an answer came from, because the answer is a blend of statistical patterns rather than a retrieved document. And updating that knowledge requires retraining, which is orders of magnitude more expensive and slower than editing a document. Retrieval-augmented generation, named in a 2020 paper from Facebook AI Research, addresses all four with the same move: retrieve relevant passages from an external store at query time and place them in the prompt, so the model reasons over supplied evidence rather than recalling from weights.

The reduction in hallucination that RAG provides is real but frequently overstated, and being precise about the mechanism matters. Grounding changes the task from open-ended recall, where the model must produce something and will produce something whether or not it knows, to reading comprehension over a provided passage, which models are substantially better at. But the model is still free to ignore the context, to blend retrieved facts with parametric ones, to synthesize across two passages that should not be combined, or to answer confidently when the retrieved passages do not actually contain the answer. RAG shifts hallucination from the generation stage to the retrieval stage, which is a genuine improvement because retrieval failures are measurable and debuggable in a way that weight-recall failures are not, but it does not eliminate the failure.

It is worth naming what RAG does not fix at all. It does not fix reasoning: if the question requires multi-step inference, retrieving the right documents does not confer the ability to reason over them. It does not fix aggregation questions such as "how many of our customers churned last quarter", because top-k semantic retrieval returns a handful of passages, not a complete set, and an answer computed from an incomplete sample is wrong in a way that looks right. It does not fix stale or contradictory source data, and in fact it faithfully amplifies it, which is why a corpus containing three versions of a policy will produce confidently contradictory answers. And it does not fix tone, format, or domain-specific output style, which are the things fine-tuning is actually good at.

That last point sets up the real decision. The useful framing is that RAG changes what the model knows, while fine-tuning changes how the model behaves. If the gap is knowledge - facts, documents, policies, anything that changes over time or is private - retrieval is the answer, because it is cheap to update, auditable, and attributable. If the gap is behavior - consistently emitting a particular JSON dialect, adopting a domain register, following a company-specific classification taxonomy, or compressing a long prompt into learned behavior to cut per-request cost - fine-tuning is the answer. The two are complementary rather than competing, and production systems that use both typically fine-tune for format and behavior while retrieving for facts.

Two alternatives deserve honest consideration before building a retrieval pipeline. Long-context prompting, simply putting the whole corpus in the window, is genuinely correct for small, stable corpora: if your entire knowledge base is 40,000 tokens and rarely changes, prompt caching makes this cheap, latency is acceptable, and the engineering cost of a retrieval stack is not justified. It stops working as the corpus grows, as per-request cost scales with corpus size rather than with question complexity, and as position effects start hiding the relevant passage. The second alternative is a plain tool call: if the answer lives in a database with a schema, "what is order 8841's status" should be a SQL query executed through a tool, not a semantic search over serialized order records. A surprising number of RAG systems are really badly implemented database queries, and recognizing that saves an enormous amount of work.

The Gap IsRight ToolWhyUpdate Cost
Private, changing, or post-cutoff factsRAGKnowledge is external, attributable, and editable per documentRe-index one document
Consistent format, tone, or taxonomyFine-tuningBehavior is learned into weights rather than re-described every callRetrain on a new dataset
A small, stable corpusLong context plus prompt cachingNo retrieval infrastructure needed, no retrieval failures possibleEdit the prompt
Structured, queryable recordsTool call to the real systemExact, complete, and verifiable rather than approximate top-kNone, it is live
Complete aggregation over a corpusAnalytics query, not retrievalTop-k returns a sample, and an answer from a sample is silently wrongNone, it is live

Worked Example: Routing Four Questions to Four Different Mechanisms

The most valuable design decision in this space is usually made before any retrieval code is written: deciding which questions should reach retrieval at all. A router that classifies intent and dispatches accordingly outperforms a system that forces every question through the same vector search.

Python
def route(question, user):
    intent = classify(question)   # small, cheap, fine-tuned or few-shot classifier

    if intent == 'account_lookup':
        # "What is the status of order 8841?"
        # Structured, exact, live. Retrieval would be strictly worse.
        return tool_call('orders.get', order_id=extract_id(question), as_user=user)

    if intent == 'aggregate_analytics':
        # "How many enterprise accounts churned last quarter?"
        # Top-k retrieval returns a SAMPLE. Counting from a sample is wrong.
        return text_to_sql(question, schema=ANALYTICS_SCHEMA, as_user=user)

    if intent == 'policy_or_docs':
        # "What is our refund policy for annual plans?"
        # Private, changing, needs citation -> this is the RAG case.
        chunks = hybrid_search(question, top_k=30, filters={'tenant': user.tenant})
        chunks = rerank(question, chunks, top_k=6)
        return generate(prompt='grounded_answer', version=3, chunks=chunks, question=question)

    # "Summarize this in our house style" -> behavior, not knowledge.
    # Style lives in a fine-tuned model, not in retrieved documents.
    return generate(model='house-style-ft-v2', question=question)
A decision path separating the four mechanisms by the kind of gap they close: retrieval for private and changing facts, fine-tuning for behavior and format, long context for small stable corpora, and direct tool or SQL calls for structured records and aggregation.
A decision path separating the four mechanisms by the kind of gap they close: retrieval for private and changing facts, fine-tuning for behavior and format, long context for small stable corpora, and direct tool or SQL calls for structured records and aggregation.

Real-World Examples

  • The technique was named in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", published by Facebook AI Research researchers in 2020, which paired a dense passage retriever with a sequence-to-sequence generator and showed that a retrieval component let a comparatively small model outperform larger purely parametric ones on knowledge-intensive tasks, establishing the architectural argument that knowledge should live outside the weights.
  • Morgan Stanley deployed an internal assistant built with OpenAI in 2023 that lets financial advisors query the firm's research library, reported at roughly 100,000 internal documents, and the design requirement that shaped it was attribution: advisors need to see which internal research document supports an answer, which is a capability retrieval provides directly and parametric recall cannot provide at all.
Key TakeawaysRAG changes what the model knows; fine-tuning changes how the model behaves. Most disappointing RAG projects are cases where the actual gap was behavior, reasoning, or a structured query, and no amount of retrieval tuning closes those. Decide which of the four gaps you have before building a pipeline, and be willing to conclude that a SQL query or a 40,000-token cached prompt is the correct architecture.

Review Questions & Answers

It reduces hallucination by changing the task from open-ended recall, where a model must produce an answer whether or not the knowledge is in its weights, into reading comprehension over supplied passages, which models perform substantially better at. It relocates rather than eliminates failure because the model can still ignore the context, blend retrieved facts with parametric ones, improperly synthesize across passages, or answer confidently when the retrieved passages do not contain the answer. The relocation is still a real win, because a retrieval failure is measurable and debuggable - you can inspect exactly what was retrieved - whereas a weight-recall failure leaves no artifact to examine.

Retrieval returns the top-k most semantically similar passages, which is a sample, not a complete set, and completeness is exactly what an aggregation requires. A count computed from whichever twenty passages happened to score highest is arbitrary, and worse, it will be delivered with the same fluent confidence as a correct answer, so the error is invisible. Aggregation belongs in an analytics query against the real data, dispatched through a tool call, where the result is exact and the query itself is auditable.

When the corpus is small enough to fit comfortably below the effective context ceiling, stable enough that the same prompt prefix is reused across requests so prompt caching makes the token cost cheap, and when the engineering and operational cost of an embedding pipeline, a vector index, chunking tuning, and retrieval evaluation is not justified by the corpus size. It also removes an entire class of failure, since a passage that is always in the context can never fail to be retrieved. The approach breaks down as the corpus grows, because per-request cost then scales with corpus size rather than with question difficulty, and because position sensitivity begins to hide relevant passages in long contexts.

They address orthogonal gaps. Retrieval supplies facts that are private, recent, or changing, and it does so in a way that is cheap to update and attributable to a source document. Fine-tuning encodes behavior: a consistent output format, a domain register, a company-specific taxonomy, or a long instruction set compressed into weights so it does not have to be paid for in tokens on every request. A production system frequently wants both, using a fine-tuned model for reliable structured output and house style while retrieving the actual facts it reasons over, because neither technique can substitute for the other.