Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 2 · Lesson 7 · 9 min read

RAG Chunking Strategies Compared

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Small-to-Big Retrieval With Contextual Prefixes
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain why chunking is the highest-leverage and least glamorous decision in a retrieval pipeline, compare fixed-size, recursive, structural, and semantic chunking, and describe the techniques - overlap, contextual prefixes, and decoupled retrieval and generation units - that resolve the core tension between retrieval precision and answer completeness.

Concept Explanation

Chunking is the process of splitting source documents into the units that get embedded, indexed, and eventually retrieved, and it deserves far more attention than it usually receives because it silently caps the quality of everything downstream. No amount of embedding model tuning, index tuning, or prompt engineering can recover information that chunking destroyed. If the sentence containing the answer was split from the sentence containing the subject it refers to, the chunk that survives is either unretrievable or unusable, and the failure will be diagnosed as a retrieval problem or a model problem for weeks before anyone looks at the splitter.

The central tension is precision against completeness. Small chunks produce focused embeddings whose position in the vector space corresponds to one specific idea, which makes them easy to match precisely, but they frequently lack the surrounding context needed to actually answer the question, and they lose the referent of every pronoun and abbreviation. Large chunks carry enough context to answer, but their embedding averages several ideas, which pulls the vector toward the centroid of the document and makes it match everything moderately and nothing strongly. There is no universally correct size. The usable range for prose is commonly a few hundred to a thousand tokens, and the only honest way to pick within it is to measure retrieval quality on a real evaluation set drawn from your own corpus and your own question distribution.

The strategies form a ladder of increasing structural awareness. Fixed-size splitting by character or token count is trivial and terrible, because it cuts mid-sentence and mid-table with equal indifference. Recursive splitting improves on it by trying a hierarchy of separators in order - paragraph breaks, then line breaks, then sentence boundaries, then characters - so it only resorts to a brutal cut when nothing better is available, and it is the sensible default for unstructured prose. Structural or document-aware splitting uses the document's own boundaries: markdown headings, HTML sections, code functions and classes, PDF sections, database record boundaries. This is almost always the best choice when the structure exists, because a human author already decided where the semantic boundaries were, and respecting a heading hierarchy also gives you a natural place to attach metadata. Semantic chunking, which splits where consecutive sentence embeddings diverge past a threshold, is conceptually appealing and genuinely helps on flowing prose with no markup, at the cost of an embedding pass over every sentence at ingest time.

Two techniques do most of the work of resolving the precision-completeness tension. Overlap, carrying the last ten to twenty percent of one chunk into the start of the next, is cheap insurance against an answer being severed exactly at a boundary, at the cost of index size and some duplicate results that the deduplication step should handle. The more powerful technique is decoupling what you retrieve from what you send to the model: embed small, precise units for matching, but expand to a larger surrounding window before generation. Variants include the parent-document pattern, where a matched child chunk resolves to its parent section, and sentence-window retrieval, where a matched sentence expands to the sentences around it. You get the precision of small units and the completeness of large ones, and the cost is bookkeeping in the index rather than a compromise in quality.

The third technique, contextual prefixing, addresses the problem that a chunk extracted from the middle of a document has lost the knowledge of what document it came from and what section it sat in. Prepending a short, chunk-specific context line before embedding - the document title, the section heading, and often a one-sentence description of what this chunk covers relative to the whole document - makes the embedding dramatically more discriminative, because "the limit is 500 requests per minute" becomes "API rate limits, Enterprise tier: the limit is 500 requests per minute". Anthropic published results on this approach under the name Contextual Retrieval in 2024, reporting a substantial reduction in retrieval failure rate from adding generated per-chunk context, with a further reduction when combined with re-ranking. The ingest-time cost is real, since it means an extra model call or a templated assembly per chunk, but it is paid once at indexing rather than on every query.

StrategyHow It SplitsBest ForMain Weakness
Fixed-sizeEvery N tokens or characters, ignoring contentNothing in production; a baseline onlyCuts mid-sentence, mid-table, and mid-code with equal indifference
RecursiveTries paragraph, then line, then sentence, then character separatorsUnstructured prose with no reliable markupStill structure-blind when the document has real sections
StructuralFollows headings, HTML sections, code functions, record boundariesMarkdown, HTML, code, and any documented formatRequires a reliable parser and produces very uneven chunk sizes
SemanticSplits where consecutive sentence embeddings diverge past a thresholdLong flowing prose without markup, such as transcriptsAn embedding pass per sentence at ingest, plus a threshold to tune

Worked Example: Small-to-Big Retrieval With Contextual Prefixes

This pattern combines the three techniques that matter: structural splitting, a contextual prefix added before embedding, and decoupled retrieval and generation units so the matched child resolves to its parent section.

SQL
-- Index stores small, precise children; generation reads large parents.
CREATE TABLE parents (
  id        BIGSERIAL PRIMARY KEY,
  doc_id    BIGINT NOT NULL,
  heading   TEXT   NOT NULL,          -- from the document structure
  body      TEXT   NOT NULL           -- the full section, 1500-3000 tokens
);

CREATE TABLE children (
  id         BIGSERIAL PRIMARY KEY,
  parent_id  BIGINT NOT NULL REFERENCES parents(id),
  text       TEXT   NOT NULL,         -- 200-350 tokens, 15% overlap
  embed_text TEXT   NOT NULL,         -- contextual prefix + text (what we embed)
  embedding  VECTOR(1024) NOT NULL
);

-- embed_text is built at ingest, not at query time:
--   "Billing Guide > Refunds > Annual Plans. This passage states the refund
--    window and the proration rule for annual subscriptions.
--    <chunk text>"
-- The prefix is why "what is the window?" can match at all - the raw chunk
-- never repeats the words "refund" or "annual".

-- Query: match precisely on children, then expand to parents for generation.
WITH hits AS (
  SELECT c.id, c.parent_id, c.embedding <=> $1 AS distance
  FROM children c
  WHERE c.embed_model = $2
  ORDER BY c.embedding <=> $1
  LIMIT 40
)
SELECT DISTINCT ON (p.id) p.id, p.heading, p.body, MIN(h.distance) AS best
FROM hits h JOIN parents p ON p.id = h.parent_id
GROUP BY p.id ORDER BY p.id, best ASC;
The precision versus completeness tradeoff across chunk sizes, resolved by small-to-big retrieval: small contextually-prefixed children are embedded and matched, then expanded to their parent sections before the text reaches the generation model.
The precision versus completeness tradeoff across chunk sizes, resolved by small-to-big retrieval: small contextually-prefixed children are embedded and matched, then expanded to their parent sections before the text reaches the generation model.

Real-World Examples

  • Anthropic published its Contextual Retrieval technique in 2024, showing that generating a short chunk-specific context statement and prepending it to each chunk before embedding substantially reduced the rate at which the correct passage failed to be retrieved, with further improvement when combined with a re-ranking stage. It is a direct demonstration that ingest-time preparation of the chunk text, not query-time tuning, was where the largest available gain sat.
  • Google's AI Overviews drew wide attention in May 2024 when it surfaced an answer recommending non-toxic glue to keep cheese on pizza, which traced back to a years-old satirical Reddit comment being retrieved and synthesized as an authoritative source. Whatever the specific pipeline detail, it is a public demonstration of the principle that generation faithfully reproduces whatever the retrieval and source-selection layers hand it, so corpus quality and chunk provenance are generation-quality decisions.
Common PitfallSplitting by a fixed token count with no regard for document structure is the most common and most damaging default in retrieval pipelines. It severs sentences, tables, and code blocks, strips the heading context that makes a passage findable, and caps the quality of every downstream stage in a way no amount of embedding or prompt tuning can recover. Use the document's own structure, add a contextual prefix before embedding, and measure chunk size against a real evaluation set rather than accepting a library default.

Review Questions & Answers

An embedding is a fixed-size lossy summary of everything in its input, so a chunk covering several distinct ideas produces a vector near the average of those ideas rather than near any one of them. That average sits closer to the centroid of the general topic space, which means it has moderate similarity to a wide range of related queries and strong similarity to almost none. Ranking depends on discriminative separation between candidates, so a vector that is mildly close to everything actively degrades ranking quality even though it contains more information in raw text terms.

It resolves the precision-completeness tension without compromising on either side. Small units are embedded and matched, giving focused, discriminative vectors, and once a match is found the system expands to the surrounding parent section or sentence window before building the prompt, so the model receives enough context to actually answer. The cost is bookkeeping rather than quality: the index must store the parent-child relationship, the query path needs a second lookup and a deduplication step because several matched children often resolve to the same parent, and the prompt assembly must budget for the larger expanded units rather than the small matched ones.

Because the retrieval failure happens in the index, not at the query. A chunk extracted from the middle of a document may never repeat the words that identify its subject, since the document established them in a heading several paragraphs earlier, so its embedding sits in the wrong region of the space and the correct chunk is simply not in the candidate set no matter how the query is phrased. Adding the document title, section heading, and a one-line description into the text that gets embedded moves the vector into the right neighborhood permanently, and the cost is paid once at ingest rather than on every query.

Structural boundaries encode a decision a human author already made about where one idea ends and the next begins, and headings additionally supply ready-made metadata for filtering and for contextual prefixes. Semantic chunking has to infer those boundaries statistically from embedding divergence, which requires an embedding pass over every sentence at ingest, introduces a threshold that needs tuning per corpus, and can still split a section that legitimately shifts topic mid-way. Semantic chunking earns its cost on unstructured continuous text such as meeting transcripts or scanned prose, where there is no authored structure to follow.