RAG Chunking Strategies: How to Prepare Documents for AI Retrieval

Chunking splits documents into passages for retrieval. For most business documents, structure-aware chunking works best: split at headings and sections, keep headings attached to their text, keep tables and lists intact, and add metadata such as source, section path, date and permissions to every chunk. Use recursive splitting as a general fallback, modest overlap where splits cut ideas, and parent-document retrieval when small chunks lack context. There is no universal chunk size; test strategies against real questions and measure retrieval recall.

Chunking is part of the indexing phase in RAG, before embeddings are created. Ranking retrieved chunks is covered in reranking, and document parsing at scale in enterprise RAG architecture.

Retrieval returns chunks, not documents. If a chunk cuts a definition in half, mixes two unrelated topics or loses the heading that gives it meaning, the right answer may never be retrieved, or may be retrieved without the context the model needs. Many RAG quality problems blamed on the model are chunking problems.

Parse the document into its structure first: headings, paragraphs, lists, tables, code blocks. Split at section boundaries, merge very small sections with neighbours, and split very long sections recursively. Prepend the heading path ('HR Policy > Leave > Parental leave') to each chunk's text or metadata so it carries context. FAQs become one chunk per question and answer; contracts often split by clause.

Smaller chunks match questions precisely but may lack context; larger chunks carry context but dilute the embedding and cost more tokens when retrieved. Start around a paragraph to a short section, then test smaller and larger variants. Overlap of a sentence or two helps fixed-size chunking; structure-aware chunking usually needs little. Check your embedding model's input limit and avoid silently truncated chunks.

Parsing, OCR and transcription before chunking are covered in unstructured data processing.

Every chunk should carry metadata: source ID, title, heading path, page, date, document type, owner and access groups. Metadata supports filtering, citations and freshness. Parent-document retrieval indexes small chunks but returns the surrounding section to the model, combining precise matching with enough context. Some systems also index a short summary per document to help with broad questions.

Simple strategies are fast to build and good enough for uniform text. Structure-aware chunking gives the best results on manuals, policies and documentation but depends on parsing quality. Semantic chunking helps with long narrative text but adds cost and makes results less predictable. Whatever you choose, re-chunking means re-embedding, so test before indexing everything.

A chunk that says 'the limit is 30 days' is useless without knowing which policy it belongs to. Besides heading paths, some teams prepend a short, generated description of where the chunk sits in its document before embedding it, an approach Anthropic has described as contextual retrieval. It can improve retrieval for chunks that are ambiguous on their own, at the cost of an extra model call per chunk during indexing. Test whether it helps on your corpus before applying it everywhere.

An illustrative scenario, not a client case: a company indexes its employee handbook with fixed 500-token chunks. Questions about parental leave return chunks that start mid-policy without the heading. Switching to section-based chunks with heading paths and keeping eligibility tables intact makes the correct section the top result for most leave questions in the test set.

Good chunking keeps meaning intact: split by structure, keep context with each chunk, handle tables carefully and test against real questions. Related: RAG guide, embeddings and reranking.