Chunking is a design decision: structure your documents for retrieval

Many RAG failures start with how documents are split. Practical rules for chunking that keeps meaning intact.

A retrieval system can only return what it has indexed, and what it has indexed is whatever your splitter produced. Cut a policy in the middle of a condition and the answer becomes confidently wrong.

Rules that help

Split on structure first — headings, sections, table rows — and on length only as a fallback.
Keep context attached — carry the document title and section path with every chunk so it makes sense in isolation.
Treat tables and lists as units — splitting them destroys the relationships that give them meaning.
Store metadata — source, date, owner and version, so answers can cite and stale material can be filtered.
Re-test after changing chunking — a different splitter is a different system.

If your test set shows right documents retrieved but wrong answers, look at chunk boundaries before changing the model.

Related capability

This is what our AI Data Strategy & ML practice is built around.

See how it's scoped

Keep reading

Data & RAG

RAG, fine-tuning or a long context window? Choosing by the problem

6 min read Data & RAG

Permissions-aware retrieval: making sure the assistant only shows what each person may see

6 min read Data & RAG

Reranking: when a second pass is worth the extra latency

5 min read