A retrieval system is only as good as the chunks it retrieves from. Get chunking wrong and no embedding model, no reranker, and no prompt engineering downstream can fully recover — the system is answering from fragments that cut sentences in half, separate a table from its header, or split a clause from the exception that governs it. Chunking is unglamorous, and it's the highest-leverage decision in the whole pipeline.
Where naive chunking fails
- Fixed character counts — splitting every 500 characters ignores sentence and paragraph boundaries, frequently cutting the answer to a question in half across two chunks.
- Tables and structured data — a table split mid-row, or separated from its header, becomes unreadable to a model that never sees both halves in the same context.
- Cross-references — a clause that says 'subject to section 4.2' is meaningless if section 4.2 lands in a different chunk that never gets retrieved alongside it.
- Document hierarchy loss — flattening a document into equal-sized chunks discards the fact that a heading governs everything beneath it, information a structure-aware split would preserve.
What structure-aware chunking looks like
Splitting on semantic boundaries — headings, paragraphs, list items — rather than raw character counts. Keeping tables intact as single retrievable units with their headers attached. Carrying a breadcrumb of parent headings into each chunk's metadata, so a chunk about 'termination' still knows it's under 'Section 7: Employment Terms.' And overlap between chunks sized to preserve context across a boundary, not just padding.
Before blaming the model for a wrong answer, pull the actual chunks it retrieved and read them as a human would. A surprising fraction of 'the AI got it wrong' turns out to be 'the AI never saw the right text in one piece.'
The business cost of getting this wrong
A support assistant that occasionally answers from a mangled chunk doesn't fail loudly — it fails confidently, with a plausible-sounding wrong answer and no visible error. That's the failure mode that erodes user trust fastest, because nothing in the interface tells anyone it happened.