Chunking, where intuition loses
One policy document, three questions about it, and every control that a team argues over: chunk size, overlap, the splitting strategy, and how many chunks the retriever returns.
Overlap is how much text repeats at the end of one chunk and the start of the next. With no overlap, an answer that happens to sit across a boundary gets cut in half by the cutting. With generous overlap, nothing is cut, and you store and search the same sentences several times over.
The chunks are real. Move the size and you are moving an actual boundary, and you can read which passages landed together.
Write down which strategy you expect to win before you touch any control. The result is worth more to you if you were wrong about it.
One warning about how to read it. The three questions are deliberately different shapes — one answer sits in a single passage, one spans two neighbours, and one is split across opposite ends of the document. A setting that wins on the first question can lose badly on the second, so read the average across all three rather than the question in front of you.
Semantic chunking is better — it splits on meaning instead of arbitrary length.
It sounds right, and the measurement says otherwise.
Chroma, a company that builds retrieval infrastructure, evaluated chunking strategies in July 2024. With its tested setup, 200-token maximum chunks and zero overlap performed well on IoU, the fraction of relevant or retrieved text that was both relevant and retrieved. The plain splitter, which cuts on paragraph and sentence boundaries, scored 6.9 against the best semantic chunker's 8.0: a gap of 1.1 points on that measure. A separate evaluation across three task types found no consistent gain at all.
A semantic chunker does split on meaning, as promised, and on Chroma's test it led on that IoU measure. Whether the gain matters depends on your product. Other measures can favour different settings: overlap helped recall in one tested setup. The other study found no consistent gain. Some semantic splitters compare numerical representations of text; others ask a language model where to split. Their costs differ. Chroma did not measure chunking runtime, so its quality table alone cannot settle whether the extra work is worth paying for.
Graph-based retrieval is the upgrade — once plain retrieval works, a graph is the next step up.
The same pattern holds one tier up. Graph-based retrieval reads the whole corpus in advance. It pulls out the people, products and events the corpus mentions, along with the links between them. It then searches that map of connections instead of searching the text. It is widely presented as the serious upgrade.
The method named GraphRAG scored 0.84 points higher than BM-25 keyword search on GraphRAG-Bench in 2025. Building its map took roughly 80 million tokens of indexing, which is the one-off work of reading the whole corpus and recording the connections before any question is asked. The paper used seven million words from twenty textbooks. Retrieving information then took about 45 seconds per question, before answer generation. The benchmark also tested RAPTOR, whose retrieval took about 0.02 seconds. That figure belongs to that method. It is not what keyword search takes, and it is easy to quote as though it were.
Is 0.84 points a lot? Those points are on that benchmark's own scoring scale, and this course has not verified what the top of that scale is. So the 0.84 cannot be called large or small on its own. It can still be compared with another gain from the same benchmark. Two differences on one scale compare fairly, whatever the scale's maximum turns out to be.
On the same benchmark, graph-based retrieval scored 1.82 points above using no retrieval at all. Subtract: 1.82 minus 0.84 is 0.98. So the step from no retrieval to plain keyword search was worth 0.98 points, and the step from keyword search to a graph was worth 0.84. The first step moved the score more than the second, and it needed no map to be built first. That comparison takes one thing on trust, which is that a bigger number on this scale means a bigger improvement. The benchmark gives no cost for keyword search, so the two steps cannot be compared on price.
The transferable habit is not "chunking is boring." It is: for retrieval, demand the measurement before you buy the sophistication. The elaborate option is not more likely to be better because it is more elaborate. Twice in this session the elaborate option was measured, on two different scales. Neither comparison alone establishes whether its lead is worth buying for your product.