Most teams spend real time tuning prompts and almost none picking a chunker. They reach for the default text splitter, ship it, and then spend the next quarter chasing what looks like hallucination but is actually retrieval failure in disguise, the model answering confidently from a fragment that was never coherent to begin with. Chunking choice can swing recall by up to 9 percentage points on the same corpus using the same embedding model, which makes it one of the highest-leverage, least-examined decisions in a RAG pipeline.
This isn't a call to over-engineer chunking, though. The honest finding across current research is that chunk size and method usually aren't the binding constraint on RAG quality, stale, ungoverned, or genuinely thin source content breaks more production systems than suboptimal boundary placement does. Get chunking right as a foundation, but don't mistake it for the whole problem.
I default to recursive character splitting at 400 to 512 tokens as the starting point for nearly everything, and upgrade a specific piece of the pipeline only once a concrete failure shows up in testing, not preemptively.
Why chunking breaks quietly
A chunk that reads "revenue grew 3 percent last quarter" is a fine sentence and a useless retrieval unit. The embedding model faithfully encodes exactly what's there, with no signal for which company or which quarter it refers to. When someone later asks "what was Acme's Q2 revenue growth," the retriever has no way to connect that query to a chunk that never mentioned Acme or Q2 in the first place. This is the core chunking failure mode, and it doesn't throw an error or a warning. It just silently returns the wrong chunk, or no relevant chunk at all, and the LLM either declines to answer or, worse, answers fluently from something adjacent and wrong.
Fixed-size splitting: the method almost nobody should ship
Pick a number, 500 characters or 256 tokens, and cut at regular intervals with no sentence detection, no structural awareness. It's the simplest method to implement and close to the worst-performing one in every current benchmark, since it routinely cuts mid-sentence or mid-idea. The only real justification for using it is extreme simplicity requirements on content where structure genuinely doesn't matter, which describes almost nothing in a real business knowledge base.
Recursive character splitting: the right default
This method tries paragraph breaks first, then sentence boundaries, then word boundaries, only falling back to a harder cut when nothing more natural is available. In current benchmarks it posts the strongest end-to-end accuracy among general-purpose methods, roughly 69 percent in one widely cited comparison, while staying fast and cheap enough to run against large corpora without a heavy indexing bill. Industry practice has converged on 400 to 512 tokens per chunk as the sweet spot: small enough that the embedding stays focused on one idea, large enough that the LLM has enough surrounding context to generate a useful answer.
One assumption worth revisiting: the long-standing advice to add 10 to 20 percent overlap between chunks isn't as universal as it used to be. A January 2026 analysis using sparse (SPLADE) retrieval found overlap provided no measurable benefit and only added indexing cost. Overlap still helps in many dense-retrieval setups, but it's a parameter worth testing against your specific retrieval method rather than applying automatically.
Semantic chunking: precision at a real cost
Instead of splitting on character or token counts, semantic chunking embeds sentences individually and groups them by similarity, cutting a new chunk where the topic genuinely shifts. This can improve recall by up to 9 percent over simpler methods on the same corpus, a real, measurable gain. The cost is real too: every sentence needs its own embedding pass before chunks can even be formed, which meaningfully raises indexing time and cost compared to a single splitting pass. Reach for this specifically when retrieval precision, not generation quality, is the demonstrated bottleneck in testing.
Hierarchical chunking: the production default for balancing both sides
This is now the most widely adopted production pattern for good reason: it resolves the fundamental tension between precision and context directly. Small child chunks get embedded and retrieved for precision, matching a specific query closely, while the system returns a larger parent chunk, or a window of surrounding sentences, to the LLM for actual generation. This gives the model enough context to avoid the "isolated fragment" failure mode without diluting the embedding signal used for matching. It consistently ranks at the top of retrieval quality benchmarks, at the cost of maintaining two linked index layers instead of one.
Late chunking: solving cross-references differently
Standard chunking splits text first, then embeds each piece in isolation, which means a chunk near the end of a document has no awareness of something defined at the beginning. Late chunking inverts the order: the full document (or a long section) passes through the embedding model first, so every token attends to every other token via self-attention, and only after that does splitting happen, at the token-embedding level rather than the raw-text level. This preserves document-level coherence in a way standard chunking structurally cannot, and it's specifically worth reaching for when cross-references, a term defined early in a document and used later without redefinition, are causing retrieval misses. It requires an embedding model that supports long context, which isn't universal across providers.
Contextual Retrieval: fixing orphaned fragments cheaply
Anthropic's own published research addresses the "revenue grew 3 percent" problem directly: before embedding each chunk, generate a short piece of context, which document it's from, what section, what it's about, and prepend it to the chunk before that chunk gets embedded and indexed. The chunk that was previously an orphaned sentence now carries its own context wherever it goes. This adds a generation cost per chunk at indexing time, a real but one-time cost, and it's one of the more directly measured fixes for the specific failure mode most chunking discussions are actually trying to solve.
When to skip chunking entirely
Anthropic's own guidance is worth taking seriously here: for a knowledge base under roughly 200,000 tokens, about 500 pages, the simplest and often best approach is to skip RAG's retrieval step entirely and inject the full document into the prompt directly, using prompt caching to keep the repeated cost down. Chunking exists to solve a context-window and cost problem at scale. Below a certain corpus size, that problem doesn't exist yet, and adding a full RAG pipeline is solving a problem you don't have.
Let's create something together
Get in touch with us today.
The bottom line
There's no universal best chunking strategy, only the right method for the specific failure mode showing up in your retrieval testing. Recursive character splitting at 400 to 512 tokens is the right default for nearly everyone starting out. Move to semantic chunking, hierarchical chunking, late chunking, or Contextual Retrieval only once a specific, measured problem points there, and remember that for a genuinely small knowledge base, skipping chunking and RAG retrieval entirely is sometimes the most correct answer of all.
If your RAG system's answers feel inconsistent or context-starved, Flowagenz can benchmark your actual chunking and retrieval setup against your real corpus before recommending a fix, not guess at it. Happy to walk through your specific pipeline on a short call.
