The promise and the catch
RAG sounds simple: embed your documents, store the vectors, and at query time pull the most similar chunks into the prompt. The model then answers grounded in your data instead of its training. The demo always works. The catch is that "most similar" and "most relevant" are not the same thing.
Chunking decides your ceiling
Before retrieval there is chunking, and it quietly caps your quality. Split too small and a chunk loses the context that makes it meaningful. Split too large and a single chunk mixes three topics, so its embedding points nowhere useful. There is no universal size — it depends on your documents — but splitting on structure (headings, sections) beats splitting on a fixed character count almost every time.
Pure vector search is not enough
Embeddings capture meaning but miss exact tokens: a product code, an error string, a rare name. Keyword search nails those and misses paraphrases. The robust answer is hybrid — run both and merge the rankings.
const dense = await vectorSearch(query, { topK: 20 });
const sparse = await keywordSearch(query, { topK: 20 });
const merged = reciprocalRankFusion(dense, sparse).slice(0, 6);
Rerank before you trust
The top vector hit is often related but not the answer. A reranker — a model that scores each candidate against the actual question — reorders the shortlist and pushes the real answer to the top. Retrieving 20 and reranking down to 5 beats retrieving 5 directly, almost every time.
Measure retrieval on its own
When a RAG system gives a wrong answer, you cannot tell if the model failed or the retrieval did — unless you test them separately. Build a small set of questions with known correct chunks and measure: did the right chunk make it into the context at all? Most "the AI is wrong" bugs are really "the right chunk never arrived" bugs.
The takeaway
Spend your effort upstream of the model. Good chunking, hybrid search, a reranker, and an honest retrieval metric will outperform any amount of prompt tuning on top of bad context.