A language model knows what was in its training data. It does not know your company's refund policy, last night's deployment, or the contents of the document a user uploaded ten seconds ago.
Ask anyway and you get one of two bad outcomes: an admission of ignorance, or — far worse — a confident, fluent, entirely invented answer.
Retrieval-augmented generation is the standard fix. Before answering, find relevant material and put it in the prompt. The model reasons over facts you supplied instead of recalling facts it half-remembers.
The pipeline
Two phases: one offline, one per request.
Diagram, as text
- Documents leads to Chunk
- Chunk leads to Embed
- Embed leads to Vector store
- Question leads to Retrieve
- Vector store leads to Retrieve
- Retrieve leads to Rerank
- Rerank leads to Prompt with context
- Prompt with context leads to Model
- Model leads to Answer + citations
The framing that matters: RAG is a search problem with a model attached. Teams spend weeks tuning prompts when the actual failure is that retrieval returned the wrong passages. The model cannot reason its way to facts you did not give it.
Say this in an interview and mean it: retrieval quality is the ceiling on answer quality.
Chunking, where most quality is won or lost
You cannot embed a 200-page manual as one vector — it would represent everything and match nothing. Documents must be split.
Fixed-size chunks are the naive approach: split every 500 tokens. Simple, and it cuts sentences in half and separates a heading from the text beneath it.
Structure-aware chunking splits on real boundaries — headings, sections, paragraphs. Considerably better, because documents already carry meaningful divisions authors created.
Overlap repeats a hundred or so tokens between neighbours, so a fact spanning a boundary survives in at least one chunk.
The size trade-off is genuine:
| Chunk size | Retrieval precision | Context supplied |
|---|---|---|
| Small (~200 tokens) | Sharp — matches the specific passage | May lack surrounding meaning |
| Large (~2,000 tokens) | Blurry — one vector covers many ideas | Plenty of context, more noise and cost |
A useful pattern resolves this: embed small, retrieve large. Index precise little chunks for matching, then supply the enclosing section to the model. You get accurate retrieval and coherent context.
Hybrid retrieval
Vector search finds paraphrases. It is unreliable on exact strings — error codes, product SKUs, surnames, version numbers — because those carry little semantic weight.
Keyword search is the opposite: exact on identifiers, useless on paraphrase.
Run both and merge. Hybrid retrieval consistently outperforms either alone, and the merge is usually reciprocal rank fusion — combining by position in each result list rather than by scores, which are not comparable across systems.
This is worth internalising because it contradicts the instinct that vectors superseded keyword search. They did not. The inverted index from the search lesson remains half of a good retrieval system.
Reranking
Retrieval optimises for speed across millions of documents, which means it is approximate. A reranker takes the top 50 candidates and scores each one properly against the query, then keeps the best 5.
It is expensive per document and only runs on a handful, so the cost is bounded. The quality improvement is typically the largest single gain available after fixing chunking — and it is frequently skipped.
The structure is a funnel, and each stage is cheaper per item than the next is accurate:
millions of chunks -> ANN search -> 50 candidates -> rerank -> 5 passages -> modelGeneration
Two decisions in the final prompt matter more than the wording.
Instruct the model to answer only from the supplied context, and to say when the context does not contain the answer. Without this it falls back on training data and you lose the guarantee RAG exists to provide.
Require citations. Ask for the chunk id or source with each claim. This gives users a way to verify, and gives you a way to detect the model asserting things absent from the context.
Ordering has a real effect too: models attend more reliably to material at the beginning and end of a long prompt than to the middle. Place the strongest passages at the edges rather than burying them.
When retrieval fails
Diagnose in order, because fixing the wrong stage is the most common wasted effort.
Retrieved passages are irrelevant. A retrieval problem: revisit chunking, add hybrid search, add reranking. Prompt changes will not help.
The right passage exists but ranked too low. Add reranking, or increase how many candidates you retrieve before reranking.
The right passage was retrieved and the answer is still wrong. Now it is a generation problem: prompt instructions, or a model not strong enough for the reasoning required.
The question needs several documents combined. Basic RAG retrieves once and struggles here. Multi-step approaches — decomposing the question, or letting an agent retrieve iteratively — are covered in the orchestration lesson.
Measure the stages separately. Retrieval quality (did we fetch the right passages?) and generation quality (given them, was the answer right?) are different numbers with different fixes, and a single end-to-end score cannot tell you which is broken.
Keeping the index current
The same problem as any derived index, and the same answers from the search lesson: avoid dual writes, drive updates from a change stream, and accept eventual consistency.
Two additional concerns specific to RAG:
Deletions must propagate. A document removed from your system but left in the index will be retrieved and quoted. For anything with access control or legal retention rules, this is a serious failure, not a stale-cache annoyance.
Permissions must be enforced at retrieval time. Filter by the requesting user's access before passages reach the model. Filtering afterwards is too late — the content is already in the prompt, and a model asked to ignore something it has been shown is not a security control.
Check your understanding
Common mistakes
Tuning prompts to fix retrieval problems. The most common wasted effort in the field.
Fixed-size chunking on structured documents. Splitting mid-sentence and orphaning headings damages retrieval before anything else runs.
Vector search only. Exact identifiers need keyword matching.
Skipping reranking. Usually the best quality-per-effort improvement available.
No citations. You lose both user trust and your own ability to detect ungrounded answers.
Filtering permissions after retrieval. By then the content is in the prompt.
No evaluation set. Without one you are judging changes by whether a handful of hand-picked questions feel better.
Practice
A RAG system over internal documentation answers well for some questions and confidently wrongly for others. Design the investigation that separates retrieval failures from generation failures.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Check whether the answer was even present in what was retrieved before blaming the model.
Show a strong answer
Separate the two stages, because the fixes have nothing in common.
Step one — was the answer retrievable at all? For each failing question, search the corpus manually. Three outcomes:
- The information does not exist in the corpus. Neither stage is at fault; the system should say it does not know. This is a coverage problem, and the fix is content, not engineering. It is also the most common cause and the one teams skip past.
- It exists but was not retrieved. A retrieval failure.
- It was retrieved and the answer is still wrong. A generation failure.
Step two — for retrieval failures, look at the chunk. Was the relevant passage split across a boundary? Was it in a table that chunked badly? Did the question use vocabulary that does not appear in the text — a synonym, an abbreviation, a product's old name? Those point to chunking, hybrid retrieval and query rewriting respectively.
Step three — for generation failures, check whether the correct chunk was in the context but ranked low. Models attend unevenly across long contexts, so a correct chunk at position 15 of 20 may be effectively ignored. That is a reranking problem, not a model problem.
What to build so this is not manual next time:
- Log the retrieved chunk IDs with every answer. Without this, none of the above is possible after the fact, and it is the single highest-value piece of instrumentation.
- Keep an evaluation set of question–answer pairs with the expected source document, so retrieval can be scored independently as "did the right document appear in the top k".
The distinction that matters: retrieval quality is measurable without a model in the loop, and should be measured that way. Debugging both stages at once is how teams spend weeks tuning prompts to fix a chunking bug.
Your corpus updates constantly — documents are edited, added and deleted every hour. Design the indexing pipeline, and say what happens when a document is deleted.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Deletion is the case that gets forgotten, and it is the one with legal consequences.
Show a strong answer
Change-driven, not scheduled. A nightly full reindex means the system answers from stale content all day and re-embeds unchanged documents at cost. Emit an event on every create, update and delete.
The pipeline:
- Change event carries the document ID and the operation, published from the same transaction that wrote the change so it cannot be lost.
- Re-chunk the whole document on update, not just the edited portion — chunk boundaries shift when text changes, so a partial update leaves inconsistent chunks.
- Embed only chunks whose text actually changed, keyed by a content hash. Most edits touch one paragraph, and this avoids re-embedding an entire document for a typo.
- Upsert into the index, keyed by chunk ID derived from document ID plus position, so a re-run is idempotent.
- Delete chunks that no longer exist — a document that shrank from 40 chunks to 30 leaves 10 orphans that will keep being retrieved and cited.
Deletion, which is the case that matters most:
- Delete from the index synchronously, or as close to it as possible. A deleted document that keeps being retrieved and quoted is a live data exposure — the strongest argument against a nightly job.
- Delete from every derived store, not just the primary index: any cache of retrieved chunks, any semantic cache holding an answer built from the document, and any reranker cache. This is routinely missed, and it is how deleted content resurfaces.
- Reconcile. A periodic job comparing document IDs in the source against the index catches whatever the event stream dropped. Without it, divergence is silent and unbounded.
One thing worth designing explicitly: permissions. If a document's access rules change, every cached answer derived from it is now potentially exposed to the wrong audience — which is why chunk-level permission filtering at query time is safer than filtering at index time.
Key takeaways
- RAG supplies facts a model never learned; it is a search problem with a model attached, and retrieval quality is the ceiling on answer quality.
- Chunking decides most of your quality. Split on structure, overlap slightly, and consider embedding small chunks while supplying their larger enclosing section.
- Hybrid retrieval beats vector search alone — keyword matching still handles identifiers and exact terms.
- Reranking a shallow candidate set is usually the largest remaining quality gain and is often skipped.
- Instruct the model to answer only from context and cite sources, and place strong passages at the beginning and end.
- RAG reduces hallucination without eliminating it — grounding checks and citations make failures detectable.
- Diagnose retrieval and generation separately; a single end-to-end score hides which one is broken.
- Propagate deletions and enforce permissions at retrieval time, not after.
Further reading
- Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020) — the original paper.
- Kelvin Guu et al., "REALM: Retrieval-Augmented Language Model Pre-Training" (ICML 2020).
- Nelson Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL, 2024) — why position within the context matters.
- Gordon Cormack et al., "Reciprocal Rank Fusion" (SIGIR 2009) — combining retrievers.
- Shahul Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation" (EACL 2024) — scoring retrieval and generation separately.