Basic RAG embeds the question, fetches the nearest passages, and answers. It works well for questions that look like the documents that answer them.
It fails predictably on everything else, and the failures share a cause: one embedding of the raw question is a weak search query.
This lesson covers the techniques that address that, roughly in order of how much value they deliver per unit of effort.
The questions basic RAG cannot answer
Conversational follow-ups. A user asks "what about the enterprise plan?" after three previous turns. Embedded alone, that phrase retrieves almost nothing useful — the subject lives in the conversation, not the question.
Comparisons. "How does our refund policy differ from our returns policy?" needs two distinct sets of passages. One embedding lands somewhere between them and may retrieve neither well.
Multi-hop questions. "Who approved the budget for the project that replaced Helios?" requires finding what replaced Helios, then finding who approved its budget. The answer is in no single passage.
Aggregations. "How many incidents did we have last quarter?" is a counting question. Retrieval returns some incidents, never the count.
Recognising which failure you have is most of the work — each has a different fix, and applying the wrong one wastes weeks.
Query rewriting
The cheapest high-value technique: transform the user's question into a better search query before retrieving.
Contextualisation resolves the conversation into a standalone question. "What about the enterprise plan?" becomes "What are the pricing tiers of the enterprise plan?" This single step fixes most conversational RAG, and skipping it is the most common reason chat-based RAG degrades after the first turn.
Expansion generates several phrasings and retrieves for each, taking the union. A user writes "app crashes on startup"; the expansions cover "application terminates during launch" and "fails to initialise". Different phrasings match different documents.
Decomposition splits a compound question into parts, retrieves for each, and answers from the combined context — the fix for comparison questions.
The cost is an extra model call before retrieval, typically with a small fast model. It adds latency and usually pays for itself several times over in answer quality.
Fusion and hybrid ranking
The retrieval lesson introduced hybrid search — vector plus keyword. Advanced RAG generalises this: run several retrievers and merge their results.
Sources might include dense vector search, keyword search, a query-expansion variant, and a metadata-filtered search over recent documents.
Merging scores directly does not work, because scores from different systems are not comparable. Reciprocal rank fusion merges by position instead: a document appearing in the top few of several lists rises, even if no single retriever ranked it first. It needs no tuning and is robust, which is why it is the default choice.
Diagram, as text
- Query leads to Vector search
- Query leads to Keyword search
- Query leads to Expanded query
- Vector search leads to Rank fusion
- Keyword search leads to Rank fusion
- Expanded query leads to Rank fusion
- Rank fusion leads to Rerank
- Rerank leads to Context
Multi-hop retrieval
Some questions need retrieval to happen more than once, where the second query depends on what the first returned.
Two approaches:
Iterative retrieval runs a fixed loop: retrieve, check whether the context answers the question, and if not, generate a follow-up query from what was learned. Bounded by a maximum number of hops.
Agentic retrieval hands the model a search tool and lets it decide when and what to search. More flexible, and covered properly in the agents lesson.
Both cost several sequential model calls, so latency climbs from about a second to several. Reserve them for question types that genuinely need it — routing simple questions down the basic path and only complex ones through multi-hop is the practical design.
Context compression
You retrieved ten passages. Perhaps two sentences in each matter. The rest is tokens you pay for, and noise the model must read past.
Extraction uses a cheap model to pull only the relevant sentences from each passage before assembling the prompt.
Filtering scores each retrieved passage for relevance and drops those below a threshold — often better than a fixed top-k, because the right number of relevant passages varies per question.
Compression matters for two reasons. It cuts cost directly, and it improves accuracy: models attend less reliably to information buried in a long, noisy context. Fewer, better passages usually beat more passages.
Structured and hybrid retrieval
Not every question is answered by text similarity. Mature systems route by question type:
- Factual lookup over structured data — "how many incidents last quarter" is a SQL query, not a retrieval. Let the model generate the query, execute it, and answer from the result.
- Relationship questions — "which services depend on the auth service" suit a graph traversal.
- Recency-sensitive questions — apply metadata filters before similarity.
The general lesson: RAG is one retrieval strategy, not the only one. A router that classifies the question and dispatches to the right backend outperforms forcing everything through vector search.
Evaluating the pipeline
With this many stages, end-to-end scores stop being actionable. Measure each stage:
| Stage | Question | Metric |
|---|---|---|
| Rewriting | Is the rewritten query better? | Retrieval quality with and without |
| Retrieval | Did we fetch the right passages? | Recall@k against labelled answers |
| Reranking | Are the best passages on top? | Precision@k, MRR |
| Generation | Was the answer grounded and correct? | Faithfulness, correctness |
Add techniques one at a time and measure. Adding rewriting, fusion, reranking and compression together tells you nothing about which helped — and some combinations hurt, particularly aggressive compression on top of aggressive filtering.
Check your understanding
Common mistakes
Adding techniques without measurement. Each adds latency and cost; some interact badly. Introduce one at a time.
Skipping query rewriting in a chat product. The cheapest large win, routinely missed.
Multi-hop for every query. Several sequential model calls on a simple lookup is latency nobody asked for. Route by complexity.
Compressing so hard the answer is removed. Aggressive extraction can drop the sentence that mattered. Validate on your eval set.
Assuming more retrieved passages help. Noise degrades answers and inflates cost.
Ignoring structured data. Counting and aggregation questions belong in SQL, not similarity search.
Practice
A question requires combining facts from two different documents — 'which of our suppliers are in countries affected by the new tariff?'. Explain why single-shot retrieval fails and design the pipeline that answers it.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Count how many separate lookups a human would do.
Show a strong answer
Why single-shot retrieval fails. The question needs two disjoint pieces of information: the list of countries affected by the tariff, and the country of each supplier. No single chunk contains both, so any retrieval over the question as written returns documents about one or the other. Ranking cannot help — the missing information is not lower down, it is in a document the query does not resemble.
The pipeline:
- Decompose the question. Use the model to break it into sub-questions: which countries does the new tariff affect? and which countries are our suppliers in? This is the step basic RAG lacks.
- Retrieve for each sub-question independently. The first hits the policy document; the second hits the supplier register.
- Answer each sub-question, keeping the intermediate results explicit rather than pouring both sets of chunks into one context and hoping.
- Join. The intersection is a set operation, and it should be done as one — in code where possible — rather than asked of the model, which will do it approximately over a long context.
- Cite both sources in the final answer, so the user can verify each half.
The important design decision is step 4. The supplier register is structured data; the tariff list may be too. Where a fact lives in a table or a database, retrieve it with a query rather than with similarity search — hybrid structured-and-semantic retrieval. Asking a model to intersect two lists of thirty items in context is a reliability problem you can simply avoid.
What this costs: several model calls and several retrievals per question, so latency and cost rise substantially. Route to it — detect multi-hop questions and use the simple path for the rest, since most questions are single-hop.
Your RAG pipeline has query rewriting, hybrid retrieval, reranking and compression. Quality has not improved over the simple version. How would you find out which stages are earning their place?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Each stage is a hypothesis. Test them one at a time against a fixed set.
Show a strong answer
Build a fixed evaluation set first — questions with known correct answers and known source documents. Without it, every comparison is anecdote, and a four-stage pipeline offers endless places to form an opinion.
Then ablate: remove one stage at a time and measure. Four runs, plus the full pipeline and the simple baseline.
Measure the stages separately, because they fail differently:
- Retrieval quality — is the correct document in the top k? Score this without a model involved. Query rewriting and hybrid retrieval should move this number and nothing else.
- Ranking quality — where does the correct document sit? Reranking should move position, not membership. If recall at k is already 95%, a reranker has almost nothing to fix and its cost is pure loss.
- Answer quality — final correctness, which is what compression and generation affect.
What I would expect to find, and it is worth predicting before running:
- Compression is the most likely culprit. It removes tokens, and the removed token is sometimes the answer. If answer quality drops while retrieval is unchanged, this is where to look first.
- Query rewriting can hurt single-turn queries. A well-formed question rewritten becomes a slightly different question, and the rewrite can lose a rare term that was the whole reason retrieval worked.
- Reranking is usually the safest win, provided the shortlist it reranks is large enough to contain the answer.
The general principle: every stage is a hypothesis, and an untested pipeline is a stack of guesses. A simpler pipeline that scores the same is strictly better — fewer calls, less latency, fewer things to debug.
Key takeaways
- Basic RAG fails on follow-ups, comparisons, multi-hop and aggregation — identify which failure you have before choosing a fix.
- Query rewriting is the highest value per unit of effort, and contextualising follow-ups is essential for any chat product.
- Fusion across several retrievers merged by rank — not score — is robust and needs no tuning.
- Multi-hop retrieval answers questions whose second query depends on the first, at the cost of sequential latency. Route only complex questions to it.
- Compression improves accuracy as well as cost, because attention degrades in long noisy contexts. Fewer, better passages usually win.
- RAG is one retrieval strategy; route aggregations to SQL and relationship questions to graphs.
- Measure each stage separately and add techniques one at a time.
Further reading
- Nils Reimers and Iryna Gurevych, "Sentence-BERT" (EMNLP 2019) — bi-encoders versus cross-encoders.
- Rodrigo Nogueira and Kyunghyun Cho, "Passage Re-ranking with BERT" (arXiv, 2019).
- Ofir Press et al., "Measuring and Narrowing the Compositionality Gap in Language Models" (EMNLP 2023) — multi-hop decomposition.
- Omar Khattab et al., "Demonstrate-Search-Predict: Composing retrieval and language models" (arXiv, 2022).
- Shahul Es et al., "RAGAS" (EACL 2024) — per-stage evaluation.