Designing a RAG Pipeline in a System Design Interview

Ask most candidates to design a retrieval-augmented generation service and you get the same picture within ninety seconds: client, API, embedding service, vector database, model. It is not wrong. It is also not an answer, because every RAG system on earth has those boxes and they differ from each other by an order of magnitude in cost and quality.

The interesting question is one layer down, and it is this: what ends up in the prompt?

Retrieval is a cost decision, not a quality knob

Here is the thing that separates a candidate who has built one of these from a candidate who has read about them. A language model bills by token. Every passage you retrieve and stuff into the prompt is tokens you pay for, on every request, forever.

So "retrieve the top 20 chunks" is not a recall setting. It is a decision to multiply your inference bill.

Work it through. Say you retrieve 20 passages at roughly 200 tokens each. That is 4,000 tokens of context on every single call, before the system prompt, before the question, before the answer. Rerank down to the 5 that actually matter and you are at 1,000. Same architecture, same diagram, same boxes — and four times the bill.

At any real volume that difference is the entire operating cost of the product. It is also invisible on a whiteboard, which is exactly why interviewers push on it.

The reranker is the answer, and it is cheap

The move is to retrieve generously and keep selectively: pull 20 candidates from the vector store, rerank them, and put 5 in the prompt. Retrieval is cheap — an ANN lookup over an index. Generation is expensive. So you spend freely on the half that costs nothing and ruthlessly on the half that costs money.

Candidates who say "I'd add a reranker for quality" are half right. Say the other half out loud: it is a cost control, and at volume it is the dominant one.

Then cache, because most questions are not new

The second lever is that questions repeat. Support queries, documentation lookups, internal search — the distribution has a fat head. A cache in front of the model turns a repeat into a lookup and a model call into nothing.

This is worth being precise about, because it is where "I'd add a cache" stops being a reflex and starts being an argument. A cache hit does not make the model cheaper. It means the model is never called. If 40% of your questions are near-duplicates of a recent one and you cache them, you have removed 40% of your inference spend, not reduced it.

That is a bigger lever than any model choice you could make.

What an interviewer will actually push on

Expect these, roughly in this order:

• "How do you keep the index fresh?" Documents change. A full rebuild is not an answer at any real corpus size — you want incremental upserts, and you want to say what happens to a query that lands mid-update.
• "What happens when retrieval returns nothing relevant?" The honest answer is that the model will confidently answer anyway. Designing for that — a relevance floor, a refusal path — is the difference between a demo and a product.
• "What does this cost at 20 requests a second?" If you cannot ballpark it, you have not really designed it. Twenty requests a second is about 52 million calls a month. At 1,200 tokens each that is 63 billion tokens. Put a per-million price on that and the number is large enough to change what you build.
• "Where does the latency go?" Embedding, retrieval, rerank, generation. One of those dominates and it is almost never the one candidates optimise first.

The shape of a good answer

A good RAG answer is not a bigger diagram. It is the same diagram with the trade-offs made explicit:

• Retrieve wide, rerank narrow, and say why in cost terms
• Cache ahead of the model, and size the cache against the real working set
• Have an answer for stale indexes and for empty retrievals
• Know roughly what it costs, and which component dominates

That last point is worth dwelling on. In almost every RAG design, the model is 80–90% of the bill and everything else is a rounding error. Candidates spend their time optimising the rounding error because it is the part that looks like traditional system design. The interviewer is waiting to see whether you know where the money actually goes.

---

Reading about token cost is not the same as watching a design's bill move when you change one number. Build the pipeline, run traffic through it, and see which component the money is actually going to.