Designing a RAG Pipeline in a System Design Interview
The boxes are the easy part. What decides a RAG design is what goes into the prompt — and that is a cost decision you make on every single request.
4 min read
Posts tagged Cost
3 posts, newest first.
How to Reduce Token Costs in an LLM Product at Scale
Four levers, in order of how much they actually save: don't call the model, call a cheaper one, put less in the prompt, and only then tune the model itself.
4 min read
Designing for a Workload That Thinks: Capacity Planning When Cost Is Per Token
Model-backed features break classical capacity planning. What changes for queueing, admission control and SLOs when a retry costs money instead of a CPU cycle.
5 min read
