LLM Inference at Scale: Why Your Serving Layer Saturates So Fast
A GPU serving an LLM falls over at a request rate a web server won't notice. Batching, buffering and caching are the answers — only one is 'add more GPUs'.
4 min read
Posts tagged LLM
3 posts, newest first.
Designing a RAG Pipeline in a System Design Interview
The boxes are the easy part. What decides a RAG design is what goes into the prompt — and that is a cost decision you make on every single request.
4 min read
How to Reduce Token Costs in an LLM Product at Scale
Four levers, in order of how much they actually save: don't call the model, call a cheaper one, put less in the prompt, and only then tune the model itself.
4 min read
