LLM Inference at Scale: Why Your Serving Layer Saturates So Fast
A GPU serving an LLM falls over at a request rate a web server won't notice. Batching, buffering and caching are the answers — only one is 'add more GPUs'.
4 min read
Posts tagged AI Systems
6 posts, newest first.
Designing a RAG Pipeline in a System Design Interview
The boxes are the easy part. What decides a RAG design is what goes into the prompt — and that is a cost decision you make on every single request.
4 min read
How to Reduce Token Costs in an LLM Product at Scale
Four levers, in order of how much they actually save: don't call the model, call a cheaper one, put less in the prompt, and only then tune the model itself.
4 min read
Designing for a Workload That Thinks: Capacity Planning When Cost Is Per Token
Model-backed features break classical capacity planning. What changes for queueing, admission control and SLOs when a retry costs money instead of a CPU cycle.
5 min read
Vector Search After the Hype: What We Got Right and What We Overbuilt
Semantic search became a reflex. Hybrid retrieval usually beats pure vectors, chunking outweighs the embedding model, and few corpora need a dedicated store.
6 min read
What AI Agents Changed About System Design: The Request Is Now a Loop
Agents made non-determinism, unbounded iteration and per-request spend ordinary concerns. What that does to timeouts, idempotency, capacity and observability.
5 min read
