LLM Inference at Scale: Why Your Serving Layer Saturates So Fast
A GPU serving an LLM falls over at a request rate a web server won't notice. Batching, buffering and caching are the answers — only one is 'add more GPUs'.
4 min read
ArchiDepth Team
Editors
The engineers who write and review the ArchiDepth course and blog. Every post is reviewed like code before it is published, and a named editor is accountable for each word.
Posts
Designing a RAG Pipeline in a System Design Interview
The boxes are the easy part. What decides a RAG design is what goes into the prompt — and that is a cost decision you make on every single request.
4 min read
How to Reduce Token Costs in an LLM Product at Scale
Four levers, in order of how much they actually save: don't call the model, call a cheaper one, put less in the prompt, and only then tune the model itself.
4 min read
Cache Invalidation Is Not the Hard Part
Deleting a key is one line. The hard parts are the copies you forgot, the read-write race that leaves stale data with clean logs, and the stampede at expiry.
5 min read
Designing for a Workload That Thinks: Capacity Planning When Cost Is Per Token
Model-backed features break classical capacity planning. What changes for queueing, admission control and SLOs when a retry costs money instead of a CPU cycle.
5 min read
Estimation Is Not Arithmetic: Find the Number That Decides the Design
Most candidates compute a storage total nobody needed, never the figure that changes the architecture. Pick the load-bearing number first; precision follows.
4 min read
Eventual Consistency Is a Product Decision
Engineers argue consistency via CAP. The answerable question is what a user will forgive seeing, and how to get that from a stakeholder instead of guessing.
5 min read
Five Causes Behind Most Outages: An Opinionated Taxonomy
Incidents feel infinitely varied and mostly are not. Five recurring shapes, the tell for each, and why the taxonomy is worth more during an incident than after.
6 min read
How to Disagree With Your Interviewer
Pushback is a probe, not a correction. Defend a decision without being defensive, hold a position with its cost attached, change your mind without collapsing.
5 min read
Microservices Were Never About Scale
Teams split monoliths for throughput and got an org-chart fix. The honest case is deploy cadence and ownership — plus narrow cases where boundaries help load.
5 min read
Postgres Until It Hurts: The Operational Case for One Database Doing Several Jobs
The pendulum swung back to one relational database for documents, queues, search and vectors: the case for it, and where a specialised system still wins.
6 min read
Premature Sharding: The Decision You Cannot Take Back
Sharding usually arrives years early and charges rent forever in lost joins, transactions and simplicity. What to exhaust first, and the signals that it's time.
6 min read
The Edge Is Not a CDN: Why Moving Compute Closer Often Makes Things Slower
Caching at the edge moves bytes closer to users. Running code at the edge moves the code away from the data. What that trade costs, and when it genuinely pays.
5 min read
The Follow-Up Question Is the Interview
Candidates over-prepare the first ten minutes and under-prepare the next thirty: the five axes interviewers probe once the boxes exist, and an answer for each.
5 min read
The Migration You Cannot Roll Back
Deploys taught us every change is reversible. Schema and data migrations are not, and teams discover which step was one-way during the incident it caused.
6 min read
The Read-Heavy Assumption
"It's read-heavy, so we'll add a cache" is the least examined sentence in system design. Four things that break it, and how to check it in an afternoon.
6 min read
The Requirements Phase Candidates Rush Through
Four clarifying questions branch a design; the rest are ritual. What to ask before drawing, and how to tell a load-bearing answer from one you'll never use.
5 min read
The Retry That Took Down the Service
A dependency got slow, every client retried, and the retries became the outage. The incident in order, including the twelve minutes it outlived its cause.
5 min read
Vector Search After the Hype: What We Got Right and What We Overbuilt
Semantic search became a reflex. Hybrid retrieval usually beats pure vectors, chunking outweighs the embedding model, and few corpora need a dedicated store.
6 min read
What AI Agents Changed About System Design: The Request Is Now a Loop
Agents made non-determinism, unbounded iteration and per-request spend ordinary concerns. What that does to timeouts, idempotency, capacity and observability.
5 min read
What Senior Sounds Like in a Design Interview
Two candidates, same components. The gap is rarely knowledge — it's unprompted trade-offs, naming non-goals, quantifying first and a qualified 'it depends'.
5 min read
You Probably Do Not Need Kafka
Most teams reaching for a distributed log want a queue, or a table with a status column. What a log costs to run, and the four properties that justify one.
5 min read
Your p99 Is Lying to You: Percentiles Don't Compose
Ten services at p99 don't make a p99 page load: why per-service percentiles mislead, why averaging them is meaningless, and what the client sees that you don't.
5 min read
Backpressure and Rate Limiting: The Two Answers Candidates Skip
Adding a queue doesn't absorb a spike, it defers one. What backpressure and rate limiting actually do, why they're different, and the failure they each prevent.
4 min read
Five Load Balancing Algorithms, and When Each One Betrays You
Round-robin, least-connections, IP hash, weighted and random: what each does to your tail latency, and when the popular default is the wrong choice.
5 min read
Why Static Diagrams Fail You in System Design Interviews
A box-and-arrow diagram can't tell you whether your design works. Here's what interviewers are probing for, and why drawing tools can't help you practice it.
4 min read
