How to Reduce Token Costs in an LLM Product at Scale
Four levers, in order of how much they actually save: don't call the model, call a cheaper one, put less in the prompt, and only then tune the model itself.
4 min read
Posts tagged Caching
3 posts, newest first.
Cache Invalidation Is Not the Hard Part
Deleting a key is one line. The hard parts are the copies you forgot, the read-write race that leaves stale data with clean logs, and the stampede at expiry.
5 min read
The Read-Heavy Assumption
"It's read-heavy, so we'll add a cache" is the least examined sentence in system design. Four things that break it, and how to check it in an afternoon.
6 min read
