Designing for a Workload That Thinks: Capacity Planning When Cost Is Per Token
Model-backed features break classical capacity planning. What changes for queueing, admission control and SLOs when a retry costs money instead of a CPU cycle.
5 min read
Posts tagged SLOs
1 post, newest first.