A chat endpoint is a request-response service. You know how to run one.
An agent is not that. It runs for minutes or hours, holds state across many steps, executes code, spends money as it goes, and cannot simply be retried from the top when something fails. Most of the operational difficulty comes from that single difference: an agent run outlives the request that started it.
Stop modelling runs as requests
The first architectural decision is refusing the obvious one. Holding an HTTP connection open for the duration of an agent run couples the work to a socket, and it breaks on the first deploy, timeout, or closed laptop.
Model a run as a job:
POST /runs → 202, { runId } start, return immediately
GET /runs/{id} → status, progress, result
POST /runs/{id}/cancel
events → stream progress to any number of watchersEverything you need later follows from this. Resumability requires durable state. Cancellation requires a handle. Progress requires the run to exist independently of any viewer. A run that lives inside a request has none of these.
Checkpoint so it can resume
Diagram, as text
- Client leads to API: accept, return run id
- API: accept, return run id leads to Durable run queue
- Durable run queue leads to Worker claims with a lease
- Worker claims with a lease leads to Step executes
- Step executes leads to Checkpoint after each step
- Checkpoint after each step leads to Worker claims with a lease
- Worker claims with a lease leads to Durable run queue (deploy or crash)
Agents fail partway through, and restarting from zero is expensive — you pay again for every token already spent and discard the context that made progress possible.
Persist after each step: the loop's position, the accumulated context or a pointer to it, budget consumed, and which actions have already been performed. That last one carries the weight, because resumption replays a boundary you were mid-way through.
Which makes idempotency a hard requirement rather than a nicety. If a run crashes after calling a payment API but before recording that it did, resumption must not call it twice. Idempotency keys on every side-effecting action are the standard answer, and agents need them more than ordinary services because their retry paths are less predictable.
Sandbox the execution environment
Agents that run code or touch systems need somewhere to do it where mistakes are contained.
Isolate per run. A container, microVM or ephemeral workspace per run, destroyed afterwards. Shared environments leak state between runs and turn one agent's mistake into another's confusing input.
Grant least privilege. Scope credentials to the run's actual needs. An agent summarising documents does not need write access to the database, and a compromised or confused agent can only reach what you handed it.
Bound resources. CPU, memory, disk, and — easy to forget — network egress. An agent with unrestricted outbound access can exfiltrate whatever it can read, whether by intent, injection, or accident.
Assume it will be exploited. Design the blast radius as though prompt injection will succeed, because eventually it will. The question is not whether the agent can be tricked but what it can reach once it has been.
Concurrency and backpressure
Agent workloads have an awkward shape: long-running, bursty, and rate-limited upstream by the model provider.
Queue runs rather than executing on arrival. Cap concurrency per tenant so one user's fan-out cannot starve everyone else. Handle provider rate limits with backoff that pauses the run rather than failing it — a run that dies at step eighteen because of a transient 429 wastes everything spent so far.
Fan-out needs particular care. An agent that spawns subagents, each spawning more, can exhaust your provider quota from a single request. Cap depth and children, and treat those caps as safety limits rather than tuning parameters.
What to watch
Agent observability differs from ordinary service monitoring in what the useful signals are.
| Signal | Watch for | Usually means |
|---|---|---|
| Steps per run | Distribution shifting up | Flailing; a tool or error path broke |
| Cost per run | Rising tail | Loops not terminating cleanly |
| Termination reason | More budget-exhausted | Stuck loops |
| Tool error rate | Spike on one tool | Dependency broke; agent may be silently working around it |
| Intervention rate | Any rise | Quality regression, visible before evaluations catch it |
| Cache hit rate | A drop | Something is mutating the prompt prefix |
Log enough to replay a run: inputs, context, every tool call and result, decisions, versions. Agent failures are frequently unreproducible, so the trace is often the only evidence you will get.
Two privacy notes. Traces contain whatever the agent read, which may include personal data — apply retention and redaction deliberately. And they can be large; sample verbose traces while always retaining failures.
Failure and recovery
Agents fail in ways ordinary services do not, and each needs a distinct response.
Transient provider errors — pause and back off; never fail the run.
Tool failures — surface as observations so the agent can adapt. Only escalate when a tool is genuinely unavailable.
Stuck runs — detect by unchanging state, not elapsed time, and stop them. This is the failure that quietly burns budget while looking healthy.
Budget exhaustion — exit cleanly with partial results and a statement of what remains. Never stop mid-action.
Poisoned context — when a run has absorbed something wrong and every subsequent step reasons from it, more iterations will not help. Restart from a clean checkpoint rather than continuing.
Check your understanding
Common mistakes
Running agents inside the request. Breaks on deploys, timeouts and disconnects, and makes cancellation and resumption impossible.
No checkpointing. Every failure costs the whole run again.
Side effects without idempotency keys. Resumption duplicates actions.
Unpinned configuration. Runs change definition mid-flight and fail inexplicably.
Shared execution environments. State leaks between runs.
Unrestricted network egress. The exfiltration path nobody bounded.
Unbounded fan-out. One request exhausts the provider quota for everyone.
Failing runs on transient rate limits. Pause and back off instead.
Monitoring only errors. The expensive failures — stuck loops, quality drift — throw none.
Shipping prompt or model changes as code changes. They need the model rollout ladder.
Practice
Design the deployment model for an agent service where runs take 2–15 minutes and deploys happen several times a day.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
A deploy must not kill work in progress, and runs must not block deploys indefinitely.
Show a strong answer
Separate the API from the workers, because they have opposite lifecycle requirements.
- The API accepts a run, writes it to durable storage as
queued, returns a run ID immediately, and serves status. It is stateless and deploys freely at any time. - Workers claim runs, execute them, and checkpoint after every step. They hold all the long-lived work.
Deploying workers without killing runs:
- Drain rather than kill. On a deploy signal, a worker stops claiming new runs and finishes its current one. With a 15-minute maximum, a drain timeout of ~16 minutes covers it.
- If a run exceeds the drain window, checkpoint and release it back to the queue rather than terminating it. A new worker claims it and resumes from the last completed step — which is why checkpointing is the load-bearing part of this design, not an optimisation.
- Roll workers gradually, a few at a time, so capacity never drops sharply.
What makes a run resumable across a version change:
- State stored externally, in a format the new version can read. This means treating the checkpoint format as a contract with the same compatibility rules as any other schema — a new worker will resume runs started by the old one.
- Idempotency keys on every side-effecting step, so a resumed run cannot repeat an action.
Also required:
- A lease on claimed runs, so a worker that dies without releasing does not strand a run forever — the lease expires and another worker takes it.
- Cancellation as a state change, not a process kill. Set
cancelling; the worker notices at the next step boundary and stops cleanly. - A cap on total run duration, so a stuck run is terminated and reported rather than occupying a worker indefinitely.
Your agent service has 200 concurrent runs and downstream APIs are being rate-limited. Design the backpressure.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The constraint is downstream, so the control must be upstream of the work.
Show a strong answer
The mismatch is that agent concurrency and downstream capacity are unrelated numbers. 200 runs each making tool calls can easily exceed what a downstream API permits, and the failures arrive as errors the agent then retries, which makes it worse.
Controls, from the outside in:
- Limit concurrent runs at the worker pool, so the number of in-flight runs is a deliberate number rather than whatever the queue delivers. Runs beyond it wait in the queue — which is exactly what a queue is for, and waiting is a far better outcome than failing.
- A shared rate limiter per downstream dependency, across all workers rather than per worker. Per-worker limits multiply by the worker count, which is the standard mistake.
- A concurrency limit per dependency, not just a rate limit, because a slow dependency accumulates in-flight calls even at a modest rate.
- A circuit breaker per dependency. When it opens, runs needing that tool should pause and be re-queued rather than fail — the work is not lost, it is deferred.
- Retry with backoff, jitter and a budget. Retries during rate limiting are the amplifier that turns pressure into an outage.
What to do with the runs that cannot proceed:
- Re-queue with a delay, preserving the checkpoint, so they resume when capacity returns. This is the advantage of runs being durable jobs — pausing costs nothing.
- Prioritise interactive runs over batch ones, so a user waiting is not behind a thousand background tasks.
What to measure: queue depth, in-flight runs, and per-dependency limiter rejection rate. Queue depth rising with in-flight runs flat means the system is correctly absorbing more work than the downstream will accept — which is the design working, not failing.
Key takeaways
- An agent run outlives its request — model it as a job with an id, status, cancellation and progress.
- Checkpoint per step and make side effects idempotent, because resumption replays boundaries.
- Pin prompt, tool schema and model per run; deploys otherwise change a run's definition mid-flight.
- Isolate per run, grant least privilege, and bound resources including egress. Design for injection succeeding.
- Queue work, cap concurrency and fan-out, and pause on provider rate limits rather than failing.
- Watch steps, cost, termination reason, tool errors, intervention rate and cache hits — not just errors.
- Keep traces sufficient to replay a run; agent failures rarely reproduce on demand.
- Ship prompt, tool and model changes through the model rollout ladder, with the previous version one switch away.
Further reading
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — bulkheads, circuit breakers and backpressure.
- Hector Garcia-Molina and Kenneth Salem, "Sagas" (SIGMOD 1987) — durable multi-step processes.
- Pat Helland, "Idempotence Is Not a Medical Condition" (ACM Queue, 2012).
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 21 — handling overload.
- Adam Wiggins, The Twelve-Factor App (2011) — disposability and graceful shutdown.