What AI Agents Changed About System Design: The Request Is Now a Loop

Open a trace for any agent-backed feature and you see something that would have been filed as a bug a few years ago: the same endpoint, called twice with the same input, produced two different call graphs. One request touched three services. The other touched eleven, took nine times as long, and cost more than the entire rest of the page.

Nothing is broken. That is simply what the request is now.

The interesting change isn't that a model is probabilistic — we've shipped probabilistic components before, behind a scoring API with a fixed cost and a fixed shape. The change is that the model is now allowed to decide what happens next, which moves the non-determinism out of the response body and into the control flow.

The unit of work is a loop

A classical handler has a call graph you can draw before the request arrives. An agent handler has a cycle: the model proposes an action, you execute it, you feed the result back, and you repeat until the model decides it's finished. How many times that happens is a function of the input, the state of the world, and the model's judgement on the day.

So the request acquires three properties it never had before. Its duration is unknown when it starts. Its fan-out across your services is unknown when it starts. And it has a marginal cost that accrues per iteration rather than being amortised into your fixed infrastructure bill.

Every difficulty below falls out of those three.

Termination is a design decision, not a property

Nothing in the loop guarantees it ends. The failure shapes are recognisable once you've seen them: two tools that quietly undo each other's work; a tool returning an error the model keeps working around in slightly different ways; a plan that expands faster than it completes.

None of these look like an outage. They look like a request that is still going.

You need explicit bounds — an iteration cap, a wall-clock deadline, a spend ceiling — and a no-progress detector, because a loop can burn all three while still appearing busy. Then you need a decision about what happens when a bound trips, and this is where most implementations are weakest. Returning a generic error and discarding everything the loop achieved is the worst option available, because you've paid for the work and thrown away the receipt. Return the partial trajectory and say plainly what it didn't finish.

Timeouts have to move up a level

Per-call timeouts stop describing the request. Every step can sit comfortably inside its own timeout while the request as a whole runs for minutes — the classic shape where nothing is slow and everything is late.

The replacement is a budget the loop spends. The request arrives with a deadline and a token allowance, each step decrements both, and the loop checks the remainder before starting the next step rather than discovering it afterwards. A step whose expected cost exceeds the remaining budget shouldn't be attempted.

The related problem is that the client's timeout is almost always shorter than the loop's. The connection drops and the loop keeps spending against a caller who has already gone. Which is really an argument that an agent invocation should be modelled as a job — accepted, given an identifier, streamed or polled — rather than as a synchronous request that happens to take a while.

Idempotency now has to cover partial progress

A retried classical handler that is idempotent produces the same effects. A retried agent does not, because the second run re-plans and may take a different path entirely. "Retry the request" and "produce the same outcome" have come apart.

Two things restore sanity. Give each tool invocation an idempotency key derived from its position in the trajectory, so a replayed step is recognised rather than repeated. And make a retry resume from the recorded trajectory instead of starting the reasoning again — which requires that the trajectory is durable state, written as you go, not a variable in memory.

Effects that genuinely cannot be made idempotent — sending mail, moving money, filing a ticket — need to sit behind an explicit approval step or an append-only ledger the loop checks before acting. That constraint is old. What's new is that a component with unpredictable control flow is now the thing calling those effects.

Capacity is counted in concurrent loops

Requests per second stops being a useful capacity unit when one in-flight request holds a slot for an unknown duration and consumes a metered resource while it does. You size for concurrent loops and for tokens, not for arrivals — a change large enough that it deserves its own treatment.

The trajectory is the observable artefact

Logging the input and the final output tells you almost nothing about an agent, because the answer arrived through a path and the path is where the behaviour lives. What you want recorded per request is the ordered sequence of steps: which tools ran, with what arguments, what came back, how much context each step consumed, and — the field people forget — why the loop ended.

That last one turns into the most useful dashboard you will build. Not latency: the distribution of iteration counts, split by terminating reason. Regressions in agent quality show up as loops getting longer well before they show up as loops getting wrong, because a model that has stopped understanding the task doesn't fail, it keeps trying.

The tools are older than the problem

Unbounded iteration, non-deterministic control flow and per-unit cost were all familiar — they lived in batch systems and data pipelines, safely away from the request path. Agents dragged them into ordinary product features, alongside a user who is watching.

The useful mental shift is to stop designing these features like request handlers with an unusual dependency, and start designing them like small batch pipelines that must finish within a person's patience: bounded, resumable, budgeted, and instrumented on the path rather than the result.