Six teams ship AI features. Each calls a provider SDK directly, holds its own API key, implements its own retries, and picks whichever model the engineer liked.
Nobody can answer what the company spends on AI, or by feature. Switching provider means six migrations. Rate limits are hit unpredictably because six services share one account quota with no coordination. And when a provider has an outage, six teams debug it separately.
This is the problem the API gateway lesson solved for services, and the answer has the same shape: one front door, owning the concerns that should not be reimplemented six times.
What it owns
An AI gateway sits between your applications and every model, self-hosted or third-party.
Diagram, as text
- Feature A leads to AI Gateway
- Feature B leads to AI Gateway
- Feature C leads to AI Gateway
- AI Gateway leads to Provider 1
- AI Gateway leads to Provider 2
- AI Gateway leads to Self-hosted
Provider abstraction. Applications call one interface. Swapping a model is configuration, not a code change across six services.
Credential management. Keys live in the gateway, not scattered across services and developer machines.
Routing. Which model serves this request — the largest cost lever, developed below.
Caching. Exact, prefix and semantic caching implemented once rather than partially in three places.
Rate limiting and quotas. Per team, per feature, per tenant — coordinating a shared provider quota so one runaway job cannot starve everything else.
Retries, timeouts and fallbacks. Resilience patterns applied consistently.
Cost attribution and observability. Every call passes through one point, so spend and latency are attributable by feature, tenant and model without instrumenting each service.
Guardrails. A consistent input and output filtering layer.
That list should feel familiar — it is the API gateway's list, specialised. The value is the same: concerns that are identical everywhere should be implemented once.
Routing is the cost lever
Inference now dominates AI spend for any system with real traffic, and the single most effective reduction is not making a model faster. It is not using the expensive model for easy requests.
Most traffic does not need frontier capability. Classification, extraction, simple lookups and formatting are handled well by far cheaper models.
Strategies, roughly in order of sophistication:
Static routing by task. The simplest and often sufficient — classification goes to a small model, complex reasoning to a large one, decided by which feature is calling.
Classifier-based routing. A cheap model or heuristic assesses difficulty and picks a tier.
Cascade with escalation. Try the small model; if it signals low confidence — via a schema escape value, as the structured outputs lesson described — retry with the large one. Most traffic settles at the cheap tier, hard cases get the expensive one, and quality is preserved where it matters. This is the distillation lesson's hybrid, generalised.
Capability routing. Some requests need a specific capability — long context, vision, a particular tool format — so route on requirements rather than cost.
Fallbacks and failure
A gateway is the natural place for provider resilience, because it already sees every call.
Provider fallback. On outage or sustained errors, route to an alternative. This is the practical payoff of abstraction — worth noting that models are not drop-in equivalents, so prompts may need per-provider variants and quality will differ. Test the fallback path rather than assuming it works.
Circuit breaking. After repeated failures, stop calling for a period. Without it, requests queue against a dead provider and the failure propagates.
Retry with backoff and jitter. Retrying a rate-limited provider immediately makes it worse.
Load shedding. Under pressure, shed low-priority traffic — background summarisation before interactive chat.
Budgets and quotas
Because cost is per request, a runaway loop is a financial incident. The gateway is where you bound it.
Per-team budgets with alerts and hard stops.
Per-tenant quotas, so one customer cannot consume shared capacity — the noisy neighbour problem from the rate limiting lesson, now with a direct monetary cost.
Per-request ceilings on tokens, capping the worst case.
Anomaly alerting on spend rate, not just totals. A monthly total tells you after the fact; a rate tells you within minutes.
Build or buy
Managed gateways exist and cover most of this. The build argument is usually specific routing logic, data residency, or unusual integration needs.
The pragmatic middle is a thin internal library wrapping provider SDKs with your routing, retries and cost logging, before committing to a separate network hop. It captures most of the benefit — one place to change models, consistent retries, attributable cost — without another service to operate.
Move to a real gateway when you need central quota enforcement across teams, cross-language support, or policy that must not depend on each service importing the right library.
Check your understanding
Common mistakes
Direct provider calls from every service. No attribution, no consistency, six migrations.
One model for everything. The most common avoidable expense.
No spend rate alerting. A runaway loop discovered on the invoice.
Untested fallback paths. They fail exactly when needed, and models are not drop-in equivalents.
Gateway with no redundancy. A single point of failure for every AI feature.
Building a full gateway on day one. A thin library captures most of the value first.
Routing without evaluation. Moving traffic to a cheaper tier changes quality — measure it.
Practice
Design the routing policy for an assistant with four steps: intent classification, retrieval query rewriting, answer generation, and a safety check. Justify each choice.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
For each step, ask what a wrong answer costs and how often it runs.
Show a strong answer
Route by required capability and consequence, not by convenience.
- Intent classification — smallest capable model. Runs on every request, output is one label from a fixed set, and the task is narrow. Ideal for a distilled small model with constrained decoding. This is the highest-volume step, so it dominates cost, and it is the easiest to make cheap.
- Query rewriting — small model. Also narrow: turn a conversational question into a standalone search query. Quality matters, but the task has a clear shape and abundant training signal. A wrong rewrite degrades retrieval rather than producing a wrong answer, and it is recoverable because retrieval returns something.
- Answer generation — the largest model the budget allows. This is where quality is visible to the user, where the reasoning is open-ended, and where errors are what people complain about. Do not economise here; economise on the other three so you can afford it.
- Safety check — a small, purpose-built classifier, not a general model. It runs on every output, must be fast, and the task is binary. A general model is slower, more expensive and no better at this.
Two policies on top:
- Escalate on uncertainty. If classification is low-confidence, retry with a larger model rather than proceeding on a bad label. The escalation is rare, so it costs little on average and removes the worst failures.
- Degrade under pressure. If the large model is unavailable or the budget is exhausted, generation falls back to a mid-tier model with the response marked. Better a slightly weaker answer than none.
The measurement that validates the policy: cost and quality per step, not per request. Without the split, nobody can tell whether the small model on classification is saving money or quietly costing accuracy.
A gateway is a single point of failure for every AI feature. Design so that this is acceptable.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The same question asked of any shared edge component.
Show a strong answer
Accept the framing and then make the component boring, because the alternative — every service holding its own credentials and retry logic — has worse failure modes and no observability.
What makes it acceptable:
- Keep it stateless. No session state, no queue, no database on the request path. Configuration and routing rules are loaded and cached, so a configuration store outage does not take the gateway down.
- Run several instances across zones, behind a load balancer, with health checks that test the ability to reach at least one provider rather than merely that the process is alive.
- Make it thin. The most common cause of a gateway outage is logic accumulating in it. Routing, credentials, limits, logging — and nothing that requires knowing what a request means.
- Fail open where it is safe. If the logging pipeline is down, log-and-continue rather than reject. If the budget service is unreachable, allow with a conservative default and reconcile afterwards. A gateway that fails closed on a non-essential dependency turns a minor problem into a total outage.
- Provide a documented bypass. A break-glass path letting a critical service call a provider directly, with credentials that exist but are normally unused. Rehearse it, or it will not work when needed.
- Deploy it carefully — canary, quick rollback, and never simultaneously with a routing-rule change, so a failure has one candidate cause.
What I would monitor as a leading indicator: added latency at P99, not average. A gateway is meant to be invisible in the latency budget, and a rising tail is the first sign it is becoming the constraint rather than the convenience.
Key takeaways
- A gateway gives AI features one front door, owning provider abstraction, credentials, routing, caching, quotas, resilience and cost attribution.
- The reasoning is the API gateway lesson applied to models: identical concerns implemented once.
- Routing is the largest cost lever — the cheapest request is the one a small model handled.
- Prefer a cascade with escalation: small model first, expensive model when confidence is low.
- The gateway is the right place for provider fallback, circuit breaking and load shedding — and fallback paths must be tested, since models are not equivalent.
- Enforce budgets and per-tenant quotas, and alert on spend rate rather than totals.
- It is a single point of failure — run several instances and document a bypass.
- Start with a thin internal library; adopt a real gateway when central policy across teams demands it.
Further reading
- Chris Richardson, Microservices Patterns (Manning, 2018), chapter 8 — the API gateway pattern, which this is.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 21 — overload and graceful degradation at the edge.
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — circuit breakers and fail-open decisions.
- Sam Newman, Building Microservices, second edition (O'Reilly, 2021), chapter 5 — the risk of logic accumulating in shared components.
- Marc Brooker, "Timeouts, retries and backoff with jitter" (Amazon Builders' Library).