Everything in this course so far assumed a property you never had to think about: given the same input, your system returns the same output.
A cache hit is the same as a cache miss. A retry produces the same result. A test asserts an exact value. That assumption underpins caching, idempotency, testing, and most of your debugging instincts.
Put a language model in the request path and it stops being true. This section is about what breaks, and what you do instead.
Five things that change
1. The output is non-deterministic
The same prompt can produce different text on each call. Not because of a bug — because sampling is probabilistic.
Consequences ripple outward. You cannot write assertEqual against a response. You cannot cache on exact output equality. A retry after a timeout may produce a different answer than the one the user never saw.
You can set temperature to zero and get near-determinism, but "near" is doing real work in that sentence — batching, hardware and model updates all introduce variation.
2. Latency is a different order of magnitude
A cached database read takes a millisecond. A large model generating a few hundred tokens takes several seconds.
That is not a slow endpoint you can optimise — it is a fundamental property of generating output one token at a time. It forces architectural responses: streaming so the user sees words appearing, background jobs for anything not conversational, and smaller models where quality permits.
3. Cost is per request, and large
Traditional systems have mostly fixed costs — servers you rent whether or not they are busy. A model call costs money per token, every time.
A feature that is free at 10,000 users can be ruinous at 10 million. This inverts a familiar instinct: caching is normally a performance optimisation here it is primarily a cost optimisation, and it is why an entire lesson in this section is devoted to it.
4. Quality is a spectrum, not a boolean
A database query is correct or it errors. A model response can be accurate, partially right, plausible-but-wrong, or confidently fabricated — and all of them return HTTP 200.
Your monitoring cannot detect a wrong answer by watching error rates. Quality has to be measured deliberately and continuously, which is why evaluation is a first-class engineering concern rather than a research activity.
5. The model is not yours
You depend on an external provider with rate limits, outages, deprecations and version changes that alter behaviour without changing your code.
Everything from the microservices lesson applies with force: timeouts, circuit breakers, fallbacks, and a plan for the day the provider is down or the model you use is retired.
What stays exactly the same
An important corrective, because "AI system design" is often marketed as an entirely separate discipline. It is not.
An AI feature is still a distributed system. It needs load balancing, rate limiting, queues, caching, observability, authentication and graceful degradation — and it needs them more than a typical CRUD service, because its dependencies are slower, costlier and less reliable.
The lessons in this section build on the earlier modules rather than replacing them. If you skipped caching, queues or idempotency, go back — they matter here more, not less.
The shape of a production AI system
Most AI features, whatever they do, look roughly like this:
Diagram, as text
- User leads to Gateway auth, rate limit
- Gateway auth, rate limit leads to Input guardrails
- Input guardrails leads to Orchestrator
- Orchestrator leads to Retrieval
- Retrieval leads to Vector store
- Orchestrator leads to Model serving
- Model serving leads to Output guardrails
- Output guardrails leads to User
- Orchestrator leads to Observability + evals (traces)
Each box gets its own lesson:
- Model serving — how inference actually runs, and why batching and KV caching dominate throughput.
- Retrieval — giving the model facts it was not trained on.
- Orchestration — when one call is not enough and the system must plan, call tools and loop.
- Guardrails — input and output filtering, because prompts are untrusted input.
- Evaluation and observability — how you know it works, and notice when it stops.
The decision that comes first
Before any of it: does this need a model at all?
A surprising share of "AI features" are better served by a database query, a rules engine, or classical machine learning. Models are slow, expensive and non-deterministic — reach for one when the problem genuinely involves open-ended language, unstructured input or judgement.
If it does, the ladder runs roughly:
| Approach | Cost | When |
|---|---|---|
| Prompting a general model | Lowest | Most tasks; start here |
| Prompting + retrieval (RAG) | Low | The model needs your facts |
| Agents and tool use | Medium | Multi-step tasks needing actions |
| Fine-tuning | High | Consistent format or tone at scale |
| Training your own | Very high | Almost never |
Climb only when the rung below has demonstrably failed. Teams routinely jump to fine-tuning for problems that better retrieval would have solved, at a fraction of the cost.
Check your understanding
Common mistakes
Designing as if the model were an ordinary function. A call that takes seconds, costs real money per invocation, and returns something different each time breaks assumptions that hold everywhere else in your architecture — retries are not free, caching is not straightforward, and timeouts are a product decision rather than an infrastructure default.
Leaving cost out of the design. Per-request cost is usually invisible in conventional systems and dominant here. A design that never mentions tokens, context size or model tier has left out the constraint most likely to kill the feature in production.
Having no answer for a wrong output. The model will be confidently wrong. Whether that surfaces to a user, gets caught by a validator, or is gated behind human approval is an architectural choice, and it is the one that determines whether the system is safe to ship.
Treating evaluation as a testing afterthought. Ordinary code is correct or not. Model-backed behaviour is better or worse, which means you need a measurement before you can change anything safely. Teams that add evaluation last spend months unable to tell whether a change helped.
Putting the model on the critical path with no fallback. Providers have outages and rate limits like every other dependency. If the feature is unavailable whenever the model is, that needs to be a decision somebody made rather than one that happened.
Assuming more context is better. Everything you put in the window costs money, adds latency, and can dilute the instruction that actually mattered. Context is a budget, not a bucket.
Practice
A product manager asks for 'AI-powered search' on a catalogue of 50,000 products where users mostly type product names and categories. Assess the request honestly and propose what you would build.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Work out what fraction of queries the existing approach already answers well.
Show a strong answer
Start by measuring, not designing. Sample real queries and classify them: exact product names, category terms, attribute filters ("waterproof jacket under £100"), and genuinely open-ended descriptions ("something for a rainy hike"). In most catalogues the first three dominate heavily.
What each needs:
- Names and categories — keyword search. A well-tuned inverted index with good analysis beats an embedding model on exact matching, and it is faster, cheaper and explainable. Semantic search is worse here: it will happily return something similar when the user typed an exact product name.
- Attribute filters — structured search. "Under £100" is a filter, not a language problem. Extracting it reliably matters far more than understanding it semantically.
- Genuinely descriptive queries — this is where embeddings earn their place, because the words the user chose need not appear in the product text at all.
What I would build: hybrid retrieval — keyword and vector search run together, results fused — with the filters extracted structurally. This is a smaller step than "AI-powered search" and it improves the common case rather than only the rare one.
What I would push back on: replacing the existing search with a model. That trades a fast, debuggable, cheap system for a slower and less predictable one, to improve the minority of queries. The honest framing for the product manager is that the interesting improvement is in the tail, and the tail should be measured before it is funded.
One thing worth adding regardless: log the queries that return nothing or are abandoned. That set is the actual evidence for whether semantic search is needed, and it costs nothing to collect.
Name three things about operating an AI feature that are identical to any other service, and three that are genuinely new. Explain why conflating them is expensive.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The novelty is real but narrower than it is usually presented.
Show a strong answer
Identical, and often forgotten in the excitement:
- It is a network call to a dependency that can be slow, fail, or rate-limit you. Timeouts, retries with backoff, circuit breakers and fallbacks apply unchanged. A team that would never call a payment API without a timeout will call a model without one.
- Cost scales with usage and must be attributed. This is capacity planning with a different unit.
- The data going in and out is subject to the same privacy, retention and access rules as any other data. A prompt containing customer records is a customer-record processing operation.
Genuinely new:
- Correctness is statistical. There is no assertion that fails. You need an evaluation set and a measured pass rate, and "it worked when I tried it" is not evidence.
- The output is untrusted input. A model's response may contain injected instructions, or be used to drive a tool call. It must be validated like anything arriving from outside.
- The dependency changes underneath you. A provider updating a model can change your behaviour without any deploy on your side — a failure mode with no equivalent in a conventional dependency.
Why conflating them is expensive, in both directions: treating the whole thing as novel means rebuilding timeouts and cost tracking as though they were AI problems, badly. Treating it as just another API means shipping with no evaluation set, no guardrails and no version pinning — and discovering the difference in production, on a Friday, when a provider updates a model.
Key takeaways
- Models break the determinism assumption that caching, retries and testing were built on.
- Latency is seconds, not milliseconds, which forces streaming and asynchronous designs.
- Cost is per request, so caching becomes a cost control before it is a performance one.
- Quality is a spectrum and every answer returns 200 — conventional monitoring cannot see a wrong answer.
- The model is a third-party dependency with outages, rate limits and behaviour changes; apply the resilience patterns you already know.
- Everything from the earlier modules still applies — AI systems need those foundations more, not less.
- Ask whether a model is needed at all, then climb the ladder only as far as required: prompting, then retrieval, then agents, then fine-tuning.
Further reading
- Google, "Machine Learning: The High Interest Credit Card of Technical Debt" (NIPS 2014) — the operational costs that outlive the model.
- Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022) — framing ML features as systems rather than models.
- Eugene Yan, "Patterns for Building LLM-based Systems and Products" (2023) — evaluation, guardrails and retrieval as first-class concerns.
- Emily Bender et al., "On the Dangers of Stochastic Parrots" (FAccT 2021) — what language models do and do not do.
- Martin Zinkevich, "Rules of Machine Learning: Best Practices for ML Engineering" (Google, 2017) — rule one is do not use machine learning.