Design an AI Assistant

The final lesson, worked as an interview: design a customer support assistant for a SaaS company. It answers product questions, looks up a customer's account, and performs a small set of actions.

Same five phases as the earlier case studies. What changes is that the answers now involve non-determinism, per-request cost and quality you must measure.

Phase 1: Clarify

Functional requirements:

1. Answer questions from product documentation.
2. Look up the customer's own account — plan, invoices, open tickets.
3. Perform limited actions: reset a password, update billing address.
4. Escalate to a human when it cannot help.

Explicitly out of scope: voice, multi-language, proactive outreach, refunds.

Non-functional — and these drive the design:

• First token under 1 second, streamed. Support chat feels broken if it stalls.
• Grounded answers with citations. A wrong policy statement is worse than no answer.
• Never expose another customer's data. Hard requirement, not best-effort.
• Actions must be safe. Irreversible operations need confirmation.
• Cost per conversation must be bounded, or a support product becomes unprofitable per ticket.

That last pair is what makes this an AI design rather than a search feature.

Phase 2: Estimate

Assume 50,000 conversations a day, averaging 6 turns.

Requests: 300,000 model calls/day ÷ 86,400 ≈ 3.5/second, peak perhaps 10/s. Modest — this is not a throughput problem.

Cost is the real constraint. At a naive 4,000 input and 300 output tokens per turn on a frontier model, a conversation costs several cents and the daily bill runs into thousands. Annualised, that dwarfs the infrastructure.

Corpus: 5,000 documentation pages, roughly 40,000 chunks. Trivial — well within a single Postgres instance with a vector extension.

State the conclusion:

"Traffic is small; this is not a scaling problem. It is a cost and correctness problem. So the design should centre on routing cheap requests to cheap models, caching aggressively, and enforcing correctness with retrieval and guardrails."

That reframing is the point of estimating here.

Phase 3: API and data

POST /v1/conversations                    -> { conversationId }
POST /v1/conversations/{id}/messages      -> streamed response
GET  /v1/conversations/{id}
POST /v1/conversations/{id}/escalate      -> creates a human ticket

Streaming is server-sent events — one-way, simpler than WebSockets, and sufficient.

conversations   (id, user_id, status, created_at)
messages        (id, conversation_id, role, content, trace_id, created_at)
doc_chunks      (id, doc_id, heading, content, embedding, updated_at)
user_memory     (user_id, facts jsonb, updated_at)

trace_id on every message is deliberate: a support complaint about a bad answer must reach the exact trajectory, as the observability lesson argued.

Phase 4: Architecture

Walk one request through it, which is where the marks are.

The gateway handles credentials, cost attribution and provider fallback — one place rather than scattered.

Input guardrails check for injection and strip PII before anything leaves the network.

The router is the primary cost control. Most support messages are simple: greetings, one-hop documentation questions, status checks. A cheap model handles them; only genuinely complex requests reach the expensive path. This is where the bulk of the estimated cost disappears.

Retrieval uses hybrid search — vector for paraphrase, keyword for error codes and plan names — with reranking, and permission filtering applied in the query, not afterwards.

Tools are deliberately narrow: get_account, get_invoices, reset_password, update_billing_address. Reads are free; the two writes require confirmation. Every tool authorises against the requesting user inside the tool.

Output guardrails verify groundedness — does every claim trace to retrieved context — and check no other customer's data appears.

The decisions worth defending

Routing before anything else

Cost falls most from not calling the expensive model. A classifier on the incoming message picks a tier, and the small model escalates when it returns low confidence.

Caching at three layers

Support questions repeat heavily — the skew that makes caching work. Exact caching for identical questions, prefix caching by putting the system prompt and tool definitions first, and semantic caching for the documentation questions that recur constantly in different words.

Semantic caching only for non-personalised answers. "How do I export data?" is cacheable across users. "What is my current usage?" never is — a shared entry there is a data breach, as the caching lesson warned.

Agent only where needed

Most turns are one retrieval and one generation — a chain, not an agent. Only multi-step requests ("why was I charged twice and can you fix the address?") enter the agent loop, with a hard step cap.

This matters because the agents lesson's arithmetic applies: shorter trajectories are more reliable, not just cheaper.

Escalation as a first-class path

The assistant should hand off when confidence is low, the user asks, sentiment turns negative, or it has failed twice. A confident wrong answer is worse than an escalation, and designing the exit is part of designing the product.

Phase 5: Failure and cost

Provider outage. Gateway falls back to a secondary provider; if both fail, the assistant offers immediate human escalation rather than an error.

Retrieval failure. Answer from the model with an explicit caveat, or escalate. Never answer ungrounded and silently.

Runaway cost. Per-conversation token ceiling; per-tenant daily quota; alert on spend rate.

Injection. Documentation is internal and sanitised at ingestion. Ticket text from users is treated as untrusted. Actions are confirmed, and limits live in tools.

Quality decay. Sampled online evaluation scoring groundedness and correctness, with escalation rate as the leading indicator — it rises before satisfaction visibly falls.

Evaluation, before launch

The gate. A set of a few hundred real support questions with known-good answers, covering the tail: ambiguous questions, questions the documentation does not answer, and adversarial attempts.

Measured per stage: retrieval recall, groundedness, correctness, refusal appropriateness, cost and latency. It runs on every prompt, model, retrieval or tool change, and blocks on per-case regressions.

Check your understanding

Common mistakes

Sending the whole conversation every turn. Cost and latency grow with the history, so a long chat gets steadily slower and more expensive until it hits the context limit and fails outright. Conversations need summarisation or windowing from the first design, not once the bills arrive.

Letting the assistant act before it is allowed to. Read-only tools and side-effecting tools are different risk categories. A design that does not separate "look something up" from "send the email" has given an unreliable component your credentials.

No grounding, then surprise at invention. A model asked a question about your data without being given your data will answer anyway. Retrieval is not an enhancement here; it is the difference between an assistant and a plausible-sounding stranger.

Streaming as an afterthought. Whether tokens stream changes the API shape, the error handling and what a retry even means mid-response. Bolting it on later usually means rewriting the transport.

Forgetting who is asking. In a multi-tenant assistant, retrieval that ignores the caller's permissions will happily quote one customer's documents to another. Authorisation belongs in the retrieval query, not in a filter applied afterwards.

Measuring nothing but latency. Fast and unhelpful is still unhelpful. Without a quality signal — thumbs, task completion, a scored eval set — there is no way to know whether the last prompt change made the product better or worse.

Practice

Key takeaways

• Estimate first: at modest traffic this is a cost and correctness problem, not a scaling one — and that reframing drives every later decision.
• Routing to a cheap tier is the largest cost lever, followed by caching at all three layers.
• Never semantically cache personalised answers — a shared entry is a data breach.
• Use a chain for simple turns and an agent only for multi-step requests, with a hard step cap.
• Enforce permissions inside retrieval and inside tools; confirmation for irreversible actions.
• Escalation is a designed path, not a failure — a confident wrong answer is worse.
• Carry a trace id into every message so a complaint reaches the exact trajectory.
• Gate launch on evaluation measured per stage, and audit the documentation before tuning the pipeline.

That completes the section. You now have the components — serving, retrieval, agents, guardrails, evaluation, cost control — and one worked example of assembling them under real constraints.

Further reading

• Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020).
• Nelson Liu et al., "Lost in the Middle" (TACL, 2024) — context ordering under a token budget.
• Shahul Es et al., "RAGAS" (EACL 2024) — faithfulness measurement, which the citation check approximates.
• Stephanie Lin, Jacob Hilton and Owain Evans, "TruthfulQA: Measuring How Models Mimic Human Falsehoods" (ACL 2022).
• Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022) — sizing and cost as first-class design inputs.