Your dashboards are green. Latency is normal, error rate is near zero, every request returns 200.
Answer quality has been degrading for three weeks.
The logging and monitoring lesson built the machinery for conventional systems: structured logs, correlation IDs, the four signals, alerting on symptoms. All of it still applies. This lesson covers the three things it cannot see.
What conventional monitoring misses
Quality is invisible. A wrong answer is a successful HTTP response. No status code, no exception, no latency anomaly. The one thing users care about does not appear in your metrics.
Cost varies per request. Two requests to the same endpoint can differ by a hundred times in cost depending on context length and output. Request count tells you almost nothing about spend.
One request is many steps. An agent trajectory is a tree of model calls, retrievals and tool executions. A single duration hides where the time went and which step failed.
Tracing
Diagram, as text
- Request leads to Trace: query, rewrite, chunks, prompt, model version
- Trace: query, rewrite, chunks, prompt, model version leads to Aggregate: latency per stage, cost, retrieval hit rate
- Aggregate: latency per stage, cost, retrieval hit rate leads to Quality: sampled scoring plus user signals
- Quality: sampled scoring plus user signals leads to Rate crossed a threshold?
- Rate crossed a threshold? leads to Alert (yes)
- Rate crossed a threshold? leads to Dashboard only (no)
The foundation. A trace captures the full tree of a request, with each step's inputs, outputs, timing and cost.
trace: "why did checkout latency rise?" 4.8s $0.042
├── guardrail: input check 0.1s $0.000
├── retrieval: search docs 0.3s $0.001 12 results
├── model: plan next action 1.1s $0.011 842 tok
├── tool: query_metrics 1.9s $0.000
├── model: interpret results 1.2s $0.028 2,140 tok
└── guardrail: output check 0.2s $0.002You can now answer the questions that matter: where the time went, which step cost the most, what the model actually saw, and what the retrieval returned.
Record inputs and outputs at every step, not just timings. This is the difference from conventional tracing, and it is what makes debugging possible — when an answer is wrong, you need to see the retrieved passages and the exact prompt, not a duration.
Correlation IDs from the logging lesson carry through unchanged; you need them more, because a trajectory spans many calls.
Cost attribution
Because cost is per request, it must be a first-class metric rather than a monthly surprise.
Attribute along several dimensions:
| Dimension | Question it answers |
|---|---|
| Per feature | Which feature is expensive? |
| Per user or tenant | Who is unprofitable to serve? |
| Per model | Is routing working? |
| Per step | Which part of the pipeline dominates? |
| Cached vs uncached | Is caching earning its keep? |
Two derived numbers are worth watching directly: cost per resolved task — the business-meaningful unit rather than cost per call — and tokens per request, which is the leading indicator. Token growth precedes cost growth, and it usually means context is expanding: history not being compacted, retrieval returning more, tool definitions accumulating.
Quality monitoring
The part with no conventional equivalent. You cannot detect wrong answers from infrastructure metrics, so quality must be measured explicitly and continuously.
Sampled online evaluation. Score a percentage of production traffic with a judge, using the evaluation lesson's machinery. Track the trend rather than absolute values — a decline is the signal.
Implicit feedback is cheaper than asking. Did the user rephrase immediately? Abandon the session? Escalate to a human? Copy the answer? These correlate with quality and cost nothing to collect.
Explicit feedback — thumbs up and down — has low response rates and strong negativity bias, but the comments attached are disproportionately valuable for finding failure modes.
Proxy metrics often move first:
- Retrieval recall falling means answers will follow.
- Groundedness dropping means more ungrounded claims.
- Escalation or fallback rate rising means the cheap path is failing more.
- Refusal rate moving in either direction indicates over- or under-blocking guardrails.
Drift
Quality decays without anything changing on your side, and the causes are worth distinguishing because the fixes differ.
Input drift. Users ask about new things, in new ways. Your retrieval corpus and eval set no longer represent traffic.
Corpus drift. Documents change, are added, or go stale. Retrieval quality moves even though the code did not.
Model drift. The provider updates the model behind an unchanged version string, and behaviour shifts. This is a real and under-appreciated risk of managed APIs.
Detect it by periodically re-sampling production traffic into your eval set and comparing against the historical baseline. A fixed eval set from launch tells you about launch.
Alerting
Same principles as the monitoring lesson: alert on symptoms, use duration thresholds, keep alerts actionable. The specifics differ.
Worth alerting on:
- Cost per hour exceeding a threshold — the runaway loop is a real failure mode
- Quality score dropping below a floor on sampled traffic
- Error and timeout rate from model providers
- Guardrail block rate spiking, which may mean an attack or a broken filter
- Agent step count rising, which signals looping
- Cache hit rate collapsing, which is usually a cost incident in progress
Not worth alerting on: individual bad answers. They are normal. Alert on trends and rates.
Check your understanding
Common mistakes
Timings without inputs and outputs. You cannot debug an answer you cannot see.
Storing raw prompts with no redaction. The common compliance failure.
Cost as a monthly total. Unattributable and too late.
No quality measurement. The one thing users care about, unmonitored.
A fixed eval set forever. Drift goes undetected.
Alerting on individual bad answers. Noise; alert on rates.
No way to find a trace from a complaint. Support reports become unactionable.
Practice
Design the observability for a RAG assistant: what you record per request, what you aggregate, and what you alert on.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Separate what you need to debug one request from what tells you the system is degrading.
Show a strong answer
Per request — enough to reconstruct it completely:
- Trace ID linking every span, and the user and session identity.
- The query as asked, and the rewritten query if rewriting occurs — a rewriting bug is invisible without both.
- Retrieved chunk IDs, scores and the corpus version. Without chunk IDs, retrieval failures are undiagnosable after the fact; this is the single most valuable field and the most often omitted.
- The final prompt sent, after templating and truncation, so you can see what actually arrived.
- Model, version and sampling parameters.
- Token counts — input, output and cached — plus cost.
- Latency broken down by stage: retrieval, rerank, model, guardrails.
- The response, and whether guardrails modified or blocked it.
Aggregate:
- Latency percentiles per stage, not just end to end, so a slow reranker is distinguishable from a slow model.
- Cost per request and per user, and cost per successful interaction.
- Retrieval hit rate: how often the top-k contained a chunk that the answer cited.
- Refusal and fallback rates.
- Quality from sampled scoring, plus user signals such as thumbs-down and follow-up rephrasing — a user rephrasing the same question is a strong implicit failure signal.
Alert on:
- Error and timeout rate from the provider — conventional and still necessary.
- A sharp change in refusal rate, which usually indicates a provider or prompt change.
- A drop in retrieval hit rate, which indicates an indexing problem before users complain.
- Cost per request crossing a threshold, which catches prompt bloat and loop regressions.
- P95 latency by stage.
Not alert on: individual bad answers. They are constant and normal; the signal is in the rate, and paging on instances trains the team to ignore the channel.
Your assistant's quality has degraded over three weeks with no deploys. List the possible causes and how the observability above would distinguish them.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
No deploy means something outside your repository changed.
Show a strong answer
No deploy narrows it usefully — something external moved. Five candidates:
- The provider changed the model behind the version label. Distinguished by: a step change in refusal rate, output length distribution or token counts on a specific date, with prompts unchanged. Pinning the model version and logging it per request is what makes this detectable — without it, this cause is nearly impossible to prove.
- The corpus drifted. Documents were added, edited or deleted, and retrieval quality moved with them. Distinguished by: retrieval hit rate falling while model metrics stay flat, and by correlating the decline with reindex events via the logged corpus version.
- User behaviour changed. A new cohort, a new marketing campaign, or a feature launch bringing questions the system was never good at. Distinguished by: query distribution shifting — cluster the queries by embedding and compare weekly. Quality per existing query type would be unchanged, which is the giveaway.
- A dependency degraded quietly. The reranker timing out and being skipped, or the vector index falling behind. Distinguished by: per-stage latency and error rates, and by fallback counts — a silently skipped stage shows up as a fallback, not as an error.
- Nothing changed except perception. Complaints rose because usage rose. Distinguished by: rate rather than count — thumbs-down per thousand requests, not per week.
The ordering of the investigation: check the provider and the corpus first, because they are the two things that change without any action from you, and both are answerable in minutes from the logged version fields. If those fields were never recorded, the investigation has no starting point — which is the argument for logging them from day one rather than adding them after the first incident.
Key takeaways
- Conventional monitoring cannot see quality, per-request cost, or multi-step structure — the three things that matter most here.
- Trace every step with inputs and outputs, not just timings; that record is what makes debugging possible.
- Traces contain user data — redact, restrict and set retention before enabling capture broadly.
- Attribute cost per feature, tenant, model and step, and watch tokens per request as the leading indicator.
- Measure quality with sampled online evaluation, implicit feedback, and proxies like groundedness and retrieval recall.
- Distinguish input, corpus and model drift, and re-sample production into your eval set to detect them.
- Alert on trends and rates, not individual bad answers — and make traces reachable from a user complaint.
Further reading
- Cindy Sridharan, Distributed Systems Observability (O'Reilly, 2018).
- Benjamin Sigelman et al., "Dapper" (Google technical report, 2010) — trace context propagation.
- OpenTelemetry, Semantic Conventions for Generative AI — standardised span attributes for model calls.
- Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering (O'Reilly, 2022) — high-cardinality data, which prompts and outputs are.
- Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022), chapter 8 — drift detection.