The previous lesson detected a dead node. This one answers a harder question: the node is alive, requests are failing, and nobody knows why.
The distinction people draw is between monitoring and observability. Monitoring answers questions you knew to ask in advance — "is CPU above 80%?" — because you built a dashboard for them. Observability is whether you can answer questions you did not anticipate, from data already collected, without shipping new code.
That second property is what matters in an incident. Every serious outage involves a failure mode nobody predicted, and you cannot add instrumentation to a system that is currently on fire.
The three signals
Three kinds of telemetry, with genuinely different cost and query characteristics. Choosing wrongly is the most common and most expensive mistake in this area.
| Signal | What it is | Answers | Cost driver |
|---|---|---|---|
| Metrics | Numbers aggregated over time | "How many, how fast, how often?" | Unique label combinations |
| Logs | Discrete events with context | "What exactly happened to this request?" | Volume × retention |
| Traces | One request's path across services | "Where did the time go?" | Sampling rate × span count |
The important property: metrics are cheap and lossy, logs are expensive and precise, traces are expensive and structural. A metric tells you error rate rose to 4%. A trace tells you those errors all pass through one downstream call. A log tells you the call returned a specific constraint violation. You usually need all three, in that order.
Metrics and the cardinality problem
A metric is a named number sampled over time, carrying labels — dimensions you can filter and group by.
http_requests_total{service="checkout", method="POST", status="500"} → 1847Three basic types cover most needs:
- Counter — only increases. Requests served, errors, bytes sent. You query its rate, not its value.
- Gauge — goes up and down. Queue depth, memory in use, active connections.
- Histogram — buckets observations so percentiles can be computed. This is how you get p99 latency, and it is the only one of the three that answers the question users actually care about.
Now the part that decides whether your design survives contact with production.
Every unique combination of label values creates a separate time series that must be stored and indexed independently. This is cardinality, and it multiplies:
5 services × 8 endpoints × 4 methods × 6 status codes = 960 seriesThat is fine. Add one label holding a user identifier, with a million users, and you have just asked for 960 million time series from a single metric. Metrics backends do not degrade gracefully at that point; they fall over, and they take your ability to see the incident with them.
This constraint is why the three signals exist as separate things. High-cardinality context is genuinely useful — you do want to find the one customer whose requests fail. Logs and traces carry that context; metrics deliberately cannot.
Logs, and why volume is a design decision
A log line is an event with context attached. The single most valuable change a team can make is moving from unstructured to structured logs.
// Unstructured: greppable by a human, unqueryable by a machine
console.log(`Payment failed for order ${id} after ${ms}ms`);
// Structured: filterable, aggregatable, joinable
logger.error('payment_failed', { orderId: id, durationMs: ms, traceId, reason: 'insufficient_funds' });The second form lets you ask "how many payments failed with insufficient funds, grouped by hour" without writing a regular expression against prose. It also survives someone rewording the message.
Logs are the most expensive signal per unit of insight, because volume scales with traffic and you pay to ingest, index, store and query all of it. Controlling that is a design decision, not an afterthought:
- Level discipline.
DEBUGoff in production by default, switchable per service without redeploying. - Sampling. Keep every error; keep one in a thousand successes. Successes are repetitive; failures are not.
- Retention tiers. Recent logs hot and queryable, older logs in cheap object storage, most deleted.
Traces and context propagation
Once a request touches more than one service, a metric on any single service can only tell you that service is slow. It cannot tell you the latency came from a cache miss three hops downstream.
A trace follows one request across every service it touches. It is a tree of spans, where each span is one unit of work with a start time, duration, and a link to its parent.
Diagram, as text
- gateway — 240ms leads to auth — 12ms
- gateway — 240ms leads to checkout — 219ms
- checkout — 219ms leads to inventory — 8ms
- checkout — 219ms leads to pricing — 196ms
- pricing — 196ms leads to currency-rates — 191ms
Read that tree and the answer is immediate: the gateway is not slow, checkout is not slow, a currency-rate lookup nobody suspected is eating 80% of the request. No amount of per-service dashboards produces that conclusion as quickly.
The mechanism that makes it work is context propagation. Every service must read the incoming trace identifier, attach it to the work it does, and pass it on to everything it calls.
This is where tracing usually fails in practice. Propagation has to hold across every boundary — HTTP headers, message queue attributes, background jobs, thread pools. A single service that drops the header does not lose one span; it severs the tree, and everything below it becomes an orphan. The most valuable trace is the one crossing the boundary nobody instrumented, which is exactly the boundary most likely to be broken.
Sampling
Tracing every request at scale is unaffordable, so you sample.
Head-based sampling decides at the entry point: keep 1% of requests, discard the rest. It is cheap and simple, and it throws away the interesting ones — a rare failure has a 99% chance of being discarded precisely because it is rare.
Tail-based sampling buffers spans until the request finishes, then decides: keep everything that errored or exceeded a latency threshold, plus a small sample of the healthy ones. You keep the traces that matter, at the cost of holding spans in memory until the decision can be made.
Correlating the three
The signals are far more valuable joined than separately, and joining them is a design choice you must make deliberately.
The mechanism is a shared identifier. Put the trace ID on every log line and attach it as an exemplar on the metrics you record. That gives you a path an on-call engineer can actually walk:
- An alert fires on the metric — p99 latency crossed the threshold.
- The exemplar links to a trace from a request in that slow bucket.
- The trace names the slow span, and its trace ID pulls the logs from that exact service and moment.
Without the shared ID, step two and step three are manual guesswork against timestamps during an incident, which is the worst possible time to be doing joins by hand.
Where it sits in the architecture
Instrumented applications should not talk to a storage backend directly. The conventional shape puts a collector between them:
Diagram, as text
- Service A leads to Collector
- Service B leads to Collector
- Service C leads to Collector
- Collector leads to Metrics store
- Collector leads to Log store
- Collector leads to Trace store
The collector batches, retries, redacts sensitive fields, applies sampling, and — critically — decouples your application code from your telemetry vendor. Changing backends becomes a collector configuration change rather than a redeployment of every service. Vendor-neutral instrumentation standards such as OpenTelemetry exist precisely so that this seam is available.
Two properties are non-negotiable. Telemetry export must be asynchronous and bounded: if the collector is unreachable, the application drops telemetry and keeps serving traffic. And the observability stack must not share failure domains with the system it observes — a monitoring system that goes down with your primary database cannot tell you why your primary database went down.
Check your understanding
Common mistakes
Unbounded labels on metrics. User IDs, request IDs and raw URL paths turn one metric into millions of series and take the metrics backend down with them.
Instrumenting only what is easy. CPU and memory are trivial to collect and rarely explain an outage. Request rate, error rate and latency distribution are what actually correlate with user pain.
Averages instead of histograms. A mean latency of 200ms is consistent with everybody being served in 200ms and with 95% at 50ms while 5% wait four seconds. Record distributions.
Broken trace propagation. One service that drops the trace header orphans every span below it, and the gap is invisible until you need it.
Head sampling on low-traffic error paths. Sampling 1% of requests discards 99% of your rare failures — the traces you would most want.
Logging entire payloads. Ruinous at volume, and it quietly replicates personal data into a store with weaker access controls than the database it came from.
Observability that shares a failure domain with production. If it dies with the system it watches, it cannot tell you why the system died.
Practice
A request is slow for 1% of users. You have all three signals. Describe the investigation in order, and say what each signal contributes and where it stops being useful.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Each signal answers one question and then hands over to the next.
Show a strong answer
- Metrics — is it real, and where? Check P99 by endpoint, region and version. This confirms the tail is genuine rather than a handful of outliers, and narrows it to a surface. Where it stops: metrics are aggregates. They can tell you the checkout endpoint's P99 doubled; they can never tell you why one request was slow, because the individual requests are gone.
- Compare the slow population with the fast one. Still in metrics, but the useful move is differential: are the slow requests concentrated in one region, one client version, one shard, one tenant? A tail that is uniformly distributed is a capacity problem; a tail concentrated on one dimension is usually a specific defect. Where it stops: it identifies a correlate, not a cause.
- Traces — which hop? Pull traces from the slow population specifically, not a random sample. The span breakdown shows where the time went: one slow dependency, a serial chain that should be parallel, or many small calls adding up. Where it stops: a trace tells you which span was slow, rarely why — a 900 ms database span does not say whether it was a bad plan, lock contention or a cold cache.
- Logs — what happened inside that span? Using the trace ID, pull exactly the lines for those requests. Now you see the query, the retry, the cache miss, the error that was swallowed.
The dependency worth noticing: step 4 is only possible if the trace ID was propagated into the logs. Without that, step 3 tells you which service to look at and step 4 becomes searching millions of lines by timestamp. Context propagation is the thing that makes the three signals one system rather than three dashboards.
Your logging bill has tripled and someone proposes sampling logs at 10%. Argue for and against, then propose something better.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask which log lines you would most regret not having, and whether sampling keeps them.
Show a strong answer
For sampling: log volume is dominated by successful, unremarkable requests that nobody will ever read. Keeping one in ten of those loses almost nothing and cuts the bill immediately. It is simple and needs no per-line decisions.
Against, and this is the serious objection: uniform sampling discards the rare events, which are the only ones anyone looks for. The 0.1% of requests that failed are exactly what you need during an incident, and a 10% sample keeps one in ten of them — so a rare failure that occurred five times leaves you with zero or one line, and you cannot tell which. Sampling optimises for the cost of the common case and pays for it with the value of the rare case.
Something better, in order of value:
- Tail-based sampling. Decide after the request completes: keep 100% of errors and slow requests, and a small percentage of successful fast ones. This is where nearly all the saving is available with none of the loss, because the expensive volume and the valuable lines are almost disjoint sets.
- Move the high-volume, low-value lines to metrics. A log line emitted on every request to record its duration is a metric wearing the wrong costume — it costs a thousand times more to store and answers the question worse.
- Reduce level in production deliberately. Debug logging left on is frequently most of the bill, and it is rarely the thing that solves an incident.
- Shorten retention rather than sampling. Most investigation happens within days. Keeping 100% for 7 days and 1% for 90 usually costs less than 10% for 90 and is far more useful.
The principle: sample by value, never uniformly. The cheapest lines to drop are the ones nobody would ever search for.
Key takeaways
- Monitoring answers known questions; observability answers unanticipated ones — which is what every real incident consists of.
- Metrics are cheap and lossy, logs are precise and expensive, traces are structural. Alert on metrics, navigate with traces, confirm with logs.
- Cardinality, not storage, is the binding constraint on metrics. A label may only hold values from a small, fixed, known set.
- Structured logs are queryable; unstructured logs are only greppable. Sample successes aggressively, keep every error.
- Traces need context propagated across every boundary — one service dropping the header severs the tree below it.
- Tail-based sampling keeps the traces that matter, because head-based sampling discards rare failures by construction.
- Correlate with a shared trace ID so an alert leads to a trace leads to logs, instead of manual timestamp matching mid-incident.
- Export asynchronously through a collector, and never let the observability stack share a failure domain with the system it observes.
Further reading
- Cindy Sridharan, Distributed Systems Observability (O'Reilly, 2018) — the three signals and what each is for.
- Benjamin Sigelman et al., "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Google technical report, 2010) — where context propagation and sampling were set out.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 6 — the four golden signals.
- Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering (O'Reilly, 2022) — high-cardinality data and why metrics cannot carry it.
- OpenTelemetry Specification — the vendor-neutral definitions of spans, context and baggage.