It is 3am. A customer reports that checkout failed twenty minutes ago. You open the logs and find this:
Error occurred
Processing...
Error occurred
Something went wrong
DoneWhich customer? Which order? Which of your six services produced these? Was the second error caused by the first, or unrelated? You cannot answer any of it, so you start guessing.
Logging is not about writing messages. It is about being able to reconstruct what happened to one request, months later, without having been watching at the time.
Structured logs
The first change that transforms debugging: stop writing sentences, start writing data.
// Unsearchable
console.log(`User ${userId} failed to checkout: ${err.message}`);
// Searchable
logger.error({
event: 'checkout_failed',
userId: 42,
orderId: 'ord_8842',
amount: 2999,
reason: 'card_declined',
durationMs: 1204,
});The first produces text a human can read and a machine cannot. The second produces JSON you can query: "show every checkout_failed for amounts over 1000 in the last hour, grouped by reason." That question is answerable in seconds against structured logs and effectively impossible against prose.
Correlation IDs
One user action often touches five services. Without a shared identifier, their logs are five unrelated streams and you cannot tell which lines belong together.
Fix this by generating an ID at the edge and passing it everywhere:
// Middleware at the entry point
app.use((req, res, next) => {
req.correlationId = req.header('X-Correlation-ID') || crypto.randomUUID();
res.setHeader('X-Correlation-ID', req.correlationId);
next();
});
// Every log line carries it
logger.info({ event: 'payment_started', correlationId: req.correlationId, orderId });
// Every outbound call forwards it
await fetch(url, { headers: { 'X-Correlation-ID': req.correlationId } });Diagram, as text
- Request id: abc-123 leads to API
- API leads to Payments
- Payments leads to Inventory
- API leads to Notifications
Now one search on correlationId:abc-123 returns the complete story across every service, in order. This single practice does more for debugging distributed systems than any other.
Return the ID to the client too. When a customer sends a screenshot with a reference code, you find their exact request immediately.
Log levels, used properly
Levels only help if applied consistently. The rule of thumb is who needs to act.
| Level | Meaning | Example |
|---|---|---|
| ERROR | Something failed that should not; a human should look | Payment gateway returned 500 |
| WARN | Recovered, but suspicious; look if it repeats | Retry succeeded on second attempt |
| INFO | A significant business event happened | Order placed, user registered |
| DEBUG | Detail for diagnosis; off in production | Cache miss for key X |
The three pillars of observability
Logs alone cannot answer every question. Mature systems use three complementary signals.
Logs — discrete events with detail. Best for "what exactly happened to this one request?" Expensive at volume, so sample high-frequency events.
Metrics — numbers aggregated over time: request rate, error rate, latency percentiles, queue depth. Cheap to store forever and ideal for dashboards and alerts. Best for "is the system healthy right now, and how does that compare to last week?"
Traces — the timing breakdown of one request across services. Best for "where did those 3 seconds actually go?" A trace shows the API took 3.1s, of which 2.9s was one database call — which no amount of log reading would have told you as quickly.
The practical division: metrics tell you something is wrong, traces tell you where, logs tell you why.
What to actually monitor
Start with the four signals that catch most real problems:
- Latency — P50 and P99, not the average. As covered in the fundamentals, averages hide the tail where complaints live.
- Traffic — requests per second. A sudden drop is often a more urgent signal than a spike.
- Errors — rate as a percentage of traffic, not a raw count. 100 errors means nothing without knowing whether that is out of 200 or 2 million.
- Saturation — how full the constrained resource is: CPU, memory, connection pool, queue depth. This is your early warning, because saturation rises before latency does.
Then add business metrics, which frequently catch outages that technical metrics miss entirely. If orders per minute drops to zero while every server reports healthy and every endpoint returns 200, something is broken that no infrastructure dashboard would have shown you.
Alerting without destroying your team
The hardest part of monitoring is not collecting data. It is deciding what deserves waking someone.
Alert on symptoms, not causes. "Checkout error rate above 5% for five minutes" matters. "CPU at 90%" often does not — a server can run hot and serve every request perfectly. Alert on what users experience.
Every alert must be actionable. If the responder's only option is to acknowledge and go back to sleep, it should have been a dashboard, not a page.
Use duration thresholds. "Error rate above 5% sustained for 5 minutes" avoids paging for a two-second blip that self-corrected.
Treat alert fatigue as an outage risk. A team that receives forty alerts a night stops reading them, and the one that mattered is missed. Fewer, better alerts beat comprehensive noise.
What must never be logged
Logs are widely readable, retained for a long time, and frequently shipped to third-party services. Treat them as semi-public.
Never log passwords, tokens, session cookies, full card numbers, or personal data beyond what you genuinely need. It is startlingly easy to do by accident:
logger.info({ event: 'login_attempt', body: req.body }); // just logged the passwordRedact deliberately, and prefer logging identifiers over content — userId: 42 rather than the user's email address.
Check your understanding
Common mistakes
Logging inside tight loops. A line per iteration over 100,000 items produces gigabytes and slows the loop. Log the summary.
Only logging failures. Without a success signal you cannot compute an error rate, and rate is what matters.
No retention policy. Logs grow without limit and become expensive. Decide how long each level is kept.
Dashboards nobody watches. A dashboard you check after an incident is documentation. Alerts are what actually notice problems.
Logging and forgetting. If nobody has queried a log line in a year, it is cost without value. Prune it.
Practice
Define what your service logs at each level, and give a concrete example of something teams commonly log at the wrong level.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The level should predict who reads it and when.
Show a strong answer
Define levels by audience and urgency, not by how the developer felt.
- ERROR — something failed that a human must eventually look at, and the service could not complete the work. A payment provider returning a malformed response; a database write that failed after retries. If nobody would act on it, it is not an error.
- WARN — something unexpected that the service handled. A retry succeeded on the second attempt; a deprecated endpoint was called; a cache was unreachable so it fell through to the database. These are the leading indicators worth graphing.
- INFO — significant business events, sparse enough to read: order placed, user registered, job completed with a summary. Roughly one to a few lines per request, not per step.
- DEBUG — the detail needed to reconstruct a decision, off in production by default.
The most common miscategorisation: logging expected failures as ERROR. A user typing the wrong password, a validation rejection, a 404 for a missing page — these are the service working correctly. Logged as errors, they flood the error rate, dominate dashboards, and train everyone to ignore the error log. Then a genuine failure arrives and nobody notices it among ten thousand wrong passwords.
The second most common: an ERROR that is logged and rethrown at every level, so one failure produces five stack traces and looks like five incidents.
The test for ERROR: would you want to be woken for a hundred of these in an hour? If not, it is WARN or INFO, and the thing you actually want is a metric with a threshold.
Your logging costs have grown faster than traffic. Explain the likely causes and how you would reduce spend without losing the ability to debug.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Volume usually comes from a small number of places. Find them before deciding a policy.
Show a strong answer
Find the sources first — the distribution is almost always extremely skewed. Group log volume by service, by level and by message template. In practice a handful of templates usually account for most of the bytes, and the fix is specific rather than a blanket policy.
Likely causes, in the order I would check:
- Debug logging left enabled in production, often from an incident months ago that nobody reverted. Frequently the single largest item.
- Per-item logging inside loops. A job that logs each of 50,000 records produces 50,000 lines to convey one fact: the job processed 50,000 records.
- Logging entire request and response bodies, which grows with payload size rather than with traffic — and is also how personal data ends up in logs.
- Logging that should be a metric. A line per request recording its duration costs orders of magnitude more than a histogram and answers the question worse.
- Errors logged repeatedly at each layer as an exception propagates.
How I would reduce spend without losing debuggability:
- Convert the high-volume, low-value templates to metrics. This is where most of the saving is, and it improves the tooling rather than degrading it.
- Sample by value, never uniformly — keep 100% of errors and slow requests, sample the routine successes. Uniform sampling discards exactly the rare lines you need.
- Tier retention rather than volume: full fidelity for 7 days, sampled for 90. Most investigations happen within days.
- Cap or drop bodies, keeping a hash or a size instead — which also removes a compliance problem.
What I would not do: lower the level threshold globally to WARN. It removes the INFO lines that reconstruct what happened while leaving the expensive misuse in place.
Key takeaways
- Structured logs beat sentences. Stable
eventnames with data in fields make logs queryable; prose does not. - Correlation IDs are the highest-value practice in distributed debugging — generate at the edge, propagate everywhere, return to the client.
- Use levels by who must act: ERROR means a human should look. Normal user mistakes are not errors.
- Metrics say something is wrong, traces say where, logs say why. You need all three.
- Monitor latency (P99), traffic, errors as a rate, and saturation — plus business metrics, which catch outages infrastructure dashboards miss.
- Alert on user-visible symptoms with duration thresholds, and treat alert fatigue as a real risk to reliability.
- Treat logs as semi-public: never log credentials, tokens or unnecessary personal data.
Further reading
- Cindy Sridharan, Distributed Systems Observability (O'Reilly, 2018).
- Benjamin Sigelman et al., "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Google technical report, 2010) — correlation and propagation.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 6.
- Charity Majors, Liz Fong-Jones and George Miranda, Observability Engineering (O'Reilly, 2022) — structured events over unstructured lines.
- OpenTelemetry Specification — trace context propagation across service boundaries.