The previous lesson collected signals. Collecting them is the easy half. This one is about the two decisions that follow: what to measure, and when to wake a human up.
Both are usually got wrong in the same direction. Teams measure whatever is easy to collect, alert on every threshold they can think of, and end up with a system that pages constantly and tells nobody anything. An alert that fires and is routinely ignored is worse than no alert, because it trains people to ignore the next one.
What to measure
Two complementary frameworks cover almost everything, and knowing which applies to what is most of the skill.
RED — for services that handle requests:
- Rate — requests per second.
- Errors — how many of those failed.
- Duration — the latency distribution, as percentiles rather than a mean.
USE — for resources such as CPU, disks, connection pools and queues:
- Utilisation — the fraction of time the resource was busy.
- Saturation — how much work is queued and waiting for it.
- Errors — error events from the resource itself.
The pairing matters. RED describes what your users experience; USE describes why. A checkout service with rising duration (RED) and a connection pool at full saturation (USE) is one diagnosis, not two observations.
From SLI to SLO in practice
You have already met the vocabulary: an SLI is the measurement, an SLO is your internal target, an SLA is the contractual promise with penalties attached. What matters here is turning that into something you can actually alert on.
A usable SLI is expressed as a ratio of good events to valid events:
availability SLI = successful requests / total valid requests
latency SLI = requests served under 300ms / total valid requestsNote that latency becomes a ratio, not an average. "99% of requests complete within 300ms" is measurable, aggregable across time windows, and unambiguous. "Average latency is good" is none of those.
Two details decide whether the SLO is honest:
- Measure where the user is. An SLI collected at your server excludes queueing at the load balancer, TLS negotiation and the network. It will look better than the experience it claims to represent.
- Define "valid" carefully. A client sending malformed requests and receiving
400responses is your system working correctly. Counting those as failures makes your SLO track your clients' bugs instead of your reliability.
Error budgets
This is the idea that makes reliability a decision rather than an argument.
If your SLO is 99.9% over thirty days, you are explicitly permitting 0.1% of requests to fail. That allowance is your error budget, and it is a real, finite quantity:
| SLO | Error budget | Monthly downtime allowed |
|---|---|---|
| 99% | 1% | ~7 hours |
| 99.9% | 0.1% | ~43 minutes |
| 99.99% | 0.01% | ~4 minutes |
| 99.999% | 0.001% | ~26 seconds |
Read the last row honestly. Twenty-six seconds a month is less time than a deployment takes, which means five nines is unreachable for anything requiring human intervention to recover. Each additional nine costs roughly an order of magnitude more than the last, and the cost is paid in redundancy, review overhead and release velocity.
The budget's real function is as a shared currency between engineering and product. "Should we ship faster or invest in reliability?" is unresolvable as an opinion. With a budget it becomes arithmetic:
- Budget remaining — you can afford to take risks. Ship the change, run the migration, loosen the rollout.
- Budget exhausted — reliability work takes priority over features until it recovers.
Alert on symptoms, not causes
The most consequential rule in alerting, and the one most often broken.
A cause-based alert fires on an internal condition: CPU above 80%, disk 90% full, a pod restarting. A symptom-based alert fires on something users can feel: error rate elevated, latency past the SLO threshold, requests failing.
Cause-based alerts fail in both directions. CPU at 85% while everyone is served perfectly is a page that should not have happened. Meanwhile a full cache, healthy CPU and a broken downstream dependency produces an outage no CPU alert will ever catch. You cannot enumerate the causes in advance — that is the whole premise of the previous lesson.
The discipline: page on symptoms; use causes for diagnosis. Causes belong on dashboards you consult after the symptom alert has already woken you, not on the pager.
The reasonable exception is genuine prediction: a disk filling at a rate that hits 100% in four hours is worth a low-urgency ticket, because it is not yet a symptom and will certainly become one.
Burn-rate alerting
Diagram, as text
- SLI: what you measure leads to SLO: the target
- SLO: the target leads to Error budget: allowed failure
- Error budget: allowed failure leads to Burn rate
- Burn rate leads to Page a human now (fast)
- Burn rate leads to File a ticket (slow)
- Error budget: allowed failure leads to Freeze risky changes (exhausted)
Given an SLO, the naive alert is "page when the error rate exceeds the budget". It performs badly in both directions.
Set it sensitively and every brief blip pages someone. Set it conservatively and a slow leak burning 3% of the budget an hour goes unnoticed for a day, by which time the month's budget is gone.
Burn rate fixes this by measuring how fast you are consuming the budget relative to spending it evenly across the window. A burn rate of 1 exhausts the budget exactly at the end of the period. A burn rate of 14.4 exhausts a thirty-day budget in about two days.
The standard approach uses multiple windows at different sensitivities:
| Burn rate | Window | Budget consumed | Response |
|---|---|---|---|
| 14.4× | 1 hour | 2% | Page immediately |
| 6× | 6 hours | 5% | Page |
| 1× | 3 days | 10% | Ticket, not a page |
A fast burn means something is badly broken right now and warrants waking someone. A slow burn means a persistent degradation that needs attention this week but not at three in the morning.
Pairing each long window with a short one that must also be firing stops a resolved incident from continuing to page: the long window still remembers the errors, but the short window has recovered, so the alert clears.
Dashboards, alerts and runbooks
Three artefacts with three distinct jobs, frequently conflated:
- Alerts interrupt a human. Few, symptom-based, always actionable.
- Dashboards support diagnosis once you are already looking. This is where cause-based metrics belong — saturation, pool usage, dependency latency.
- Runbooks say what to do about a specific alert. Linked directly from the alert, listing the likely causes, the checks that distinguish them, and the mitigations.
A page without a runbook delegates the entire diagnosis to whoever is least prepared: someone woken at 3am with no context. The runbook is what turns an alert from an interruption into an action.
Check your understanding
Common mistakes
Alerting on causes rather than symptoms. CPU alerts page you when users are fine and stay silent when they are not.
Static error-rate thresholds. Either too noisy to respect or too slow to catch a sustained low-level failure. Burn rate handles both.
Measuring the SLI at the server. It excludes the load balancer, TLS and the network, and so systematically flatters the number.
Counting client errors against your budget. A 400 response to a malformed request is your system working; including it makes your SLO track your clients' bugs.
Averages in an SLO. Latency objectives must be expressed as a proportion under a threshold, or the tail that generates complaints stays invisible.
Pages with no runbook. The diagnosis falls entirely to the least prepared person at the worst possible hour.
Chasing nines nobody asked for. Each additional nine costs roughly ten times the last, and consistently finishing the month with the budget untouched means you over-invested.
Practice
Your team is paged 30 times a week and most pages need no action. Describe how you would fix this, and what you would refuse to do.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Every page should correspond to something a human must decide. Start by finding which do not.
Show a strong answer
Measure before changing anything. For two weeks, record for every page: did a human take an action, and would the situation have resolved itself? This produces the actionability rate, and it is the number to improve — not the page count, which can be reduced by simply ignoring things.
Then, in order:
- Delete alerts that never led to action. This is usually half of them, and it is pure gain. An alert nobody acts on is training people to ignore alerts they should act on.
- Convert cause alerts into symptom alerts. Replace "CPU > 80%", "disk 70% full", "queue depth > 1000" with a small number of user-facing symptom alerts. Keep the cause metrics on dashboards — they are how you diagnose, not what should wake someone.
- Move anything non-urgent off the pager. If it can wait until Monday, it is a ticket. The test is simple: would you want to be woken for this? If not, it must not page.
- Add burn-rate windows so a short blip does not page and a sustained problem does — with a fast-burn alert that pages and a slow-burn one that files a ticket.
- Require a runbook per remaining alert. If nobody can write what to do about it, it should not page.
What I would refuse to do: raise thresholds until it goes quiet. That reduces pages without reducing problems, and it removes the signal for the case the alert was originally built for. The goal is not fewer pages; it is that every page means something. Those coincide only if you remove the ones that never mattered rather than muting the ones that do.
Define an SLI, SLO and error budget for a checkout API, then explain what the team does differently when the budget is nearly exhausted versus untouched.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The budget is only useful if it changes a decision.
Show a strong answer
SLI — the measurement. The proportion of checkout requests that succeed within 1 second, measured at the load balancer rather than inside the service, because that is closer to what the customer experiences. Excluding client errors (4xx caused by bad input) but including everything the service is responsible for.
SLO — the target. 99.9% of checkout requests succeed within 1 second, measured over a rolling 28 days.
Error budget. 0.1% of 28 days is roughly 40 minutes of full failure, or a proportionally larger period of partial failure.
When the budget is untouched: the team is being too cautious. Ship faster, take on riskier migrations, run chaos experiments, do the database upgrade now. Budget left over at the end of a period is not a prize — it is evidence that reliability was bought with velocity that could have been spent.
When the budget is nearly exhausted: stop shipping features to that service. Freeze non-essential changes, put the next sprint into reliability work, and require an explicit senior decision for anything that risks further burn. This is the point of the budget: it converts "should we prioritise reliability?" from an argument between engineering and product into a number both already agreed on.
The failure mode to avoid is treating the budget as a report rather than a control. If exhausting it changes nothing about what the team does next, the SLO is decoration.
Key takeaways
- RED for services, USE for resources. RED tells you users are suffering; USE tells you why.
- Saturation warns, utilisation confirms — utilisation flatlines at 100% while queues keep growing behind it.
- Express SLIs as good events over valid events, measured where the user is, with client errors excluded from "valid".
- The error budget makes reliability a trade-off with a number attached — budget remaining buys risk, budget exhausted buys reliability work.
- Every extra nine costs about ten times the previous one, and 99.999% leaves under half a minute a month for all human intervention.
- Page on symptoms, diagnose with causes. You cannot enumerate causes ahead of time.
- Use multi-window burn-rate alerts, so a fast burn pages and a slow burn raises a ticket.
- Every page must be user-visible, actionable and urgent, and must link a runbook — otherwise it is a dashboard or a ticket.
Further reading
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapters 4 and 6 — service level objectives and the four golden signals.
- Google SRE, The Site Reliability Workbook (O'Reilly, 2018), chapters 2 and 5 — implementing SLOs and alerting on burn rate.
- Rob Ewaschuk, "My Philosophy on Alerting" (Google internal document, published 2013) — the case for symptom-based alerting.
- Charity Majors, "The Future of Ops Careers" (2019) — on alert fatigue and actionability.
- John Allspaw and Jesse Robbins, Web Operations (O'Reilly, 2010) — runbooks and on-call practice.