The average organisation does not discover a breach itself. It is told — by a customer, a researcher, a payment processor, or an attacker demanding money.
That statistic is the whole argument for this topic. Every control in this module reduces the chance of compromise; none of them reduce it to zero. So the honest question is not only how do we prevent this? but how would we know, and what would we do? A team that has never answered the second is improvising during the worst hour of its year.
Detection is a design problem, not a tool
You cannot detect what you never recorded. Detection is decided at design time, by what gets logged and whether anything looks at it.
The signals that matter for security are not the ones that matter for debugging. An application log is written to answer "why did this request fail?". A security log answers "who did what, to what, and when?" — and it needs to survive an attacker who would prefer it did not.
The events worth recording deliberately:
- Authentication — successes and failures, with the account, source and method. Failures alone tell you about guessing; successes tell you where an account is being used from.
- Authorisation denials. A user being refused access to objects they do not own, repeatedly, is one of the clearest signals of enumeration there is — and almost nobody alerts on it.
- Privilege and permission changes. Who granted what to whom.
- Data access at volume. One record read is business as usual; fifty thousand read by one account in an hour is not.
- Configuration and credential changes, including new API keys and changed webhooks.
What to alert on, and what merely to record
Most security telemetry is for investigating afterwards, not for waking someone. Alerting on everything reproduces the alert fatigue problem, and a security alert that is routinely ignored is worse than none, because it creates a record of a warning nobody acted on.
Alert on a small number of things that are specific, rare and actionable:
- A successful login from a new country immediately after failures from the same account.
- A spike in authorisation denials from one account.
- Any use of a break-glass or administrative credential.
- Data export or bulk read above a threshold.
- A change to logging, alerting or authentication configuration — attackers turn off the lights before working.
Everything else is recorded and searched during an investigation.
The first hour
The value of a plan is that it removes decisions from the moment when judgement is worst.
Diagram, as text
- Detect leads to Triage: is it real, how bad?
- Triage: is it real, how bad? leads to Contain: stop it spreading
- Contain: stop it spreading leads to Eradicate: remove the access
- Eradicate: remove the access leads to Recover: restore service
- Recover: restore service leads to Learn: change the system
Triage. Confirm it is real and establish scope. What is affected, what data, which accounts, is it ongoing. Resist the urge to fix anything yet — the first action taken is often the one that destroys the evidence needed to understand what happened.
Contain. Stop it spreading. Revoke sessions and credentials, isolate the affected component, block the path. Preserve state before destroying it: snapshot the machine rather than terminating it, because a terminated instance takes the answer with it.
Eradicate. Remove the access, not just the symptom. A rotated password does nothing if the attacker holds a persistent token, an added SSH key, or a webhook they installed.
Recover. Restore service, from a known-good state rather than the compromised one, and watch closely for re-entry.
Learn. Change the system so this class of event is prevented or detected faster.
Communication is part of the response
Two audiences, and both have deadlines that are not negotiable.
Regulators and affected people. Many jurisdictions require notification of a personal-data breach within a fixed window — 72 hours is common — and the clock starts when you become aware, not when you finish investigating. That means legal and communications must be involved early, not after engineering is satisfied.
Customers. Say what happened, what data was involved, what you have done, and what they should do. Vagueness is read as evasion, and the reputational damage from a badly handled disclosure routinely exceeds the damage from the breach.
The review afterwards decides whether it recurs
A post-incident review is worth more than the response if it is blameless and specific.
Blameless because a review that identifies a person to blame teaches everyone else to be less forthcoming next time, and the accuracy of future reviews is worth more than the satisfaction of this one. Systems that permit a single mistake to become a breach are the finding, not the mistake.
Specific because "improve security awareness" changes nothing. "Authorisation denials are not alerted on" produces a change with an owner and a date.
Check your understanding
Common mistakes
Assuming prevention is enough. Every control has a failure mode, and a plan that begins after the failure is the one nobody writes.
Logging for debugging and calling it security logging. Application logs rarely record who accessed which object, and that is the question an investigation asks.
Alerting on everything. A security channel full of noise is a channel nobody reads, and the record of the ignored alert makes the eventual incident worse.
Destroying evidence while containing. Isolate and snapshot before terminating; the instinct to remove the problem removes the answer with it.
Rotating the obvious credential only. Attackers establish persistence. Eradication means finding what they added, not just changing what they took.
A plan that has never been rehearsed. An untested runbook is a document, not a capability — the first time it is read should not be during an incident.
Practice
You are told at 09:00 that a customer can see another customer's invoices. Walk through your first hour, in order, and say what you would deliberately not do.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Establish whether it is real and how far it goes before changing anything.
Show a strong answer
09:00–09:15 — Triage. Reproduce it. Is this a data leak, a caching bug, or a support agent looking at a shared screen? Establish the mechanism, because the response differs entirely. If it is an access-control flaw, determine whether it is exploitable by anyone or only by this customer's specific circumstances.
09:15–09:30 — Scope. Query the access logs for the pattern: how many accounts have retrieved objects they do not own, over what period. This answers the question every subsequent decision depends on — one accident or systematic exploitation. Do this before fixing, because deploying a fix changes the logs and can make the historical query harder.
09:30–09:45 — Contain. If exploitation is active, disable the affected endpoint or feature rather than waiting for a correct fix. A feature being unavailable is recoverable; continued exposure is not. Preserve the logs by exporting the relevant window to somewhere the deploy pipeline will not rotate away.
09:45–10:00 — Notify internally and start the clock. Bring in whoever owns legal and communications. If personal data was accessed by unauthorised parties, a regulatory deadline has already started and it began at 09:00, not when engineering finishes.
What I would deliberately not do:
- Not deploy a hotfix in the first fifteen minutes. A fix written before the mechanism is understood frequently addresses the symptom and leaves the class of flaw, and it disturbs the evidence.
- Not tell the reporting customer "it is fixed" until scope is established. That statement is repeated publicly.
- Not treat it as a single bug. Broken object-level authorisation is rarely isolated to one endpoint; the same missing check usually exists elsewhere, and finding those is part of eradication rather than follow-up.
Design the security logging for a product handling personal data: what you record, where it goes, how long you keep it, and what you deliberately do not log.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The last part is as important as the first — logs are themselves a place data gets breached.
Show a strong answer
What to record — the events that answer "who did what, to what, when, and from where":
- Authentication successes and failures, with account, source address, client and method.
- Authorisation denials, with the subject, the object requested and the rule that refused it.
- Reads and exports of personal data, with the record identifiers and the count.
- Permission, role and credential changes, including API key creation.
- Changes to logging and alerting configuration.
Where it goes: shipped as written to a separate account or project, append-only, with retention the application's own credentials cannot alter. The production service must be able to write and not to delete.
How long: long enough to investigate something discovered late — a year is a common floor, since breaches are frequently found months afterwards. But retention is itself a liability, so tier it: full detail for 90 days, reduced fields for a year, and delete on schedule rather than accumulating indefinitely.
What deliberately not to log — this is where teams create the breach they were trying to detect:
- Never passwords, tokens, session identifiers, API keys or card numbers, including inside request bodies and stack traces.
- Not full request or response bodies on endpoints handling personal data. Record identifiers and shapes, not contents.
- Not more personal data than the event needs. "User 4821 exported 300 records" is sufficient; the records themselves are not.
The rule that ties it together: a security log should be enough to reconstruct what happened and never enough to repeat the harm. If an attacker who obtained your logs would gain the same data as breaching the database, the logs have become a second copy of the problem.
Key takeaways
- Assume prevention fails, because most organisations learn about breaches from outside. The design question is how you would know.
- Detection is decided at design time by what you record. Security events — authentication, authorisation denials, bulk access, permission changes — are not the same as debugging logs.
- Logs a compromised application can delete are not evidence. Ship them off the machine to somewhere the application cannot modify.
- Alert on few things, record many. A noisy security channel is an ignored one.
- Triage, contain, eradicate, recover, learn — and preserve state before destroying it, or you lose the scope.
- Decide who decides in advance, and rehearse; an unpracticed runbook is not a capability.
Further reading
- NIST Special Publication 800-61 Revision 2, Computer Security Incident Handling Guide — the standard lifecycle.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapters 14–15 — incident management and blameless postmortems.
- John Allspaw, "Blameless PostMortems and a Just Culture" (Etsy engineering, 2012).
- Richard Bejtlich, The Practice of Network Security Monitoring (No Starch Press, 2013) — detection as an ongoing discipline rather than a product.
- Regulation (EU) 2016/679 (GDPR), Articles 33 and 34 — the notification obligations and their timing.