Two failure detection problems that look unrelated and share a deep similarity: both catch failures that would otherwise be silent.
A dead server that nobody notices keeps receiving traffic. A corrupted file that nobody notices gets served, replicated, and backed up over the good copy.
Heartbeats: is that node alive?
The mechanism is simple. Each node periodically sends "I am alive" to a monitor, or to its peers. Miss enough consecutive beats and you are presumed dead.
every 1s: node -> monitor: "alive"
after 3 missed beats (3s): mark node as failedThe simplicity hides the hard part, which is not the messaging. It is that you cannot distinguish a dead node from a slow one, or from a healthy one you cannot currently reach.
When the monitor stops hearing from a node, three explanations are indistinguishable:
- The node crashed.
- The node is alive but overloaded and its heartbeat is delayed.
- The node is fine and the network between you is broken.
This is the same fundamental ambiguity the introduction lesson raised, and it has no solution — only trade-offs.
The timeout trade-off
Everything reduces to how long you wait before declaring death.
Too short, and you get false positives. A node pauses for garbage collection, misses two beats, and is declared dead. You promote a replacement — and now two nodes think they are the leader, which is the split-brain scenario from the replication lesson.
Too long, and failures go undetected. Requests keep routing to a dead machine and users see errors for the whole detection window.
| Timeout | Detection speed | Risk |
|---|---|---|
| 1s | Very fast | Frequent false positives under load |
| 5s | Moderate | Reasonable default |
| 30s | Slow | Half a minute of failed requests |
Push or pull
Push (node sends beats) scales well — the monitor does no work per node — but a node that is wedged in a way that still lets a background thread emit beats will look healthy while serving nothing.
Pull (monitor polls an endpoint) actually exercises the node's request path, so it detects more real failure modes. This is what load balancer health checks do, and it is why the health endpoint's quality matters so much.
Gossip removes the central monitor entirely: nodes exchange liveness information peer-to-peer, and knowledge spreads without any single component needing to know about everyone. That has its own lesson.
What to do on suspicion
Detecting a failure is only useful if something happens. Standard responses:
- Remove from the load balancer so no new traffic arrives.
- Reassign its work — its shards, its queue partitions, its leadership role.
- Fence it so that if it comes back, it cannot resume acting as leader. This is what prevents split brain, and it is the part beginners omit.
Checksums: is that data intact?
The second problem. Data corrupts silently — a disk bit flips, a network packet is altered, a buggy write truncates a file. Nothing throws an error. You simply have wrong bytes that look exactly as valid as right ones.
A checksum is a small fingerprint computed from the data. Store it alongside, recompute on read, and compare. Different value means the data changed.
const digest = crypto.createHash('sha256').update(buffer).digest('hex');
// on read
if (crypto.createHash('sha256').update(read).digest('hex') !== stored) {
throw new CorruptionError(); // fetch from a replica instead
}Choosing one
CRC32 is fast and catches accidental corruption — random bit flips, transmission errors. It is not secure: anyone can craft different data with the same CRC.
MD5/SHA-1 detect accidental corruption reliably but are cryptographically broken; do not use them where an attacker might deliberately forge a match.
SHA-256 is slower and safe against deliberate tampering.
The rule: CRC for accidents, SHA-256 for adversaries. Verifying your own disk blocks is an accident problem; verifying a downloaded binary is an adversary problem.
Where they are used
- Storage systems checksum every block and verify on read, fetching a replica when a check fails.
- Networks — TCP has a checksum, though a weak one, which is why storage systems add their own rather than trusting the transport.
- Replication uses them to confirm a copy matches the source.
- Deduplication uses content hashes to recognise identical data.
Scrubbing
Checksums only help if you look. Data sitting untouched for years can rot — bit rot is real, and a file you never read can be corrupt for a decade.
Scrubbing is a background process that reads all data, verifies checksums, and repairs from replicas. Without it you discover corruption when you finally need the file — often during a restore, which is the worst moment.
This is why "we have backups" is insufficient. Unverified backups are hope, not a recovery plan.
Merkle trees: comparing large datasets efficiently
A neat extension worth knowing, because it appears in Cassandra, DynamoDB, Git and blockchains.
Comparing two replicas holding a billion records by checking each one is impractical. A Merkle tree hashes records into leaves, then hashes pairs of hashes upward to a single root.
To compare: exchange root hashes. If equal, the datasets are identical and you are done in one comparison. If not, descend only into subtrees whose hashes differ — logarithmic work to locate the divergence rather than linear.
Diagram, as text
- Root hash leads to Hash A-B
- Root hash leads to Hash C-D
- Hash A-B leads to data 1
- Hash A-B leads to data 2
- Hash C-D leads to data 3
- Hash C-D leads to data 4
This is how anti-entropy repair works: replicas periodically compare Merkle trees and sync only the ranges that actually differ.
Check your understanding
Common mistakes
Heartbeats that bypass the real work path. A background thread emitting beats proves the process exists, not that it can serve requests.
No fencing after failover. A recovered node that resumes as leader corrupts data. Use epoch numbers or leases so stale leaders are rejected.
Trusting TCP checksums for storage integrity. They are weak and only cover transmission, not what happens on disk afterwards.
Checksums stored with the data on the same disk. If that disk corrupts both, you detect nothing. Keep them separated or replicated.
Never scrubbing. Corruption discovered during a restore is corruption discovered too late.
Practice
Choose a heartbeat interval and failure threshold for a cluster where a false positive costs a needless failover and a slow detection costs 30 seconds of downtime. Show the reasoning, not just the numbers.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Detection time and false-positive rate move in opposite directions. Decide which error is more expensive first.
Show a strong answer
Decide the asymmetry first. A needless failover costs a leader election, a brief write pause and a cache that starts cold — disruptive but recoverable in seconds. Slow detection costs 30 seconds of genuine downtime. So detection speed is worth more than certainty here, and I should tolerate some false positives.
A workable configuration: heartbeat every 1 second, declare failure after 5 consecutive misses — detection in ~5 seconds, well inside the 30-second cost.
Why not faster? Because the threshold must exceed the longest normal pause the node experiences. If garbage collection routinely stops the process for 2 seconds, a 3-second threshold declares healthy nodes dead several times a day and the cluster spends its life failing over. Measure the actual pause distribution and set the threshold above its high percentile, not above its average.
Why consecutive misses rather than one: a single dropped packet is common and means nothing. Requiring several makes one lost heartbeat harmless while barely delaying detection of a real failure.
Two refinements worth adding:
- Phi-accrual detection rather than a fixed threshold: output a suspicion level from the observed distribution of arrival times, so a node on a consistently slow link is not treated the same as one that has just gone quiet.
- Fencing regardless, because the configuration above accepts false positives by design. Being wrong must be survivable, and fencing is what makes it so.
A storage system reports every replica healthy, but a customer says a file they downloaded is corrupt. Explain how this is possible and what would have caught it.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Heartbeats and checksums answer different questions.
Show a strong answer
Heartbeats prove liveness, not correctness. Every replica is running, responding and serving requests — so every health signal is green. None of them says anything about whether the bytes are still the bytes that were written. This is the availability-versus-reliability distinction in concrete form.
How the corruption happened without anyone noticing: silent data corruption — a bit flipped on disk, a firmware bug, a bad cable, or memory corruption during a write. The record was written correctly and degraded afterwards. Nothing errored, because nothing was checking.
What would have caught it:
- Checksums stored with the data and verified on read. The reader recomputes the hash and compares. This turns silent corruption into a loud, immediate error, and it is the single highest-value defence — it converts "wrong answer" into "detected failure".
- Verify on write as well, so corruption introduced in transit or by a faulty controller is caught before it becomes the stored copy.
- Background scrubbing. Data that is rarely read can rot undetected for years. A process that continuously reads and verifies every replica finds it while other good copies still exist — which is the whole point, because a corruption found after all replicas have rotted is unrecoverable.
- Repair from a good replica automatically, using Merkle comparison to find the divergent ranges cheaply rather than comparing everything.
The design principle: a health check should measure the property you actually care about. If you care that data is correct, something must verify correctness on a schedule, not just check that a process is alive.
Key takeaways
- Heartbeats detect dead nodes, but you cannot distinguish dead from slow from unreachable — only choose trade-offs.
- Timeout is the whole design: too short causes false positives and split brain, too long leaves users hitting dead machines.
- Adaptive detectors that output a suspicion level beat fixed thresholds; pull checks exercise the real path better than pushed beats.
- Always fence a suspected node, and cap how many can be removed at once, or a load spike becomes a self-inflicted outage.
- Checksums catch silent corruption. Use CRC for accidents and SHA-256 where an attacker may forge a match.
- Scrub in the background — corruption found during a restore is found too late, so unverified backups are not a recovery plan.
- Merkle trees compare huge datasets in logarithmic work, which is how replicas repair divergence efficiently.
Further reading
- Naohiro Hayashibara et al., "The φ Accrual Failure Detector" (SRDS 2004) — suspicion levels instead of fixed timeouts.
- Tushar Chandra and Sam Toueg, "Unreliable Failure Detectors for Reliable Distributed Systems" (JACM, 1996) — what detection can and cannot guarantee.
- Ralph Merkle, "A Digital Signature Based on a Conventional Encryption Function" (CRYPTO 1987) — the tree.
- Lakshmi Bairavasundaram et al., "An Analysis of Data Corruption in the Storage Stack" (FAST 2008) — measured rates of silent corruption.
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 8 — unreliable clocks, pauses and fencing tokens.