You need to store a 10 TB video archive. No single disk holds it, and even if one did, that disk failing would lose everything.
A distributed file system solves both problems with one idea: split files into chunks, spread the chunks across many machines, and keep several copies of each.
The design questions that follow are where it gets interesting — who remembers which chunk is where, and how do you survive that component failing?
Chunking
A file is divided into fixed-size blocks, typically 64 MB to 256 MB.
video.mp4 (10 GB) -> chunk 1 (128MB) -> nodes A, B, C
chunk 2 (128MB) -> nodes B, D, E
chunk 3 (128MB) -> nodes A, C, F
...Three properties fall out immediately:
Files can exceed any single machine. Chunks are distributed, so file size is bounded by the cluster, not by one disk.
Reads parallelise. Different chunks come from different machines simultaneously, so aggregate throughput scales with the number of nodes involved.
Failures are contained. Losing a node loses chunks that exist elsewhere, and only those chunks need re-replicating.
The metadata service
Something must record which chunks belong to which file, and where each replica lives. In HDFS this is the NameNode; GFS calls it the master.
It holds the directory tree, the chunk list per file, and the current location of every replica. Critically, it is consulted to find data but is not in the data path — clients talk to it once to learn locations, then read bytes directly from the storage nodes.
Diagram, as text
- Client leads to Metadata service (1. where is file X?)
- Metadata service leads to Client (2. chunk list + node addresses)
- Client leads to Data node A (3. read chunks directly)
- Client leads to Data node B
That separation is the central design decision. If file bytes flowed through the metadata service it would be an immediate bottleneck; instead it handles small, infrequent lookups while enormous data transfers happen point-to-point.
It is still the weak point
Two consequences follow, and both are standard interview material.
It is a single point of failure. Lose it and the cluster still holds every byte, but nobody knows where anything is. Modern deployments run a standby with a shared edit log, plus a coordination service — from the previous lesson — to elect the active one.
It is memory-bound. All metadata lives in RAM for speed, so the number of files the cluster can hold is limited by one machine's memory. This is why many small files are the classic anti-pattern: a million 1 KB files consume far more metadata than one 1 GB file while storing a thousandth of the data.
Replication
The default durability mechanism: store each chunk on three nodes.
Placement is deliberate rather than random. A typical policy puts one replica on the local node, a second on a different rack, and a third on a different rack again.
The reasoning is the failure domain point from the availability lesson: three copies in one rack share a power supply and a top-of-rack switch, so a single failure destroys all three. Spreading across racks means the copies fail for genuinely different reasons.
When a node dies, the metadata service notices missing heartbeats and schedules re-replication of every chunk that is now under-replicated — the system heals itself back to three copies.
Erasure coding
Three-way replication costs 300% of your data in storage. At petabyte scale that is a very large bill.
Erasure coding is the alternative. Split data into k fragments, compute m parity fragments, and store all k + m. Any k of them can reconstruct the original.
With a common configuration of 6 data and 3 parity fragments:
| Replication (3x) | Erasure coding (6+3) | |
|---|---|---|
| Storage overhead | 200% extra | 50% extra |
| Failures tolerated | 2 | 3 |
| Read cost (healthy) | Read one copy | Read 6 fragments |
| Recovery cost | Copy one chunk | Read 6, recompute |
| CPU | None | Encoding and decoding |
Better durability at a third of the overhead — but recovery is far more expensive, because rebuilding one lost fragment requires reading six others across the network.
The practical split, used by most large systems: replication for hot data where read latency matters, erasure coding for cold and archival data where the storage saving dominates.
Consistency
Most distributed file systems adopt a write-once, read-many model: files are created, written, and then immutable. Appends may be allowed; arbitrary in-place edits usually are not.
This is a deliberate simplification, and it removes an enormous amount of difficulty. If a chunk never changes after it is written, replicas cannot disagree, caching needs no invalidation, and there are no concurrent-write conflicts to resolve.
You have met this pattern before — it is exactly why photos on a CDN can have year-long TTLs. Immutability makes distribution easy, and systems that can adopt it should.
Object storage: the version you will actually use
HDFS-style file systems are mostly encountered in analytics clusters. Most applications use object storage — S3, GCS, Azure Blob — which shares the same principles with different ergonomics.
Objects are addressed by key rather than a directory path, accessed over HTTP rather than a filesystem API, and durability, replication and erasure coding are handled for you.
For the case studies in this module, object storage is the correct answer for user uploads: photos, videos, documents. You never build one of these, but you should understand why the design looks as it does — and be able to say that images belong in object storage behind a CDN rather than in a database.
Check your understanding
Common mistakes
Storing many small files. The metadata service is the constraint, not disk. Aggregate them.
Assuming random writes work. These systems are optimised for sequential access and append-only workloads. Random in-place updates are slow or unsupported.
Ignoring rack awareness. Three replicas in one rack is one power failure from zero replicas.
Using a distributed file system as a database. No indexes, no transactions, no efficient point lookups. Different tool.
Forgetting the metadata service needs its own high availability. The data survives its failure; your ability to find the data does not.
Practice
Your metadata service holds every chunk location in memory and is approaching its limit. Give three ways to extend its life and say which you would do first.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The table's size is chunk count, and chunk count is total bytes divided by chunk size.
Show a strong answer
The table size is total bytes ÷ chunk size, so there are three levers: fewer bytes, bigger chunks, or a table that need not fit in one machine.
- Increase the chunk size. Doubling from 64 MB to 128 MB halves the entry count immediately. Cheap, effective, and it costs efficiency for small files — which may be irrelevant if the workload is large files, and severe if it is not. This is what I would do first, because it buys time in a single configuration change while the harder work is planned.
- Stop storing small files here. Often a large share of chunk entries come from a small share of the bytes — millions of tiny files each occupying an entry. Moving those to object storage, or packing them into container files with an index, can remove most of the entries while barely touching total storage.
- Shard the metadata service. Partition the namespace across several metadata servers, each owning a subtree or a hash range. This is the real fix and removes the ceiling, but it introduces cross-shard operations — renaming a directory that spans shards becomes a distributed transaction — and it is a substantial project.
Order: chunk size to buy months, small-file cleanup to buy more, sharding as the structural answer. Doing them in that order means the risky change is made with time to do it properly rather than during an incident.
What I would also add immediately: an alert on entry count as a proportion of capacity, because running out of metadata memory takes the entire cluster down — no chunk can be located, even though every byte of data is intact.
Choose between three-way replication and erasure coding for: user profile photos served on every page load, and monthly database backups kept for seven years. Justify each.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask how often each is read, and what a slow read costs.
Show a strong answer
Profile photos — three-way replication.
- They are read constantly and latency matters on every page load. A replicated read is one request to one node; an erasure-coded read of a degraded chunk means fetching several fragments and reconstructing, which adds latency to a user-facing path.
- Replication also gives read parallelism: three copies means three nodes can serve the same photo, which matters when one goes viral.
- The 3× storage cost is real, but profile photos are small and the read volume is enormous, so the cost that dominates is per-read, not per-byte.
Seven-year backups — erasure coding.
- Read approximately never. The reconstruction cost applies only during a restore, which is rare and where minutes are acceptable.
- Volume is enormous and grows monotonically, so per-byte cost is the dominant term. Going from 3× to 1.5× halves a bill that runs for seven years.
- Durability can be made higher than three-way replication while still using less space, by choosing the coding parameters appropriately — which matters for something whose entire purpose is to still exist in seven years.
The general rule this illustrates: replication buys read performance and fast recovery; erasure coding buys storage efficiency. Choose by read frequency, not by how important the data feels.
Key takeaways
- Files are split into large chunks (64–256 MB) spread across nodes, so files exceed any single machine and reads parallelise.
- Blocks are large because metadata is the scarce resource — small files are the classic anti-pattern.
- The metadata service is consulted for locations but stays out of the data path; clients then transfer bytes directly from storage nodes.
- It remains the weak point: a memory-bound single point of failure needing a standby and coordination service.
- Replication is simple and fast to recover but costs 200% overhead; erasure coding cuts that to ~50% with better durability at much higher recovery cost. Hot data replicated, cold data coded.
- Rack-aware placement matters, or three copies share one failure domain.
- Write-once, read-many immutability removes consistency problems — the same reason immutable assets cache so well.
- In practice you will use object storage, which applies these principles behind an HTTP key-value interface.
Further reading
- Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung, "The Google File System" (SOSP 2003) — chunking and a single metadata service.
- Konstantin Shvachko et al., "The Hadoop Distributed File System" (MSST 2010) — the same design in the open.
- James Plank, "Erasure Codes for Storage Systems: A Brief Primer" (USENIX ;login:, 2013).
- Cheng Huang et al., "Erasure Coding in Windows Azure Storage" (USENIX ATC 2012) — local reconstruction codes and recovery cost.
- Brad Calder et al., "Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency" (SOSP 2011).