Distributed File Systems

In a Nutshell

A distributed file system (DFS) spreads files across many machines while presenting them to clients as a single, unified filesystem. When your data outgrows what one machine can hold — petabytes of logs, video, or scientific data — you can't just buy a bigger disk. A DFS splits large files into chunks, stores copies of each chunk across a cluster of commodity servers for durability, and coordinates it all so applications see one coherent namespace. Systems like HDFS (Hadoop) and Google's GFS pioneered this for big-data processing, enabling storage and computation across thousands of nodes. A DFS is how you store data too big for any single machine, reliably, on cheap hardware.

How It Actually Works

The Core Idea: Chunk, Replicate, Coordinate

A 10 TB file can't fit (comfortably) on one disk. A DFS:
  1. SPLITS it into fixed-size chunks (e.g., 128 MB blocks)
  2. DISTRIBUTES chunks across many servers (parallel I/O, capacity)
  3. REPLICATES each chunk (e.g., ×3) across servers for durability
  4. COORDINATES via metadata so clients see ONE filesystem

The Master/Metadata + Data Node Architecture

Most classic distributed file systems separate metadata from data:

Component	Role
Metadata/Master node (NameNode in HDFS)	Tracks the namespace, file→chunk mapping, and chunk locations
Data nodes (DataNodes)	Store the actual chunks; serve reads/writes; report health
Client	Asks the master "where are the chunks?", then reads/writes data nodes directly

Client wants file X:
  1. Client → Master: "where are the chunks of X?"
  2. Master → Client: "chunk1 on nodes {A,B,C}, chunk2 on {B,D,E}..."
  3. Client → Data nodes: reads chunks DIRECTLY (in parallel)
  → The master handles only metadata (small, fast); the bulk data flows
    directly between clients and data nodes (scales throughput).

Keeping the master out of the data path is the key scaling insight — it only serves small metadata, so it isn't a bandwidth bottleneck.

Replication for Durability and Availability

Each chunk is stored on multiple nodes (typically 3). If a node dies, its chunks still exist on other nodes, and the system re-replicates to restore the target copy count:

Chunk1 replicas: {A, B, C}
Node A dies → chunk1 still on {B, C}
System detects under-replication → copies chunk1 to a new node D → {B,C,D}
→ Durability maintained automatically on commodity hardware that WILL fail.

Rack awareness: replicas are placed across different racks/zones so a whole rack failure (power, switch) doesn't lose all copies.

Design Assumptions (Why GFS/HDFS Look the Way They Do)

Classic DFS designs were built for big-data batch workloads, which shaped their trade-offs:

Assumption	Design Consequence
Failures are the norm (commodity HW)	Built-in replication + auto-recovery
Files are huge	Large chunk size (64–128 MB) reduces metadata
Writes are mostly appends	Optimized for append, not random writes
Reads are large/sequential	Optimized for throughput, not low latency
Move compute to data	Enables data-local processing (MapReduce/Spark)

Data locality is a defining benefit: instead of shipping petabytes to the compute, you ship the computation to the nodes holding the data — dramatically reducing network traffic for big-data jobs.

DFS vs Object Storage vs NAS

	Distributed File System	Object Storage	NAS (File Storage)
Interface	Filesystem (POSIX-ish)	HTTP API	Filesystem (NFS/SMB)
Scale	Petabytes, thousands of nodes	Effectively unlimited	Moderate
Optimized for	Big-data throughput + locality	Durability, cost, web access	Shared files
Example	HDFS, GFS, CephFS	S3, GCS	EFS, NetApp

Modern cloud data lakes increasingly use object storage (S3) in place of HDFS for the storage layer (cheaper, separates storage from compute), while HDFS remains common in on-prem Hadoop clusters.

Seeing It in Action

Scenario: Storing and processing petabytes of logs with HDFS + Spark.

Problem: 5 PB of clickstream logs; run daily analytics. No single machine
can store or process this.

HDFS storage layer:
  - Logs split into 128 MB blocks, spread across 500 data nodes.
  - Each block replicated ×3, rack-aware (copies in different racks).
  - NameNode holds the metadata (namespace + block locations); DataNodes
    hold the blocks and report health via heartbeats.
  - A DataNode dies → NameNode detects under-replication → blocks
    re-replicated elsewhere automatically. No data lost.

Processing with data locality:
  - A Spark job reads the 5 PB. Instead of pulling 5 PB across the network
    to a compute cluster, the scheduler sends TASKS to the data nodes that
    already hold each block → each node processes ITS local data → only
    small aggregated results move across the network.
  - This is the "move compute to data" principle — the whole reason big-data
    frameworks pair with distributed file systems.

Why not a single big NAS or a database?
  ✗ No single filesystem/DB holds 5 PB affordably or processes it in
    parallel across hundreds of nodes with locality.
  ✓ The DFS provides the capacity, durability (replication), and
    parallelism (distributed chunks) that make petabyte-scale batch
    analytics feasible on commodity hardware.

Modern variant:
  Many teams now store the logs in OBJECT storage (S3) as the "data lake"
  and run Spark/Presto against it — decoupling storage from compute and
  cutting cost — trading some data-locality benefit for elasticity.

Why the architecture works at petabyte scale: a distributed file system solves three problems at once that no single machine can — capacity (chunks spread across the cluster), durability (replication survives the constant failures of commodity hardware), and throughput (parallel I/O plus data locality). The separation of a lightweight metadata master from the bulk data path is what lets it scale: the master answers small "where are the chunks?" questions while terabytes flow directly between clients and data nodes, so the master never becomes a bandwidth bottleneck. And the defining big-data insight — moving computation to where the data already lives rather than shipping petabytes across the network — is only possible because the DFS exposes chunk locations to the scheduler. The modern shift toward object-storage-backed data lakes keeps the same principles (distribution, replication, durability) while decoupling storage from compute for cloud elasticity, but the fundamental idea is identical: to store and process data bigger than any one machine, you distribute it, replicate it, and bring the work to it.

Interview Questions

1. Q: How does a distributed file system store a file too large for one machine? Hint: It splits the file into fixed-size chunks (e.g., 128 MB), distributes those chunks across many commodity servers (for capacity and parallel I/O), replicates each chunk across multiple nodes (for durability), and coordinates via metadata so clients see one unified filesystem namespace. No single node holds the whole file; the system reassembles it transparently.

2. Q: Why separate a metadata/master node from data nodes, and why keep the master out of the data path? Hint: The master (e.g., HDFS NameNode) tracks the namespace and chunk locations — small, fast metadata — while data nodes store and serve the actual chunks. Clients ask the master where chunks are, then read/write data nodes directly. Keeping bulk data off the master means it only handles lightweight metadata requests, so it never becomes a bandwidth bottleneck; data throughput scales with the number of data nodes.

3. Q: How does a DFS maintain durability on unreliable commodity hardware? Hint: Replication — each chunk is stored on multiple nodes (typically 3), placed rack-aware across failure domains. When a node dies, its chunks still exist on other replicas, and the system detects under-replication and re-replicates to restore the target copy count automatically. It assumes hardware will fail and recovers without data loss or manual intervention.

4. Q: What is data locality and why does it matter for big-data processing? Hint: Data locality means sending computation to the nodes that already hold the data, rather than moving huge datasets across the network to a central compute cluster. Since the DFS exposes chunk locations, the scheduler can run tasks where the data lives, so each node processes its local chunk and only small results traverse the network. This drastically cuts network traffic and is why frameworks like MapReduce/Spark pair with distributed file systems.

5. Q: How do classic DFS design assumptions (GFS/HDFS) shape their trade-offs, and how are modern data lakes different? Hint: They assumed frequent failures (→ replication/auto-recovery), huge files (→ large chunk size, less metadata), append-mostly writes (→ optimized for appends, not random writes), and large sequential reads (→ throughput over latency), plus moving compute to data. Modern cloud data lakes often replace HDFS with object storage (S3) to decouple storage from compute and cut cost, keeping distribution/replication/durability while trading some locality for elasticity.

References

• The Google File System (paper) — the foundational DFS design
• HDFS Architecture — the open-source implementation
• Designing Data-Intensive Applications by Martin Kleppmann — distributed storage and batch processing

Dive Deeper

• Ceph architecture — a modern unified distributed storage system
• The evolution from HDFS to cloud data lakes — decoupling storage and compute
• Colossus: Google's successor to GFS — DFS at modern scale