Ephemeral Storage

9 min read

In a Nutshell

Ephemeral storage is temporary storage that lasts only as long as the compute instance using it — when the instance stops, restarts, or is replaced, the data is gone. It's the opposite of persistent storage, which survives independently of any compute. The local disk of a container or VM, the memory of a serverless function, and a scratch space for intermediate processing are all ephemeral. Understanding the distinction is critical: putting data you need to keep on ephemeral storage is a classic cause of data loss, especially in cloud-native systems where instances are routinely killed and recreated. The rule is simple — ephemeral for transient/regenerable data, persistent for anything you must not lose.

2D minimalistic diagram contrasting two storage types: on the left an instance with attached ephemeral storage (local disk) where data vanishes with a "poof" when the instance is terminated; on the right the same instance connected to persistent storage (a separate durable volume/object store) that survives independently when the instance is replaced

How It Actually Works

Ephemeral vs Persistent

	Ephemeral Storage	Persistent Storage
Lifespan	Tied to the instance/process	Independent of compute
Survives restart/replace?	No	Yes
Speed	Often faster (local)	Varies (network volumes slower)
Cost	Cheap / included	Charged separately
Examples	Container local disk, instance store, function memory/`/tmp`, RAM	EBS, persistent volumes, S3, managed DB
Use for	Scratch, cache, temp files, intermediate results	Anything you must keep

Where Ephemeral Storage Shows Up

- Container filesystem:  writes to a container's local layer vanish when
                         the container is recreated (deploys, crashes, scaling).
- VM instance store:     locally-attached NVMe on some cloud instances is
                         fast but LOST when the instance stops/terminates.
- Serverless /tmp + RAM: a Lambda's local disk and memory are gone after
                         the invocation environment is recycled.
- Kubernetes emptyDir:   a pod's scratch volume deleted when the pod dies.

The Cattle, Not Pets Principle

Cloud-native design treats instances as cattle, not pets — interchangeable and disposable, killed and recreated freely for scaling, deploys, and healing (see Containers & Orchestration). This is only safe if instances hold no irreplaceable state:

Instances are constantly:
  - autoscaled up/down      - redeployed (new image)
  - rescheduled to new nodes - killed and replaced on failure

If important data lived on the instance's ephemeral disk → GONE each time.
→ Cloud-native apps must be STATELESS: keep durable state in external
  persistent stores (databases, object storage, Redis), not on the instance.

This connects directly to why statelessness enables horizontal scaling: instances can be disposable precisely because they hold nothing you can't afford to lose.

Good Uses of Ephemeral Storage

Ephemeral storage isn't bad — it's the right tool for transient data, and often faster than persistent storage:

Use	Why Ephemeral Fits
Scratch space	Intermediate files in a processing job (regenerable)
Local cache	Cached data that can be re-fetched on miss
Temp files	Uploads being processed, then moved to durable storage
Build artifacts	CI/CD intermediate outputs
Session compute	A function's working memory during one invocation

The key test: if this data disappeared, could the system regenerate it or continue without harm? If yes, ephemeral is fine (and cheap/fast). If no, it must be persistent.

The Classic Data-Loss Mistakes

❌ Storing user uploads on a container's local disk
   → next deploy recreates the container → uploads GONE.
   ✅ Fix: write uploads to object storage (S3) immediately.

❌ A database writing its data files to instance-store (ephemeral) disk
   → instance stops → the entire database is LOST.
   ✅ Fix: put the DB data on a persistent volume (or use a managed DB).

❌ Relying on local disk for logs
   → instance replaced → logs vanish before anyone reads them.
   ✅ Fix: ship logs to a central aggregator (see logging.md).

Ephemeral Can Be Faster — Use It Deliberately

Local instance-store NVMe is often faster than network-attached persistent volumes (no network hop). So a valid pattern is: use fast ephemeral storage for high-throughput intermediate work (scratch, temp indexes, spill space for big sorts), while the durable results land in persistent storage. The point isn't to avoid ephemeral storage — it's to use it knowingly for the right (transient) data.

2D minimalistic diagram showing a data-processing pipeline: raw input read from persistent storage, intermediate results written to fast ephemeral scratch disk during processing (highlighted as fast/temporary), and only the final durable output written back to persistent storage — illustrating using ephemeral storage deliberately for regenerable intermediate data

Seeing It in Action

Scenario: A video-processing service — using ephemeral and persistent storage correctly.

Pipeline: user uploads a video → transcode into multiple resolutions →
store outputs. Runs on autoscaled, disposable containers.

Persistent (must survive):
  - Original uploaded video → written IMMEDIATELY to object storage (S3).
  - Transcoded output videos → written to S3 when done.
  - Job metadata + status    → in the database.
  → All the data that matters lives OUTSIDE the disposable container.

Ephemeral (transient, deliberately fast):
  - During transcoding, the container downloads the source to LOCAL disk
    (fast ephemeral scratch), writes intermediate frames/segments there,
    and assembles outputs — all on fast local storage, no network per-frame.
  - This scratch data is REGENERABLE: if the container dies mid-job, the
    job is simply retried on a new container from the S3 source.

What makes this safe:
  - The container is CATTLE — killed on deploy/scale/failure at any time.
  - Nothing irreplaceable lives on it: source + outputs are in S3, status
    in the DB. Losing the container loses only regenerable scratch work.
  - Idempotent job design (see idempotency.md) means a retried job on a
    fresh container produces the same result.

The mistake this avoids:
  ✗ Transcoding to local disk and serving outputs FROM the container →
    next deploy/scale-in deletes the outputs → users' videos vanish.
  ✓ Outputs go to S3 the moment they're ready; the container holds only
    transient scratch it can always recreate.

Why the distinction is load-bearing in cloud-native systems: modern infrastructure treats compute as disposable — containers and instances are created and destroyed constantly for scaling, deployments, and self-healing — and this disposability is exactly what makes horizontal scaling and resilience work. But it's only safe if instances hold no irreplaceable state. Ephemeral storage is perfect for the transcoding scratch space: it's fast (local, no network per frame), cheap, and holds only regenerable intermediate data, so losing a container costs nothing but a retry. Persistent storage (S3, the database) holds everything that must survive — the source video, the outputs, the job status — living independently of any container. The catastrophic mistakes all share one shape: putting must-keep data (uploads, database files, logs) on storage tied to a disposable instance, so a routine deploy or scale-in silently destroys it. The discipline is to ask of every piece of data, "if the instance vanished right now, would this matter?" — and route regenerable/transient data to fast ephemeral storage and irreplaceable data to durable persistent storage. Getting this right is what lets you enjoy disposable, autoscaling compute without ever losing data.

Interview Questions

Q: What is ephemeral storage and how does it differ from persistent storage? Hint: Ephemeral storage is tied to the lifecycle of a compute instance/process — when the instance stops, restarts, or is replaced, the data is gone (container local disk, instance store, function /tmp/memory, Kubernetes emptyDir). Persistent storage exists independently of compute and survives instance replacement (EBS/persistent volumes, S3, managed databases). Ephemeral is for transient/regenerable data; persistent for anything you must keep.
Q: Why is the ephemeral/persistent distinction especially critical in cloud-native systems? Hint: Cloud-native design treats instances as "cattle, not pets" — routinely killed and recreated for autoscaling, deploys, and self-healing. Any important data on an instance's ephemeral storage is lost on every such event. So apps must be stateless, keeping durable state in external persistent stores. The disposability that enables scaling and resilience only works if instances hold nothing irreplaceable.
Q: What's a good test for whether data can live on ephemeral storage? Hint: Ask: "If this data disappeared right now, could the system regenerate it or continue without harm?" If yes (scratch files, re-fetchable cache, intermediate results, build artifacts), ephemeral is fine — and often faster and cheaper. If no (user uploads, database files, anything of record), it must be persistent. The question is regenerability/replaceability, not importance of the current computation.
Q: Give examples of classic data-loss mistakes with ephemeral storage. Hint: Storing user uploads on a container's local disk (deleted on next deploy), a database writing data files to instance-store ephemeral disk (lost when the instance stops), or keeping logs only on local disk (gone when the instance is replaced). The fix in each case: write to persistent/external storage (object storage for uploads, persistent volumes or managed DB for databases, a central aggregator for logs).
Q: Is ephemeral storage always something to avoid? When is it the right choice? Hint: No — it's the right tool for transient data and is often faster than network-attached persistent volumes (local, no network hop). Use it deliberately for scratch space, spill/sort space, local caches, temp files during processing, and per-invocation working memory — regenerable data where speed and low cost matter. The goal is using ephemeral storage knowingly for the right (transient) data, not avoiding it.

References

The Twelve-Factor App: Processes (stateless) — why processes should be stateless
AWS: Instance store vs EBS — ephemeral vs persistent volumes
Kubernetes: Volumes (emptyDir vs persistent) — ephemeral vs persistent in K8s

Dive Deeper

Cattle vs pets (the origin) — the disposable-infrastructure philosophy
Kubernetes ephemeral volumes — the range of ephemeral storage options
Designing stateless services — the broader pattern that makes ephemeral compute safe