Caching

5 min read

Doing Less Work by Remembering Answers

Caching is the single highest-leverage performance technique in system design: instead of recomputing or re-fetching the same data over and over, you keep a copy somewhere fast and reuse it. The payoff is enormous because the cost of data access spans many orders of magnitude — a value in memory is thousands of times cheaper to retrieve than the same value from a database query, and millions of times cheaper than a cross-continent round trip. Nearly every fast system at scale is fast because of caching, layered from the user's browser all the way down to the database's internal buffers.

But caching is deceptively hard, and the difficulty is almost entirely about correctness over time. A cache is a second copy of data, and the moment the original changes, that copy is a lie waiting to be served. So caching is a discipline of managing staleness: deciding what to cache, how to read and write it consistently, when to invalidate it, what to evict when memory fills, and which technology to store it in. Get these right and caching is a superpower; get them wrong and you serve stale data, stampede your database, or crash on out-of-memory.

When This Comes Up

System design interviews: Once you've sketched a data flow, "how do you make this faster / handle this read volume?" almost always leads to caching. Strong candidates specify where the cache sits, which pattern keeps it consistent, how it's invalidated, and what the staleness/consistency trade-off is — not just "add a Redis." Cache invalidation and stampede handling are favorite deep-dive probes.
Real architecture: Choosing TTLs, read/write patterns, eviction policies, and Redis-vs-Memcached are everyday decisions that shape latency, database load, cost, and correctness. A well-tuned cache can cut origin load by 90%+ and turn an overwhelmed database into a comfortable one.
Production incidents: Stale data bugs, cache stampedes after a mass expiry, out-of-memory from unbounded local caches, and cold-start thundering herds after a cache flush are classic outages. The concepts here are exactly what prevent and diagnose them.

How the Sub-Topics Connect

The sub-topics move from where caches live (layers) → how to keep them consistent on reads and writes (patterns) → when to remove stale entries (invalidation) → what to remove when full (eviction) → and finally which technology to use (Redis vs Memcached):

1. Caching Layers

Caching isn't one thing in one place — it happens at many layers between the user and the origin: browser, CDN, reverse proxy, in-process application cache, distributed cache (Redis), and inside the database. Each layer answers the requests it can, so fewer fall through to the slow, expensive origin. Understanding the full stack — and the latency numbers that justify it (memory is ~100×, a DB query is millions× the cost of L1) — plus the trade-off between fast-but-divergent local caches and shared-but-networked distributed caches, is what turns caching from a bolt-on into an architecture. The metric that ties it together is cache hit ratio.

2. Cache Read/Write Patterns

Once you cache, you must coordinate the cache and the database. The read patterns — cache-aside (app manages loading) and read-through (cache manages loading) — differ in who handles a miss. The write patterns — write-through (sync to both), write-behind (async to DB), write-around (skip the cache) — differ in when writes reach the database. The pragmatic default for most systems is cache-aside reads plus write to DB then delete the cache key (invalidation, not update), guarded by a lock on miss to prevent the thundering-herd stampede.

3. Cache Invalidation

The famously hard part: deciding when a cached copy is no longer valid and removing it, balancing freshness against efficiency. Strategies range from simple TTL (bounded staleness, self-healing, often enough on its own) to explicit event-driven deletion (near-immediate freshness, must hook every write path) to stale-while-revalidate (serve stale instantly, refresh in the background — freshness without the stampede). The distributed case is hardest: many nodes holding copies must all be evicted, which argues for a shared cache tier or pub/sub invalidation events. As with consistency, invalidation is chosen per data type.

4. Eviction Policies

When a cache fills up, an eviction policy decides which item to remove, aiming to keep what's most likely to be reused. LRU (least recently used) is the safe default because temporal locality holds in most workloads; LFU (least frequently used) wins when popularity is stable and skewed and resists scan pollution; FIFO and random are simpler bets. Modern hybrids (W-TinyLFU) combine recency and frequency. Eviction (space-driven) is distinct from expiration (time-driven), and it only matters because memory is bounded — so policy choice goes hand-in-hand with sizing the cache at the knee of the hit-ratio curve.

5. Redis vs Memcached

The two classic distributed in-memory caches. Memcached is a deliberately simple, multi-threaded, strings-only cache that does one thing extremely well and scales across cores. Redis is a richer "data structures server" — lists, sets, sorted sets, hashes, streams, pub/sub, persistence, replication, and clustering — useful for far more than caching (sessions, leaderboards, rate limiting, WebSocket backplanes). For pure key-value caching they're comparable, so the decision hinges on whether you'll benefit from Redis's extras. Since most systems eventually want some of them, Redis is the pragmatic default; Memcached shines for a simple, massive, multi-core cache.

Sub-Topics

#	Sub-Topic	What You'll Learn
1	Caching Layers	Where caches live, from browser to database, and hit ratio
2	Cache Read/Write Patterns	Keeping cache and database consistent on reads and writes
3	Cache Invalidation	Removing stale entries while balancing freshness and efficiency
4	Eviction Policies	Choosing what to discard when the cache fills up
5	Redis vs Memcached	Picking the right in-memory cache technology