Caching Strategies

Tier 1 — Building Block

1. What Is It?

A cache is a fast, in-memory data store that sits between your application and a slower, durable data store (database, file system, external API). Its purpose is simple: serve frequently accessed data from memory instead of paying the latency and I/O cost of hitting the source of truth on every request.

Without caching, even a moderately loaded service can overwhelm a relational database. A single PostgreSQL node typically handles ~10K–30K simple queries per second. A Redis node handles ~100K–200K ops/sec. Caching bridges that gap by absorbing read traffic before it reaches the database, dramatically reducing latency (sub-millisecond from cache vs. 5–50ms from DB) and protecting the database from overload.

The "strategy" is the policy that defines when cache entries are written, when they are invalidated, and how the application interacts with both the cache and the database.

2. How It Works

The three core write strategies differ in when and who writes to the database:

Cache-Aside (Look-Aside / Lazy Loading)

The application code manually manages the cache. On a read:

1. Check cache — if hit, return data.
2. If miss, read from database.
3. Write result to cache with a TTL (Time-To-Live) — an expiration time after which the cached entry is automatically deleted.
4. Return data to caller.

On a write:

1. Write directly to the database.
2. Invalidate (delete) the cache entry.

Read Path:
  App → Cache (miss) → DB → App → Cache (write + TTL)

Write Path:
  App → DB → Invalidate Cache Key

-+ResetFullscreen

Write-Through

Every write goes through the cache — the application writes to the cache, and the cache synchronously writes to the database before acknowledging success.

Write Path:
  App → Cache → DB (synchronous) → ACK

-+ResetFullscreen

Write-Back (Write-Behind)

The application writes to the cache, and the cache asynchronously flushes dirty entries to the database later (batched). High write throughput, but data loss risk if cache crashes before flush.

Write Path:
  App → Cache (immediate ACK) → [async batch] → DB

-+ResetFullscreen

3. Variants & Comparisons

Specific Technologies:

• Redis: In-memory, supports strings/hashes/sets/sorted sets/streams. ~100K–200K ops/sec per node. Supports replication, clustering, persistence (RDB snapshots + AOF). Use for session data, leaderboards, pub/sub, distributed locks.
• Memcached: Pure key-value, no persistence, multi-threaded. ~200K–500K simple gets/sec. Simpler than Redis — use when you need raw throughput for simple objects and don't need Redis data structures.
• Caffeine (Java in-process): JVM-local LRU/LFU cache. Zero network latency. Use for per-instance local caches (e.g., config, small lookup tables). Not shared across instances.
• Varnish: HTTP reverse proxy cache. Layer 7. Caches full HTTP responses. Use for CDN-like behavior at the edge without a full CDN.

4. When to Use It (and When NOT To)

Use Caching When:

• Read-to-write ratio > 5:1 — cache hit rate will be high enough to justify the complexity
• Data has temporal locality — recently accessed data is likely to be accessed again soon
• DB is the bottleneck — measured DB CPU/IO at >60% utilization under load
• Acceptable staleness window exists — even 1–5 minutes of stale data is OK (profile pages, product listings)
• Computation is expensive — caching aggregation query results, rendered templates, ML model outputs

Decision triggers:

• "If QPS > 50% of your DB's max throughput → add read cache (cache-aside)"
• "If P99 latency is DB-bound (>50ms) → cache the hot query set"
• "If you have > 10K DAU with repeated reads of the same objects → cache-aside with 5-min TTL"

Do NOT Use Caching When:

• Every request is unique — cache hit rate ~0%; you're just adding latency overhead
• Strong consistency is required — financial balances, inventory counts during checkout (stale cache = oversell)
• Write-heavy with no repeated reads — you'll thrash the cache constantly
• Data changes faster than TTL — cache is always stale; misleading at best

Anti-patterns:

• Cache stampede / Thundering herd: TTL expires for a popular key; 10K concurrent requests all miss simultaneously and hammer the DB. Fix: probabilistic early expiration, mutex lock on populate, or background refresh.
• Cache poisoning: Writing bad data to cache (e.g., null from a downstream API). Fix: never cache null/errors, or cache with very short TTL.
• Over-caching: Caching everything including writes leads to memory pressure and stale data everywhere.
• No TTL: Cache entries grow forever and consume all memory. Always set TTL.

5. Real-World Usage

Facebook (Memcached at scale): Facebook's "Scaling Memcache at Facebook" paper (2013) describes serving trillions of reads/day from Memcached clusters. They use cache-aside, with invalidation messages sent via McSqueal (a daemon that tails MySQL binlogs and sends delete notifications). At peak, they served >1 billion requests/sec across all Memcached pools. The key insight: they tolerate eventual consistency in exchange for extreme throughput.

Twitter (Redis for timelines): Twitter pre-computes user timelines (fan-out on write — when a user tweets, the system pushes that tweet into the timeline cache of every follower) and stores them in Redis sorted sets, keyed by user ID with tweet IDs as members. For users following >10K accounts (celebrities), they switch to fan-out on read (assembling the timeline from the DB at query time, instead of pre-computing it) because pre-computing for 100M followers per tweet is prohibitively expensive. This is a real-world example of choosing cache strategy based on data shape and scale.

Amazon DynamoDB DAX: DynamoDB Accelerator (DAX) is a write-through cache for DynamoDB. Every write to DynamoDB goes through DAX first. Reads that hit DAX return in ~microseconds vs. ~milliseconds from DynamoDB. Amazon uses this for product catalog reads where consistency matters but writes are infrequent relative to reads.

6. Interview Cheat Sheet

Key sentences to demonstrate depth:

1. "Cache-aside is my default choice because the application controls what gets cached — only read data that was actually needed gets populated, which avoids wasting cache memory on cold data."
2. "Write-through trades write latency for read consistency — every write pays the two-write cost (cache + DB), so I'd only use it when read-after-write correctness outweighs the latency penalty."
3. "Write-back maximizes write throughput by making DB writes async and batched, but the dirty cache window means you can lose writes on crash — acceptable for metrics and counters, not for financial data."
4. "Cache stampede is the most common cache failure mode: a popular TTL expires, N concurrent requests all miss, all hit the DB simultaneously. The fix is probabilistic TTL jitter or a cache populate mutex."
5. "Cache invalidation is one of the two hard problems in computer science (along with naming). My rule: invalidate on write (DEL the key), don't update — it's simpler and avoids race conditions between update and concurrent reads."

Common follow-up questions:

Connections to other building blocks:

• Consistent Hashing: When you have a Redis cluster, consistent hashing determines which node holds each key. Essential for avoiding rehashing all keys when adding/removing nodes.
• Read Replicas: Caching and read replicas are complementary strategies — cache handles repeated reads of the same objects; read replicas scale the database for diverse query patterns.
• CDN: A CDN is essentially a globally distributed cache for HTTP responses. Same cache-aside semantics: miss → origin pull → store at edge → serve from edge on subsequent requests.
• CAP Theorem: Caches are inherently AP (available + partition tolerant) — they sacrifice consistency (stale data) to serve requests fast even when the DB is slow/down.