Caching
Caching is what you reach for when reading from the database is too slow or too expensive. It keeps frequently accessed data in a faster layer so most reads never touch the database — at the cost of new problems around staleness and failure.
0What Caching Is & Why It's Worth It
In interviews, caching comes up almost every time you handle high read traffic. The database becomes the bottleneck, latency creeps up, and the interviewer is waiting for you to say the word: cache.
A cache is temporary storage that keeps recently used data handy and close by so you can fetch it faster the next time. Systems avoid reaching all the way back to the slower source on every request.
The whole idea rests on a speed gap between where data lives and where it can be cached. The article and the video measure that gap through two different lenses — both are worth keeping in mind.
| Comparison | Slow path | Fast path | Speedup |
|---|---|---|---|
| End-to-end query (article's example) | User profile from Postgres ≈ 50 ms | Same data from Redis ≈ 1 ms | ~50× |
| Raw storage access (video's example) | Disk / SSD ≈ 1 ms | Memory / RAM ≈ 100 ns | ~10,000× |
These aren't contradictory — they measure different things. The 50× figure compares a full database query (which pays for disk access plus query processing) against a cache lookup. The 10,000× figure isolates the raw hardware gap between disk and RAM. Databases store data on disk and pay disk cost on every query; memory sits far closer to the CPU. Either framing supports the same conclusion: reads from memory are dramatically cheaper.
Caching trades a bit of storage and complexity for speed. It reduces load on the database and cuts latency — but it creates new challenges around invalidation and failure handling that the rest of these notes are really about.
1Where to Cache
Most engineers picture Redis or Memcached sitting between the app and the database — and that is the type interviewers care about most. But caching lives in multiple layers: browsers cache, CDNs cache, applications cache, even databases have built-in caching. There are four places worth knowing, each optimizing something different.
1.1 External Caching
An external cache is a standalone cache service your application talks to over the network — e.g. Redis or Memcached. It runs on its own server, manages its own memory, and is a separate component from your app and database.
When the app needs data it checks the cache first. Found → cache hit, returned instantly. Not found → cache miss, so it falls back to the database, fetches the data, stores a copy back in the cache, and returns it.
In a scaled system with many application servers, they all share the same external cache. Once one server fetches and caches a value, every other server reuses it instantly instead of hitting the database separately. That shared, global view is exactly what an in-process cache can't give you.
External caches scale well, support eviction policies like LRU, and support expiration via TTL so memory stays controlled.
External caching with Redis is the default answer for caching in interviews. Start here for any high-traffic system, then layer on CDN or client-side caching only if the problem calls for it.
1.2 CDN — Content Delivery Network
A CDN is a geographically distributed network of servers that caches content close to users. Instead of every request traveling to your origin, edge servers around the world hold copies. Here you're not optimizing the memory-vs-disk gap — you're optimizing network latency.
- A user requests an image from your app.
- The request goes to the nearest CDN edge server.
- If the image is cached there → returned immediately (hit).
- If not (miss) → the CDN fetches from your origin (e.g. S3), stores it, and returns it.
- Future users in that region get it instantly from the edge.
Origin in Virginia, user in India: each request adds 250–300 ms of latency. With a CDN, the same image is served from a nearby edge in 20–40 ms. (In the video Evan uses an Australia example at 300–350 ms — same point, different geography.)
Modern CDNs (Cloudflare, Fastly, Akamai) can cache far more than static files — public API responses, HTML pages, even edge logic to personalize content or enforce security before requests reach you.
The safest trigger is serving static media at scale (images, video, static assets) to global users. Lead with that reason, then expand to dynamic/API caching only if the problem calls for it.
1.3 Client-Side Caching
Client-side caching stores data close to the requester to avoid unnecessary network calls — usually the user's device: a browser (HTTP cache, localStorage) or a mobile app using local memory or on-device storage.
"Client-side" isn't only end-user devices. A Redis client caches cluster metadata — which nodes exist and which slots they own — so it can route requests directly to the right node without querying the cluster on every operation.
From the backend you have limited control: data can go stale and invalidation is harder. Examples: the Strava app keeps run data on the device while offline and syncs later; a browser reusing a previously downloaded image from disk is also caching.
This is the least important layer for system design interviews. It's usually only relevant when the system involves offline functionality or client-heavy workloads.
1.4 In-Process Caching
Most candidates overlook that servers run on machines with a lot of memory — and each hardware generation ships more of it. You can cache data directly inside the application process instead of always calling out to Redis or the database.
Reads from local process memory are even faster than Redis because they avoid the network hop entirely — the data already sits in the same memory space as your application.
It fits small, frequently requested data such as:
- Configuration values & feature flags
- Small reference datasets / lookup tables
- Hot keys
- Rate-limiting counters
- Precomputed values
Each application instance has its own in-process cache — cached data is not shared across servers. If one instance updates or invalidates a value, the others won't know, so you can get inconsistencies or wasted memory.
Mention in-process caching only as an optimization layer, after you've already introduced an external cache — for small values that rarely change or for ultra-low-latency needs (e.g. config every request depends on). Your default stays external caching.
2Cache Architectures
A cache architecture defines how your application interacts with the cache — specifically the order in which reads and writes flow between application, cache, and database. Four patterns are worth knowing.
2.1 Cache-Aside (Lazy Loading)
The most common pattern, and the one to default to in interviews.
- Application checks the cache.
- If the data is there, return it (hit).
- If not (miss), fetch from the database, store it in the cache, then return it.
Cache-aside keeps the cache lean — you only cache what users actually request. The downside: a cache miss adds latency (the request must hit the DB, then populate the cache).
If you remember only one caching pattern for interviews, make it cache-aside. It's the one you'll almost always use.
2.2 Write-Through Caching
The application writes only to the cache. The cache then synchronously writes to the database before returning. The write isn't complete until both cache and database are updated.
Redis and Memcached don't natively support write-through. Either you handle writing to both stores yourself in application code (with all its complications), or you use a library/framework that triggers the DB write for you — e.g. Spring Cache or Hazelcast.
Trade-offs: writes are slower (you wait for both operations), and the cache can be polluted with data that's never read again.
If the cache update succeeds but the DB write fails (or vice versa), the two enter an inconsistent state. You need retry logic and error handling — and in a distributed system perfect consistency is very hard without distributed transactions.
Use write-through when reads must always return fresh data and the system can tolerate slightly slower writes. But first convince yourself cache-aside doesn't already satisfy the use case — it usually does.
2.3 Write-Behind (Write-Back) Caching
Like write-through, the application writes only to the cache — but the cache flushes to the database asynchronously in the background, usually in batches.
This makes writes very fast, but introduces risk: if the cache crashes before flushing, you lose data.
Use when you need high write throughput and eventual consistency is acceptable — common in analytics and metrics pipelines where occasional data loss is tolerable.
If you're an expert with a real use case, use it. But for a novice, Evan suggests you probably avoid it in interviews — other patterns solve the same problems, and it opens more follow-up questions than you'd like unless you can strongly justify it.
2.4 Read-Through Caching
The cache acts as a smart proxy. Your application never talks to the database directly. On a miss, the cache itself fetches from the DB, stores the data, and returns it. It's essentially cache-aside with the cache doing the lookup instead of the application.
A CDN is a form of read-through cache: on a miss it fetches from your origin, caches the result, and serves it next time. For application-level caching with Redis, cache-aside is far more common because it needs no special adapter.
There are few reasons to propose read-through in interviews unless you're discussing CDNs or edge caching. Otherwise cache-aside wins on simplicity.
Interviewers don't care whether you recall the exact term. What matters is describing the behavior clearly. If "cache-aside" slips your mind, just say: "I'll check the cache first, and if it's not there I'll go to the database and then update the cache." Understanding beats memorization.
| Pattern | Who reads/writes the DB | Best for | Interview use |
|---|---|---|---|
| Cache-aside | Application | General read-heavy workloads | Default — always |
| Write-through | Cache (sync, on write) | Reads must be fresh; slower writes OK | Rare; needs library |
| Write-behind | Cache (async, batched) | High write throughput; loss tolerable | Usually avoid |
| Read-through | Cache (on miss) | CDN / edge caching | Only for CDNs |
3Eviction Policies
Memory is limited, so a cache needs a strategy for what to remove when it fills up. Three policies you might actually use, plus one that's just good to know for its simplicity.
| Policy | Evicts… | Good when |
|---|---|---|
| LRU Least Recently Used | The item unused for the longest time (tracked via linked list / ring buffer, O(1) removal). | Recently used data is likely reused — the common default. |
| LFU Least Frequently Used | The item accessed the fewest times (per-key counter; sometimes approximate). | A few keys are consistently popular over time (trending videos, top playlists). Access is highly skewed. |
| FIFO First In First Out | The oldest item by insertion time (simple queue; ignores usage). | Rarely the right choice — may evict items that are still hot. |
| TTL Time To Live | Any key older than its expiration (not a standalone eviction policy — combined with LRU/LFU). | Freshness matters: API responses, session tokens. |
TTL sets an expiration per key; it doesn't decide which entry to drop when memory is full. It's typically layered on top of LRU or LFU to balance freshness against memory usage.
Implementation detail of eviction (the exact linked list vs. priority queue) is almost always out of scope in a system design interview. Know what each policy does and when to pick it; don't rathole on internals.
4Common Caching Problems
Caching makes systems faster but introduces new failure modes — the ones interviewers love to probe. If you bring up caching, show you can handle these edge cases, not just the benefits.
There are only two hard problems in computer science: naming things and cache invalidation.
4.1 Cache Stampede (Thundering Herd)
A cache stampede happens when a popular entry expires and many requests try to rebuild it simultaneously. For a brief window — even a second — every request misses and goes straight to the database. One query becomes hundreds or thousands, which can overload the DB.
You cache the homepage feed with a 60s TTL and serve ~100,000 requests/second happily from cache. At the instant it expires (say 12:01:00), all 100,000 requests miss at once and hit the database together — potentially taking it down and causing cascading failures.
Two common defenses:
Request coalescing (single flight). When multiple requests try to rebuild the same key, let only the first do the work; the rest wait and then read from the cache. The most effective solution.
Cache warming. Proactively refresh popular keys just before they expire — e.g. re-fetch at the 55s mark on a 60s TTL, so the key effectively never expires.
Cache warming works when keys expire on a TTL. If you instead invalidate the cache on writes, warming does not prevent stampedes.
4.2 Cache Consistency
The most commonly discussed caching issue in interviews. It occurs when the cache and database return different values for the same data — common because most systems read from the cache but write to the database first, creating a window where the cache holds stale data.
A user updates their profile picture. The new value is written to the database, but the old one still sits in the cache. Other users keep reading the outdated image from the cache until that entry is eventually evicted or refreshed.
You choose a strategy based on how fresh the data must be — a case-by-case decision, not a universal answer.
Invalidate on write. After updating the database, delete the cache key so the next read repopulates it with fresh data.
Short TTLs. Accept brief staleness — keep the entry but expire it quickly (e.g. the 60s feed).
Accept eventual consistency. For feeds, metrics, and analytics, a short delay is usually fine.
Framing the trade-off explicitly scores points: "We have a 5-minute TTL on profile data. That means some users see a stale profile image for up to 5 minutes — and I'm okay with that, because it's not the end of the world." A defensible justification tied to the design is exactly what interviewers want.
4.3 Hot Keys
A hot key is a cache entry receiving a huge amount of traffic relative to everything else. Even with a great overall hit rate, one key can overload a single cache node or Redis shard and become a bottleneck.
Building Twitter/X, the key user:taylorswift may take millions of requests per second. That one key can overload a single Redis node even though the cache is technically "working correctly."
Replicate hot keys. Store the same value on multiple cache nodes/shards and load-balance reads across them.
Add a local fallback cache. Keep extremely hot values in-process so repeated requests never even reach Redis.
Apply rate limiting. Slow down abusive traffic patterns on specific keys.
Hot keys aren't unique to caches — databases have hot rows and partitions too. But it's a common follow-up once you introduce caching to scale reads: you're now hitting memory instead of disk, yet a single overwhelmingly popular key can still bottleneck you. Interviewers love to probe this, so be ready. (Replication here assumes sharding/clustering — splitting cached data across multiple cache instances.)
5Caching in System Design Interviews
5.1 When to Bring Up Caching
Don't throw down a cache with no justification. Even if you happen to be right, the lack of reasoning is a red flag. Establish why it's necessary first.
Bring up caching when you can point to one of these four problems, quantified with rough numbers:
| Trigger | How you'd phrase it |
|---|---|
| Read-heavy workload | "10M DAU × 20 requests/day = 200M reads. Even with indexes that's 20–50 ms/query; a cache drops it under 2 ms and offloads the DB." |
| Expensive queries | "The personalized feed joins posts, followers, and likes across tables — 200 ms. Cache the computed feed for 60 s and serve it in 1 ms from Redis." |
| High database CPU | "DB CPU hits 80% at peak just serving the same reads repeatedly; caching hot queries cuts load 70–80%." (Real-world signal — you won't have this metric in an interview.) |
| Latency requirement | "We need sub-10 ms responses but DB queries take 30–50 ms — we have to cache." Ties directly to a non-functional requirement. |
Identify the bottleneck → quantify it with rough numbers → explain how caching solves it.
5.2 How to Introduce Caching — 5 Steps
Walk through these in order and you'll sound like a caching pro.
- Identify the bottleneck. Be specific about what's slow and why. "User profile queries hit the DB 500×/sec at peak, 30 ms each — that's our bottleneck."
- Decide what to cache. Only data that is read frequently, changes rarely, and is expensive to fetch/compute. Be explicit about cache keys — e.g.
user:123:profile,trending:posts:global. - Choose the architecture. Match the pattern to your consistency needs. "Cache-aside: check Redis, on miss query the DB, store in Redis, return." Mention CDN for static media, in-process for extremely hot keys.
- Set an eviction policy. LRU is the safe default; TTL prevents stale data. "LRU with a 10-minute TTL on profiles, and invalidate the entry immediately when a user edits."
- Address the downsides. Pick the one or two relevant to your system: invalidation strategy, cache failure fallback, thundering herd.
Especially in junior/mid-level interviews, candidates say "I'll add a cache here" and stop. The reliable follow-up is: "What exactly are you caching? What's the key? What values live in it?" Answer that proactively.
Have an answer for "what if Redis goes down?": requests fall back to the database, with circuit breakers to avoid stampeding it, and possibly a small in-process last-resort cache. And note that caching usually comes up during deep dives when discussing scale or latency — that's the natural moment to introduce it. For staff-level candidates, focus on non-obvious scenarios rather than the obvious ones the interviewer already assumes.
★Active Recall
Answer each from memory before expanding. These test reconstruction, not trivia.
Q1Why is reading from a cache so much faster than from a database — and what are the two different speed lenses used to show it?
Databases store data on disk and pay disk-access cost on every query; memory sits far closer to the CPU. The article frames it end-to-end (Postgres query ~50 ms vs Redis ~1 ms → ~50×). The video frames the raw hardware gap (disk ~1 ms vs RAM ~100 ns → ~10,000×). Different measurements, same conclusion.
Q2Reconstruct the cache-aside read path from scratch, and state its one main downside.
Check the cache → if present, return it (hit) → if absent (miss), fetch from the database, store the result in the cache, then return it. It keeps the cache lean (only requested data is cached), but a cache miss adds latency because the request must hit the DB and then populate the cache.
Q3How do write-through and write-behind differ, and what risk does each carry?
Both write only to the cache. Write-through updates the DB synchronously before acknowledging — slower writes, plus the dual-write problem (cache and DB can diverge if one fails). Write-behind flushes to the DB asynchronously in batches — fast writes, but data loss if the cache crashes before flushing.
Q4Why is TTL not really an eviction policy, and what is it combined with?
TTL only sets an expiration time per key; it doesn't decide which entry to drop when memory fills up. It's layered on top of a real eviction policy like LRU or LFU to balance freshness against memory usage.
Q5A key with a 60s TTL expires and 100k requests arrive that second. What is this, and how do you prevent it?
A cache stampede / thundering herd — all requests miss and hit the DB together, risking cascading failure. Prevent with request coalescing (only the first request rebuilds the key; the rest wait) or cache warming (refresh popular keys just before expiry). Note: warming only helps with TTL expiry, not invalidate-on-write.
Q6Why does cache consistency break so easily, and name three ways to manage it.
Most systems read from the cache but write to the database first, leaving a window of stale cache data. Manage it by (1) invalidating the key on write, (2) using short TTLs to bound staleness, or (3) accepting eventual consistency where a brief delay is fine (feeds, metrics). There's no perfect fix — you choose based on required freshness.
Q7The overall cache hit rate is excellent, yet one Redis node is melting. What's happening and what are the fixes?
A hot key — one entry (e.g. user:taylorswift) takes a disproportionate share of traffic and overloads a single node/shard. Fixes: replicate the hot key across nodes and load-balance reads, add a local in-process fallback cache, or rate-limit the key. Caching scales reads but doesn't make a single key infinitely scalable.
Q8Give the five-step script for introducing caching in an interview.
(1) Identify the bottleneck, (2) decide what to cache and name the cache keys, (3) choose the architecture (default cache-aside), (4) set an eviction policy (LRU default + TTL), (5) address the relevant downsides (invalidation, cache failure, thundering herd). Precede all of it by justifying why caching is needed with quantified numbers.
↺Quick Revision
Fast refresh — for a return visit once you've read the full notes.
Must Remember
- Default = external cache (Redis) + cache-aside.
- Justify the cache before drawing it.
- Describe behavior; don't memorize pattern names.
- Don't cache everything.
Where to Cache
- External (Redis) — default, shared across servers.
- CDN — static media to global users; network latency.
- Client-side — offline/client-heavy; least interview weight.
- In-process — fastest, but per-instance; optimization layer only.
Architectures
- Cache-aside — app orchestrates; the default.
- Write-through — sync DB write; fresh reads, slow writes.
- Write-behind — async DB write; fast, risky; usually avoid.
- Read-through — cache as proxy; think CDNs.
Eviction
- LRU — recency; the default.
- LFU — frequency; skewed access.
- FIFO — insertion order; rarely right.
- TTL — expiry, not eviction; layer on LRU/LFU.
Problems & Fixes
- Stampede → request coalescing / cache warming.
- Consistency → invalidate-on-write / short TTL / accept eventual.
- Hot keys → replicate / local fallback / rate-limit.
Interview Script
- 1. Identify bottleneck.
- 2. Decide what to cache + keys.
- 3. Choose architecture.
- 4. Set eviction policy.
- 5. Address downsides.