What it is
Caching means serving a read from a fast, bounded copy of data instead of recomputing or refetching it from the system of record, accepting that the copy may occasionally be stale or absent. In a system design interview this is rarely "add a cache" as an afterthought — it's a specific decision about what to precompute versus compute on read, how long an entry stays valid, what happens to callers on a miss, and how the cache is kept from serving something both wrong and expensive to detect. The interesting design work is usually not the cache itself but the pipeline that keeps it warm: what triggers a write into it, and what a reader does when the entry isn't there.
When it applies
- Reads vastly outnumber writes and the same values are requested repeatedly, so a small hot set absorbs most of the traffic.
- A read would otherwise require an expensive fan-in (merging many sources, running a ranking model, walking a large index) that's cheaper to do once and keep than to redo on every request.
- Bounded staleness is acceptable — the product can tolerate an entry that's a few seconds or minutes old in exchange for consistently low latency.
- If every read must reflect the absolute latest write (financial balances, inventory counts at the moment of purchase) a cache without a correctness story is the wrong default — you need explicit invalidation or read-your-writes guarantees, not just a TTL and a shrug.
Tells in the problem
- Explicit read:write ratio given in the prompt ("20 reads per write," "read-heavy") as a hint that a cache tier is expected.
- "Losing an entry is acceptable" or "serving a stale entry for a bounded window is fine" — the interviewer stating the staleness budget you're allowed to design against.
- A precomputed per-user or per-key structure that's rebuilt or appended to on some upstream event, then read directly rather than recomputed per request (a feed list, a suggestion list).
- A latency target tight enough that a database or ranking call on every request can't meet it (sub-millisecond or low-tens-of-milliseconds p99).
- A skewed access distribution mentioned explicitly (a small number of keys or prefixes account for most traffic) — a signal that a cache sized well below the full dataset still captures most of the load.
The classic mistake
Treating the cache as if it were the source of truth once it's warm, and forgetting to design what happens on a miss or after an invalidation — a design that only describes the cache-hit path leaves the interviewer to ask "what happens on a cold cache, or when this entry expires, or when the underlying data changes before the TTL is up," and a candidate without an answer has actually only designed half the system. The fix is to always narrate the miss path (recompute from the system of record, populate the cache, return) and the invalidation path (what event or TTL removes or refreshes an entry) as part of the same design as the hit path, not as an afterthought once asked.
Questions to start with
- Design a Distributed Cache — the cache as the whole product: TTLs, eviction, and what consistency it promises (and doesn't) across replicas.
- Design a News Feed — a precomputed per-user cache populated by a fan-out pipeline, read directly instead of recomputed per request.
- Design Search Autocomplete — a small cache of the most popular prefixes sitting in front of a much larger precomputed index, refreshed on a fixed schedule rather than per write.