A Simple Question: Why Read the Same Data Over and Over?
In the previous lesson, we analyzed how grueling database scaling can be. We deployed read replicas, separated read/write workloads, partitioned tables, and examined sharding clusters.
Now, ask yourself a fundamental engineering question:
“Suppose your e-commerce platform lists a popular pair of running shoes. Every second, 10,000 active shoppers open that product page to view its name, image, and price. The price has not changed in three days. Does it make engineering sense to execute 10,000 SQL queries per second against persistent disk storage just to fetch that identical information?”
Certainly not. No matter how powerful a database server is, reading from non-volatile storage wastes valuable compute cycles and network bandwidth.
If we can store that product information in an ultra-fast temporary storage location after the first request, the subsequent 9,999 users can retrieve it in sub-millisecond time. Database load drops by 99.9%!
This elegant yet revolutionary architectural pattern is called Caching.
What is Cache and Why is it Blazing Fast?
Formally defined:
The Physical Reality of Memory Hierarchies
Why does caching deliver such dramatic performance gains? It exploits the physics of modern computer hardware:
RAM access operates orders of magnitude faster than non-volatile disk storage. While a relational SQL engine might spend 50ms evaluating a complex join and pulling pages off an SSD, an in-memory Redis cluster returns the same cached payload in 1 to 2 milliseconds.
The Multi-Tier Caching Topology
In modern web architectures, caching is not confined to a single box. Instead, requests traverse a defensive multi-layered caching pipeline:
Cache-Control: max-age=86400, ETag), eliminating network requests completely for repeat visits.
Redis vs. Memcached: The In-Memory Powerhouses
When architecting application-tier caching, engineering teams invariably choose between Redis and Memcached:
| Feature | Redis | Memcached |
|---|---|---|
| Data Structures | Strings, Lists, Hashes, Sets, Sorted Sets, Bitmaps, Streams | Pure Key-Value Strings only |
| Persistence (Disk Durability) | Yes (RDB periodic snapshots and AOF append logs) | No (Pure volatile memory; restart deletes all data) |
| Threading Model | Single-threaded event loop (zero locking overhead) | Multi-threaded (scales across multiple CPU cores) |
| Extended Capabilities | Pub/Sub messaging, Rate limiting, Distributed locking (Redlock), Leaderboards | Strictly simple, lightweight caching |
In modern production architectures, Redis is the default standard for over 90% of systems because its rich native data structures and snapshotting support make it far more than just a cache.
Cache Hit, Cache Miss, and Hit Rate Mechanics
Three core telemetry metrics measure cache health:
- Cache Hit: The application queries the cache and finds the requested key. No database call is made. Latency: ~1ms.
- Cache Miss: The requested key is absent from cache (expired or never loaded). The application falls back to querying the database. Latency: ~50ms.
- Cache Hit Rate: The percentage of all queries successfully resolved by the cache.
The Multiplier Effect of Cache Hit Rates
Consider an application receiving 10,000 queries per second (QPS):
- At a 0% Hit Rate (no cache): All 10,000 QPS crash into the database.
- At an 80% Hit Rate: 8,000 QPS return from cache; 2,000 reach the database (5x load reduction).
- At a 95% Hit Rate: 9,500 QPS return from cache; only 500 reach the database (20x load reduction!).
- At a 99% Hit Rate: Only 100 QPS reach the database (100x load reduction!).
High-scale production systems typically target sustained cache hit rates between 85% and 99%.
Caching Architectural Patterns
How data flows between the application, cache, and database is governed by three primary architectural patterns:
The application coordinates reads and writes directly. On a read, it queries the cache first. If a Cache Miss occurs, the app reads the database, writes the resulting payload to the cache with a TTL, and returns the response.
2. Cache Hit? → Return immediately (1ms)
3. Cache Miss? → Fetch from DB → Save in Cache with TTL → Return to User
Pros: Only actively requested data is cached (memory efficient). If the cache crashes, the application falls back gracefully to the database.
When application data updates, writes pass through the cache. The cache engine synchronously writes to the database before acknowledging success to the caller.
Pros: Data in the cache is never stale; cache and database remain strictly synchronized.
Cons: Every write incurs double latency because it writes to both storage layers synchronously.
The application writes exclusively to the in-memory cache and returns immediately. The cache asynchronously flushes batched writes to the database in the background.
Pros: Exceptional write throughput (ideal for high-frequency video view counters or likes).
Severe Risk: If the cache crashes before flushing to disk, unwritten transactions are lost permanently.
Stale Data & The Hard Art of Cache Invalidation
Computer science pioneer Phil Karlton famously declared:
“There are only two hard things in Computer Science: cache invalidation and naming things.”
Why is invalidation so difficult?
Suppose an e-commerce item price is cached as $50. A merchant adjusts the price in the database to $40.
Because the cache is decoupled, it continues serving the outdated $50 value to buyers. This outdated state is termed Stale Data, leading to severe discrepancies at checkout.
Two Core Weapons for Freshness
EXPIRE product:101 300 for 5 minutes). After 300 seconds, Redis automatically evicts the entry. Even if an update goes unnotified, data will remain stale for at most 5 minutes.
redis.del("product:101"). The subsequent read request triggers a clean Cache Miss and fetches fresh data.
The Cache Stampede (Thundering Herd) Catastrophe
Consider a viral live event (e.g., World Cup scores or a Black Friday flash sale item) cached in Redis. The key experiences 10,000 QPS with a 99.9% hit rate. The database is idling comfortably.
Suddenly, its 5-minute TTL expires, and the key vanishes from the cache!
2. In that exact millisecond, 5,000 incoming requests check the cache → All trigger a Cache Miss!
3. All 5,000 application threads simultaneously query the database: "SELECT * FROM live_scores"
4. The database connection pool is instantly overwhelmed! CPU surges to 100%! Database crashes! 💥
This cascading outage is termed a Cache Stampede or Thundering Herd problem.
The Solution: Mutex Locks (Singleflight)
When a cache miss occurs, the first arriving request acquires a temporary distributed lock (e.g., redis.setnx("lock:score", 1, "EX", 5)). The other 4,999 requests detect the lock and pause for 100ms. The single winner queries the database, repopulates the cache, and releases the lock. The remaining 4,999 requests then read directly from memory. Only one single query touches the database!
Cache Eviction: What Happens When Memory Fills Up?
RAM is bounded (e.g., 32GB or 64GB). When Redis reaches its configured maxmemory threshold, it must discard existing keys to accommodate new incoming writes. This selection is governed by Eviction Policies:
What to Cache and What NEVER to Cache
- Static or Slow-Changing Catalogs: Product details, country lists, and app configurations.
- Computationally Expensive Queries: Aggregated reporting metrics joining multiple tables.
- User Sessions & Permissions: Auth tokens and RBAC capability lists.
- Viral Trending Items: Breaking news articles and leaderboard scores.
- Transactional Balances: Bank accounts and digital wallets where absolute consistency is non-negotiable.
- Hyper-Volatile Data: Millisecond stock market tickers where cache becomes stale instantly.
- Single-Use Tokens: One-time password (OTP) verification secrets.
- Huge Binary Blobs: High-definition video or large PDFs (use S3/Cloudflare R2, not Redis RAM).
Interactive Caching Workbench
Simulate Cache-Aside throughput, thundering herd crises, and LRU eviction live
Caching & Performance Quiz
Verify your understanding of Lesson 11.6 core principles
1. Under the standard Cache-Aside pattern, what action must the application take immediately upon experiencing a "Cache Miss"?
2. What is the most effective engineering defense against a Cache Stampede (Thundering Herd) when a hot key expires?
3. Under a Least Recently Used (LRU) cache eviction policy, which entry is removed when memory reaches full capacity?
Asynchronous Systems: Message Queues, Background Jobs & Event-Driven Architecture
Why force a customer to wait 4 seconds for email confirmation, invoice generation, and analytics processing during checkout? RabbitMQ, Apache Kafka, decoupled background workers, and asynchronous architectures kick off Part 3!