Lesson 11.14 — Designing a Social Media Feed: How to Design a Scalable News Feed (Facebook / X) — System Design Made Simple

1. What is a Social Media Feed and Why is it Architecturally Challenging?

In the previous lesson, we designed a Real-Time Chat System. There, the core challenge was routing: "How does a message get delivered directly and reliably to a specific recipient's socket?" Chat relationships are fundamentally One-to-One or small group (One-to-Few) connections.

A social media platform (such as Facebook, X/Twitter, Instagram, or LinkedIn) presents a completely different engineering challenge. The moment a user launches the mobile app, the system must assemble and display an engaging News Feed or Home Timeline. Conceptually:

A Social Media Feed is a continuously curated, personalized, and prioritized stream of posts, media, and social signals tailored specifically to each individual user.

Conceptual Social Feed Flow

User Opens App (Alice)
       │
       ▼
 ┌─────────── Feed Service ───────────┐
 │ • Posts from accounts Alice follows│
 │ • Viral & recommended content      │
 │ • Sponsored advertisements         │
 │ • Algorithmic relevance ranking    │
 └─────────────────┬──────────────────┘
                   │
                   ▼
  Alice's Personalized Feed:
  [ Post A: Bob's new photo (2 mins ago) ]
  [ Post B: Tech News (10 mins ago · High engagement) ]
  [ Post C: Sponsored Advertisement ]
  [ Post D: University Group Discussion ]

On the surface, this might look deceptively simple: query the database for recent posts and render them on screen. However, when the platform scales to 500 Million Daily Active Users (DAU) and generates over 1 Billion new posts per day, a formidable engineering paradox emerges:

Asymmetric Traffic: A single post is created once (Write), but that post may be viewed by ten people or one hundred million people (Massive Reads). Social feeds are severely Read-Heavy (typically a 100:1 to 1,000:1 Read:Write ratio).
Sub-100ms Latency Expectation: Users swipe continuously. If feed generation takes 2 seconds, engagement plummets and users abandon the platform.
Hyper-Personalization: No two users on Earth share the exact same feed. Each feed is computed from unique social graph connections, past interaction affinity, and personalized interest models.

2. System Requirements Specification

Before designing the architecture, let us establish concrete functional and non-functional requirements:

Functional Requirements

• Create Post: Users can publish new posts containing text, images, and videos.
• Follow Graph: Users can follow and unfollow other accounts.
• View Feed: Users can view a personalized timeline of posts from followed accounts in reverse-chronological or ranked order.
• Infinite Scroll (Pagination): Users can continuously scroll down without stuttering or page reloads.
• Engagement: Users can like, comment on, and share posts with near-real-time counter updates.

Non-Functional Requirements

• Ultra-Low Latency: Feed generation and retrieval must complete in < 100ms at p99.
• High Availability (99.99%): Feed availability must be resilient against partial backend outages through graceful degradation.
• Freshness SLA: When a followed user posts, it should appear in their followers' feeds within 5 seconds.
• Eventual Consistency: Minor delays in feed appearance are acceptable, but feed reads must never fail or block the UI.

3. Back-of-the-Envelope Capacity Estimations

Let us quantify the scale and calculate the throughput and memory requirements:

1. Traffic Volume (QPS Estimations):

Daily Active Users (DAU) = 500 Million (500M).
Average user visits/refreshes their feed 10 times per day.
Total Daily Feed Requests = 500M × 10 = 5 Billion requests/day.
Average Feed Read QPS = 5,000,000,000 / 86,400 ≈ 58,000 QPS (Peak ≈ 120,000 QPS).
Post Creation: Assume 100M active creators posting an average of 2 posts/day = 200 Million posts/day.
Average Write QPS = 200,000,000 / 86,400 ≈ 2,300 Write QPS (Peak ≈ 5,000 QPS).

2. Storage Requirements:

Average post record (ID, Author ID, Text, Media URLs, Timestamps) ≈ 1 KB.
Daily Text & Metadata Storage = 200M × 1 KB ≈ 200 GB / day (≈ 73 TB / year).
5-Year Storage Capacity ≈ 365 TB (binary photos/videos stored separately on S3).

3. In-Memory Feed Cache Sizing (Redis Cluster):

Users rarely scroll past 500 to 800 posts in a single session.
The feed cache stores only post references `(post_id, score)` rather than entire post payloads.
Each index item: 8-byte ID + 8-byte score = 16 bytes.
800-post feed index per user = 800 × 16 B ≈ 12.8 KB.
Caching feeds for 100M daily active concurrent users = 100M × 12.8 KB ≈ 1.28 TB RAM (easily distributed across a modest Redis cluster).

4. Naive Architecture & "The SQL Query from Hell"

A junior developer tasked with building a news feed typically starts with a standard relational database and a single SQL query:

-- The Naive Feed Query
SELECT p.id, p.author_id, p.content, p.media_url, p.created_at
FROM posts p
JOIN follows f ON p.author_id = f.following_id
WHERE f.follower_id = :current_user_id
ORDER BY p.created_at DESC
LIMIT 20 OFFSET 0;

Why does this query catastrophically fail at scale?

1. Massive Join & Sorting Overhead: If a user follows 2,000 accounts, the database engine must scan millions of candidate rows across 2,000 index partitions, merge them into memory, sort them by timestamp, and paginate.

2. Concurrency Collapse: When 120,000 users per second execute this query simultaneously, database CPU saturates at 100%, disk I/O thrashes, connection pools exhaust, and the entire database cluster grinds to a halt.

Querying and sorting posts dynamically on every feed request is fundamentally unviable at scale. This brings us to the two foundational architectural strategies in system design: Fan-Out on Read vs. Fan-Out on Write.

5. Fan-Out on Read (The Pull Model)

In Fan-Out on Read (the Pull Model), no precomputation takes place when a post is created. The post is simply written to the author's personal post table. When a follower requests their feed, the system pulls and aggregates posts from all followed accounts in real time.

Fan-Out on Read Architectural Flow

[Alice Creates Post] ──────► [Post Store] (Saved to Alice's table in 2ms!)
                                  │
                                  ▼ (Zero fan-out work at write time)

[Bob Opens Feed] ──────────► [Feed Service]
                                  │
                                  ├─► 1. Query: Who does Bob follow? (e.g., 1,500 users)
                                  ├─► 2. Fetch recent posts for all 1,500 users
                                  ├─► 3. Merge & sort candidate posts in memory
                                  └─► 4. Return top 20 posts to Bob

Advantages (Pros):

• Blazing Fast Writes: Publishing a post requires only a simple single-row insert ($O(1)$ write time).
• Zero Wasted Storage: Inactive users who haven't logged in for months consume zero cache or precomputation resources.

Disadvantages (Cons):

• Slow Feed Reads: Every feed fetch requires querying hundreds of candidate sources, resulting in high latency (200ms to several seconds).
• Hotspot Read Load: Frequent feed refreshes during peak hours place unsustainable read pressure on storage systems.

6. Fan-Out on Write (The Push Model)

Recognizing that social media is predominantly read-heavy, engineers designed the opposite strategy: Fan-Out on Write (the Push Model).

The core philosophy is: "Make reads as fast as an $O(1)$ memory lookup; perform the heavy computational lifting asynchronously at write time." When a user publishes a post, asynchronous background workers look up all their followers and push a reference of that post into each follower's dedicated Feed Cache (Inbox).

Fan-Out on Write Architectural Flow

[Alice Creates Post] ──► [Post Service] ──► [Kafka Queue]
                                                 │
                                                 ▼
                                        [Fan-Out Workers]
                                                 │
                        ┌────────────────────────┼────────────────────────┐
                        ▼                        ▼                        ▼
                [Bob's Feed Cache]       [Charlie's Feed Cache]   [Diana's Feed Cache]
                (Redis Sorted Set)       (Redis Sorted Set)       (Redis Sorted Set)
                ZADD feed:bob <ts> P1    ZADD feed:charlie <ts>   ZADD feed:diana <ts>

[Bob Opens Feed] ──────► [Feed Service] ──► [Bob's Feed Cache] ──► Returns in 2ms!

Advantages (Pros):

• Instantaneous Reads ($O(1)$): When a user opens their app, their feed is already precomputed in Redis. Reading the top 20 post IDs takes < 5 milliseconds.
• Protects Primary Databases: Read traffic bypasses transactional databases entirely, serving 99% of requests directly from in-memory caches.

Disadvantages (Cons):

• High Write Amplification: A single post triggers thousands of downstream cache write operations.
• Wasted Precomputation: Inactive users receive feed updates despite never opening the application, wasting valuable memory.

7. The Celebrity Problem (The Hotspot Bottleneck)

The pure Fan-Out on Write model operates gracefully when users have moderate follower counts (e.g., 200 to 5,000 friends). However, when a mega-celebrity or high-profile public account (such as Cristiano Ronaldo or Elon Musk) with 100 Million followers posts, the pure push architecture breaks down completely:

What Happens During a Celebrity Post?

1 single post = 100,000,000 Redis write operations!
Kafka queues are flooded with tens of millions of fan-out jobs. Workers spend hours processing this single avalanche of writes. Meanwhile, regular posts published by ordinary users get stuck behind this massive queue backlog (Queue Starvation). As a result, your friend's post takes hours to appear on your timeline.

8. The Hybrid Push/Pull Architecture (Industry Standard)

Modern large-scale production systems (Twitter/X, Facebook, Instagram) solve this dilemma through an intelligent Hybrid Approach:

Hybrid Architectural Decision Matrix

                      ┌────────────────────────┐
                      │  User Publishes Post   │
                      └───────────┬────────────┘
                                  │
                  Is Author a "Celebrity"?
                  (Followers > 25,000 Threshold)
                         /                                      NO /                   \ YES
                       ▼                     ▼
             [Standard User]          [Celebrity User]
                   │                         │
                   ▼                         ▼
            Fan-Out on WRITE          Fan-Out on READ
                   │                         │
                   ▼                         ▼
         Push to Followers'        Store in Celebrity's
            Feed Caches               Recent Posts Cache
         (Redis Sorted Sets)          (Single Cache Key)
                                             │
   ══════════════════════════════════════════╪════════════════════════════════════
                                             │
      When Follower (Bob) Opens Feed:        │
                   │                         │
                   ├─────────────────────────┘
                   ▼
     1. Read Bob's Precomputed Feed (Friends' posts from Redis)
     2. Read Recent Posts of Celebrities that Bob follows
     3. Merge & Re-rank the two streams in memory (< 10ms)
     4. Return Unified Timeline to Bob!

1. Follower Threshold: We define an operational threshold—such as 25,000 followers. For 99% of normal users who fall below this threshold, we employ Fan-Out on Write, as pushing to 500 followers takes just a few milliseconds.

2. Fan-Out on Read for Celebrities: Accounts with over 25,000 followers never trigger write fan-out jobs. Their post is stored once in a dedicated high-throughput celebrity cache partition.

3. Read-Time Stream Merging: When Bob opens his feed, the Feed Service fetches Bob's precomputed feed (standard friends' posts) and simultaneously pulls recent posts from any celebrities Bob follows (typically 10-20 accounts). Both streams are merged and sorted in memory in under 10ms.

4. Active User Filtering: Fan-out writes are only dispatched to followers who have logged in within the past 7 days. Inactive users have their feeds computed on-demand upon their return.

9. Feed Store vs. Post Store Separation

A foundational rule of social feed system design is: Never store post content directly inside the feed index!

Feed Store (Redis Sorted Sets)

Contains only the candidate post IDs and their ranking scores. No heavy text bodies or media blobs reside here.

// Redis Key: user:101:feed
// Type: ZSET (Sorted Set)
ZADD user:101:feed 1719283000 post_9841
ZADD user:101:feed 1719283150 post_9845
ZADD user:101:feed 1719283900 post_9890

// Retrieve top 20 latest post IDs:
ZREVRANGEBYSCORE user:101:feed +inf -inf LIMIT 0 20

Post Store (Hydration Cache & DB)

The canonical post entity (text, media links, author metadata, counters) is stored once as an independent object.

// Multi-Get Post Details from Memcached/Redis
MGET post:9890 post:9845 post:9841

// Post Object Schema:
{
  "id": "post_9890",
  "author_id": "alice",
  "text": "Hello System Design!",
  "media_url": "https://cdn.example/img.jpg",
  "likes_count": 341,
  "created_at": 1719283900
}

The Benefit of Hydration: If Alice edits or deletes her post, there is no need to mutate millions of followers' feed caches. Updating the canonical Post Store immediately reflects across all followers on their next refresh.

10. Pagination: Offset vs. Cursor-Based Deep Dive

In social feeds, users continuously scroll down to reveal more content (Infinite Scrolling). Implementing pagination requires choosing between two distinct mechanisms:

A. Offset Pagination (Why it breaks in active feeds)

GET /feed?page=2&limit=10 ➔ SQL: LIMIT 10 OFFSET 10

Offset pagination suffers from two severe flaws:

Performance Degradation: As offset values increase (e.g., `OFFSET 10000`), the database must scan and discard 10,000 records ($O(N)$ execution time).
Duplicate & Skipped Posts Anomaly: Suppose Bob is viewing Page 1 (Posts 10 to 7). While reading, 2 new posts are published at the top of the timeline. When Bob scrolls down to fetch Page 2 (`OFFSET 4`), the entire dataset has shifted down by 2 positions. Bob sees Post 8 and Post 7 a second time! Conversely, if posts are deleted, items are silently skipped.

B. Cursor-Based Pagination (The Production Standard)

GET /feed?cursor=1719283150_post9845&limit=10

Instead of passing a page number, the client sends a Cursor Pointer representing the last seen item. The query performs an efficient index seek:

SELECT * FROM feed 
WHERE (created_at, post_id) < (:cursor_timestamp, :cursor_post_id)
ORDER BY created_at DESC, post_id DESC
LIMIT 10;

Regardless of how many new posts are added to the top of the feed, the cursor guarantees continuous sequential retrieval. Zero duplicates, zero skipped posts, and consistent $O(1)$ index seek performance.

11. Two-Stage Feed Generation: Candidate Generation & Ranking

Modern social feeds do not merely order posts chronologically; they utilize machine learning ranking algorithms to deliver the most engaging content. Selecting the top 20 posts from billions of candidates within 50 milliseconds is achieved through a Two-Stage Pipeline:

Two-Stage Candidate Selection & Ranking Pipeline

      [ 1,000,000,000 Global Posts ]
                     │
                     ▼
  ┌─────────────────────────────────────┐
  │ STAGE 1: Candidate Generation       │ ◄─── Fast Filtering (< 15ms)
  │ • Posts from followed friends       │      Inverted index, Graph service,
  │ • User interest topics & trends     │      Precomputed Redis Inboxes.
  │ • Viral local content               │
  └──────────────────┬──────────────────┘
                     │ (Filters down to ~500 to 1,000 Candidate Posts)
                     ▼
  ┌─────────────────────────────────────┐
  │ STAGE 2: Ranking & Scoring (ML)     │ ◄─── Deep Scoring Model (< 35ms)
  │ • Recency Decay (Time elapsed)      │      Two-Tower Neural Networks,
  │ • Affinity Score (Social closeness) │      XGBoost, CTR Prediction:
  │ • Historical Engagement (Likes/RTs) │      P(click), P(like), P(share)
  └──────────────────┬──────────────────┘
                     │ (Selects Top 50 Ranked Posts)
                     ▼
  ┌─────────────────────────────────────┐
  │ STAGE 3: Business Logic & Filters   │ ◄─── Final Polish (< 5ms)
  │ • Deduplication                     │
  │ • Diversity (No 10 consecutive posts)│
  │ • Sponsored Ad Insertion            │
  └──────────────────┬──────────────────┘
                     │
                     ▼
         Deliver to User Screen!

Simplified Mental Model of Ranking Score (EdgeRank Concept): $$ ext{Score} = w_1 \cdot ext{Recency} + w_2 \cdot ext{Affinity} + w_3 \cdot ext{Engagement} + w_4 \cdot ext{ContentTypeMatch}$$

Affinity ($w_2$): How frequently does Bob view Alice's profile or reply to her posts? Stronger interaction history yields higher affinity.
Engagement ($w_3$): What is the post's rate of likes, comments, and shares? High engagement velocity signals compelling content.
Content Type Match ($w_4$): Does Charlie prefer video content or long-form text? Charlie's feed will weight video posts higher.

12. High-Concurrency Engagement & The Sharded Counter Pattern

When a viral post receives hundreds of thousands of likes and comments per second, direct database updates fail:

// Naive Update: UPDATE posts SET likes_count = likes_count + 1 WHERE id = 123;

With 50,000 concurrent updates hitting the same database row, Row-Level Exclusive Locks create severe contention, queueing threads and triggering deadlocks. Two proven architectures eliminate this bottleneck:

1. The Sharded Counter Pattern

Instead of maintaining a single counter row, the counter is split into $N$ separate slots (e.g., 20 shards). Incoming likes are assigned to a random shard:

shard_id = hash(user_id) % 20;
UPDATE post_likes_shards SET count = count + 1 WHERE post_id = 123 AND shard_id = :shard_id;

Total likes are computed by summing the shards: $ ext{Total} = \sum ext{Shards}$. Lock contention drops by a factor of 20.

2. Redis Write-Behind Buffer

Likes are incremented atomically in Redis RAM (`INCR post:123:likes`). Asynchronous Kafka workers flush aggregated counts in batches to persistent databases every 5 seconds.

13. Media Storage & Global CDN Delivery

Social feeds are media-intensive. Streaming high-resolution media directly through application servers is an anti-pattern.

Direct Presigned S3 Upload: Clients request a short-lived Presigned URL from the API Gateway and upload media binaries directly to Amazon S3 or Google Cloud Storage, completely bypassing application servers.
Asynchronous Transcoding Pipeline: Media arrival triggers event-driven transcoding workers that generate multiple adaptive resolutions (HLS/DASH for video, WebP/AVIF for images).
Global CDN Edge Caching: All media URLs point to CloudFront or Cloudflare Edge PoPs, serving media in < 50ms worldwide.

14. Master End-to-End Distributed Architecture Blueprint

Let us integrate all architectural components into a comprehensive end-to-end production blueprint:

Master Social Media Feed Architecture Blueprint

                           ┌──────────────────────────┐
                           │      Mobile & Web Users  │
                           └─────────────┬────────────┘
                                         │
                                         ▼
                           ┌──────────────────────────┐
                           │   Cloudflare Edge / WAF  │
                           └─────────────┬────────────┘
                                         │
                                         ▼
                           ┌──────────────────────────┐
                           │    API Gateway / ALB     │
                           └──────┬────────────┬──────┘
                                  │            │
            ┌─────────────────────┘            └─────────────────────┐
            ▼                                                        ▼
┌───────────────────────┐                                ┌───────────────────────┐
│     Feed Service      │                                │     Post Service      │
│  (Read Path - <50ms)  │                                │ (Write Path - <10ms)  │
└───────────┬───────────┘                                └───────────┬───────────┘
            │                                                        │
            ├─► 1. Check Redis Feed Cache                            ├─► 1. Save to Post DB
            ├─► 2. Pull Celebrity Posts                              │   (Cassandra/Postgres)
            ├─► 3. Two-Stage ML Ranker                               └─► 2. Publish Event
            │   (Scoring Pipeline)                                           │
            ├─► 4. MGET Post Hydration Cache                                 ▼
            │   (Memcached/Redis)                                ┌───────────────────────┐
            │                                                    │   Kafka Event Bus     │
            ▼                                                    └───────────┬───────────┘
┌───────────────────────┐                                                    │
│  Personalized Feed    │                                                    ▼
│  Returned to Client   │                                        ┌───────────────────────┐
└───────────────────────┘                                        │    Fan-Out Workers    │
                                                                 │ (Async Queue Service) │
                                                                 └───────────┬───────────┘
                                                                             │
                                              ┌──────────────────────────────┴──────────────────────────────┐
                                              ▼                                                             ▼
                                 ┌─────────────────────────┐                                   ┌─────────────────────────┐
                                 │   Social Graph Service  │                                   │    Redis Feed Cache     │
                                 │   (Followers Lookup)    │                                   │   (Active User ZSets)   │
                                 └─────────────────────────┘                                   └─────────────────────────┘

15. 🎮 Interactive Workbench: Social Feed Simulator

Experiment with the three dynamic simulations below: observe the queue impact of celebrity posts, tweak ML ranking weights in real time, and inspect cursor pagination consistency.

Social Feed Interactive Workbench

Explore fan-out trade-offs, algorithmic scoring, and cursor pagination

16. 💡 Concept Check Quiz

Test your mastery of social media feed architecture:

1. Why does a pure Fan-Out on Write (Push) architecture collapse when an account with 80 Million followers publishes a post?

2. What makes Cursor-Based pagination superior to Offset pagination in high-velocity social media timelines?

3. What is the primary purpose of the Sharded Counter pattern on viral posts receiving thousands of likes per second?

Next Step · Part 4 · Lesson 15

Designing a Video Streaming System — How to Design a Platform like YouTube or Netflix

We have conquered real-time messaging and personalized social feeds. But video consumes over 80% of global internet traffic! How do you ingest gigabytes of 4K video, transcode into adaptive bitrates (HLS/DASH), and stream globally without a millisecond of buffering? Welcome to the next challenge!

Course Syllabus Go to Lesson 11.15

Lesson 11.14 — Designing a Social Media Feed: How to Design a Scalable News Feed (Facebook / X)

What You Will Master in This Lesson