Lesson 11.13 — Designing a Real-Time Chat System: How to Design a Scalable Messaging Service like WhatsApp

1. URL Shortener vs. Real-Time Chat: A Fundamental Paradigm Shift

In the previous lesson, we built a URL shortener using the traditional request-response model:

Instant messaging requires a completely different architectural paradigm. Two or more humans are engaged in bi-directional, real-time communication:

• Alice transmits a message, and it must appear on Bob's screen within milliseconds.
• Some users are online, others are offline, and many transition between flaky Wi-Fi and mobile cells.
• Users demand delivery receipts: Sent (✓), Delivered (✓✓), and Read (✓✓ blue ticks).
• Message ordering must remain strictly preserved regardless of network arrival jitter.

2. Requirements & Back-of-the-Envelope Capacity Estimations

Functional Requirements

1. 1-on-1 & Group Messaging: Users can send and receive text and media in real time.
2. Delivery Receipts: Sent (stored on server), Delivered (received by client device), Read (opened by user).
3. User Presence: Online, offline, and last seen timestamps.
4. Offline Storage & Sync: Offline users receive push alerts; messages sync seamlessly upon reconnect.
5. Media Sharing: Images, voice notes, and videos supported via asynchronous pipelines.

Non-Functional Requirements

• Ultra-Low Latency: End-to-end delivery latency must remain sub-100ms globally.
• Zero Message Loss: High durability; messages must survive gateway or node crashes.
• Deterministic Message Ordering: Messages within a conversation must render in the exact order sent.
• High Concurrency: Sustaining millions of active, simultaneous persistent connections.

3. Protocol Selection: Why Traditional HTTP Fails

Standard HTTP is half-duplex and client-initiated. A server cannot push data to a client without the client first requesting it.

4. The Message Lifecycle: Why Persist Before Delivery?

When Alice sends a message, should the chat server immediately forward it to Bob in memory?

Production messaging systems enforce: "Persist First, Deliver Second":

Alice WebSocket Message
         |
         v
Chat Gateway Server
         |
         +--- 1. Validation & Content Security Filter
         |
         +--- 2. Generate Distributed Unique Message ID (Snowflake)
         |
         +--- 3. Commit to Durable Database (Status: SENT ✓)
         |
         +--- 4. Attempt Real-Time Push to Recipient

5. Multiple Chat Servers & Distributed Presence

With millions of concurrent users, a single server cannot hold all WebSocket connections. We maintain a fleet of connection gateways:

The Presence Registry (Redis Cluster)

Whenever a client connects, the gateway records the active mapping in an in-memory Redis cluster:

Message Routing Broker (Redis Pub/Sub or Kafka)

Chat Server 1 does not establish direct TCP links with every other server. Instead, it publishes to a central message broker:

Alice ➔ Chat Server 1
             |
             v (Queries Redis: "Where is Bob connected?")
       Redis Presence ➔ "Bob is on Server 3"
             |
             v
Publish to Message Broker (Channel: "server:chat-3")
             |
             v
Chat Server 3 (Consumes Event)
             |
             v (Pushes over Bob's active WebSocket)
         Bob's Device!

6. Handling Offline Users: Push Notifications & Synchronization

If Bob is offline (no active socket registered in Redis):

1. The message remains safely stored in the database with status SENT.
2. The gateway publishes an event to the Push Notification Service.
3. Apple APNs or Google FCM dispatches a lightweight alert to Bob's device: "Alice sent you a message".

The Offline Synchronization Protocol

When Bob opens the application, he establishes a fresh WebSocket connection and executes a catch-up sync:

The server streams all unread messages in bulk. This decouples Real-Time Delivery (pushing to active sockets) from Synchronization (pulling missed state upon reconnect).

7. Message Ordering & Idempotency

If Alice rapidly transmits: 1. "Where are you?", 2. "Don't be late", 3. "I'm leaving". If network jitter causes message 3 to arrive before message 1, conversational context breaks.

Duplicate Prevention via Idempotency Keys

On flaky mobile networks, Alice's phone might retry sending a message because it didn't receive the server's ACK in time. Each message carries a client-generated UUID (client_msg_id). If the server receives an existing ID, it acknowledges receipt without creating a duplicate record in storage.

8. Media Handling: Decoupling Binary Uploads

Never pipe large binary files (images, audio, video) through the WebSocket chat gateways! Large binaries saturate socket buffers and starve real-time text frames.

1. Request Presigned URL: Client asks the REST API for an upload token: POST /api/v1/media/upload-url.
2. Direct S3 Upload: Client uploads the raw binary directly to AWS S3 / Cloud Storage, bypassing chat servers.
3. Lightweight Message Metadata: Client sends a tiny JSON message over the WebSocket containing the media URL and a blurhash thumbnail.
4. Edge CDN Download: Recipients download media from global CDN edge caches.

9. Group Chats & Fan-Out Dynamics

In a 10-person group, when Alice sends a message, the server replicates it across 9 active connections (Fan-Out on Write). This is fast and simple.

However, in public channels with 100,000+ members (e.g. Telegram broadcast channels), writing 100,000 socket copies causes catastrophic write amplification. Large channels rely on Fan-Out on Read, where messages append once to a channel timeline and active members pull or stream updates.

10. Complete Real-Time Chat System Architecture Blueprint

                             MOBILE / WEB CLIENTS
                                      |
                                      v
                        Load Balancer (TLS & WSS Edge)
                                      |
             +------------------------+------------------------+
             |                                                 |
             v                                                 v
     [ WebSocket Gateway 1 ]                         [ WebSocket Gateway 2 ]
    (Holds Long-Lived Sockets)                      (Holds Long-Lived Sockets)
             |                                                 |
             +------------------------+------------------------+
                                      |
                   +------------------+------------------+
                   |                                     |
                   v                                     v
       [ In-Memory Presence Store ]             [ Message Broker Cluster ]
       (Redis Cluster: User ➔ Server)             (Kafka / RabbitMQ / PubSub)
                                                         |
                                 +-----------------------+-----------------------+
                                 |                                               |
                                 v                                               v
                     [ Chat Persistence Workers ]                  [ Push Notification Workers ]
                                 |                                               |
                                 v                                               v
                    [ Distributed NoSQL Store ]                             [ APNs / FCM ]
                   (Cassandra / ScyllaDB / DynamoDB)                             |
                                                                                 v
                                                                           Offline Users

              [ Media Pipeline: S3 Object Storage + Global CDN Edge Caching ]
              [ Observability: Prometheus Sockets Metric + Jaeger Distributed Tracing ]

11. The Three-Layer Chat Mental Model