Lesson 11.12 — Designing a Real-World System: How to Design a Scalable URL Shortener

1. All Concepts Together: Entering Real-World Architecture

In previous lessons, we studied the foundational building blocks of system design in isolation:

• How a Load Balancer evenly spreads requests across a fleet of servers.
• How Databases scale using Replication, Partitioning, and Sharding.
• Why Caching is indispensable for relieving database load and dropping response latencies to single-digit milliseconds.
• How Message Queues & Workers offload heavy background tasks to preserve fast user responses.
• How Distributed Systems & CAP Theorem keep clusters coherent across unavoidable network partitions.
• The practical trade-offs between Monoliths and Microservices.
• How Cloud, Docker & CI/CD automate deployments and eliminate downtime.
• Why Reliability, Security & Observability keep production systems protected, resilient, and transparent.

Until now, we examined each concept inside its own room. But in the real world, a software architect doesn't design "just a cache" or "just a load balancer." They build a living, breathing system where every component collaborates like precision clockwork.

Today, we put it all together. For the first time, we will assemble all these building blocks to design a classic high-scale system: A Scalable URL Shortener (such as TinyURL or Bitly).

2. What is a URL Shortener and Why is it Needed?

Modern web applications frequently generate long, cumbersome URLs containing nested paths, affiliate IDs, and marketing campaign parameters:

Sending this via SMS wastes character budgets (exceeding the standard 160-character limit), looks cluttered on social media feeds, and is impossible to print cleanly on billboards or brochures. A URL shortener generates a concise, elegant alias:

When a user visits this shortened URL, our system instantaneously resolves the mapping and redirects the client's browser to the destination address.

3. Deconstructing the Problem: The Two Core Operations

The first rule of system design is to distill complexity down to fundamental core operations:

4. Steps 1 & 2 — Functional & Non-Functional Requirements

Functional Requirements

1. URL Shortening: Given a long URL, return an ultra-short alias (e.g. https://short.example/aB91x).
2. Redirection: Visiting the short URL instantly redirects the client to the original long URL.
3. Custom Alias (Optional): Users can request vanity aliases (e.g. short.example/salim).
4. Expiration (Optional): Users can set link lifetimes (e.g. 7 or 30 days). Expired links return 404 Not Found.
5. Analytics (Commercial requirement): Track click count, geographical location, referrer, and user-agent.

Non-Functional Requirements

• High Availability (99.99%): Redirection must never drop; links printed in newspapers or marketing emails cannot fail.
• Ultra-Low Latency: Redirection response must complete in < 10ms so users perceive immediate navigation.
• Deterministic Uniqueness: Zero collisions—two different URLs must never receive the same active short code.
• Scalability: The system must handle viral link traffic spikes effortlessly without database degradation.

5. Steps 3 & 4 — API Design & Initial Architecture

Our service exposes two primary REST endpoints:

// Request Body
{
  "url": "https://example.com/very/long/url/for/summer-sale",
  "custom_alias": "summer26",  // Optional
  "expires_in_days": 30         // Optional
}

// Response (201 Created)
{
  "short_url": "https://short.example/summer26",
  "short_code": "summer26",
  "original_url": "https://example.com/very/long/url/for/summer-sale",
  "created_at": "2026-09-25T16:00:00Z",
  "expires_at": "2026-10-25T16:00:00Z"
}

// Request: GET /aB91x

// HTTP Response Headers:
HTTP/1.1 302 Found
Location: https://example.com/very/long/url/for/summer-sale

6. Steps 5 & 6 — Database Schema & Unique Short Code Generation

Our relational schema requires a lean, index-optimized table:

CREATE TABLE urls (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    short_code VARCHAR(16) NOT NULL UNIQUE,
    original_url TEXT NOT NULL,
    user_id BIGINT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP NULL,
    INDEX idx_short_code (short_code)
);

The Algorithmic Challenge: Generating Unique Short Codes

We want compact codes like aB91x (6 to 7 characters). How do we mathematically guarantee zero collisions across billions of records?

Approach A: Base62 Encoding (The Standard Industry Solution)

Standard human numbers use Base10 (0-9). Hexadecimal uses Base16 (0-9, a-f). For clean, URL-safe strings, Base62 is the gold standard:

• Digits: 0-9 (10 characters)
• Lowercase letters: a-z (26 characters)
• Uppercase letters: A-Z (26 characters)
• Total Alphabet Size = 10 + 26 + 26 = 62 characters!

How it works: Take a sequential database auto-increment ID (e.g. 125). Divide repeatedly by 62 and collect the remainders to form the Base62 characters:

Database Auto-Increment ID (e.g. 125)
  ↓
Divide by 62: Quotient 2, Remainder 1  → '1'
Divide by 62: Quotient 0, Remainder 2  → '2'
  ↓
Base62 Short Code = "21" (Guaranteed 0% collisions!)

Approach B: Cryptographic Hash (MD5 / SHA-256) & Truncation

Hashing the URL with MD5 produces a 128-bit hex string (32 characters). Taking the first 7 characters yields compact codes, but introduces Hash Collisions: two distinct URLs may generate the identical first 7 characters! To resolve collisions, the service must query the database to verify uniqueness. If a collision occurs, it appends a salt/timestamp and re-hashes. At scale, these database lookups create latency spikes.

7. Steps 7 & 8 — Redirection Optimization & Caching

When redirecting user traffic, choosing between HTTP 301 and 302 represents an essential architectural trade-off:

The Redis Cache-Aside Pattern

Because 302 redirects hit our infrastructure every time, querying the disk database on every click would overwhelm storage IOPS. We insert an in-memory Redis Cache:

User Click (GET /aB91x)
       |
       v
Application Server
       |
       v
+--------------+
| Redis Cache  |
+--------------+
       |
       +--- [CACHE HIT: ~2ms] ----> Original URL found!
       |                                   |
       |                                   v
       |                             Return HTTP 302 Redirect to User
       |
       +--- [CACHE MISS: ~35ms]
                   |
                   v
           Database Replica
                   |
                   v
            Original URL retrieved
                   |
                   v
         Cache backfilled in Redis (with TTL)
                   |
                   v
         Return HTTP 302 Redirect!

The 80/20 Pareto Rule: In practice, 20% of shortened URLs generate 80% of total click volume. Keeping this hot 20% in memory delivers sub-2ms lookups for 95%+ of incoming traffic.

8. Step 16 — Asynchronous Analytics Pipeline

Every click carries valuable telemetry: IP address, geolocation, referrer URL, and browser user-agent. However, executing synchronous database writes during the redirection request introduces intolerable latency.

User Request (GET /aB91x)
       |
       v
Application Server
       |
       +---> [Sync Path]  Redis Lookup ➔ 302 Redirect (User redirected in < 3ms!)
       |
       +---> [Async Path] Publish Event: Kafka / RabbitMQ Queue
                                |
                                v
                       Analytics Background Workers
                                |
                                v
                       Columnar OLAP Storage
                    (ClickHouse / TimescaleDB)

9. Complete URL Shortener System Architecture Blueprint

Here is the complete end-to-end production architecture:

                             GLOBAL CLIENTS
                                   |
                                   v
                        CDN / Cloudflare (WAF & DDoS)
                                   |
                                   v
                          Load Balancer (ALB)
                                   |
                +------------------+------------------+
                |                  |                  |
                v                  v                  v
           App Node 1         App Node 2         App Node 3
         (Stateless API)    (Stateless API)    (Stateless API)
                |                  |                  |
                +------------------+------------------+
                                   |
                 +-----------------+-----------------+
                 |                                   |
                 v                                   v
        [ In-Memory Cache ]                 [ Database Tier ]
          Redis Cluster                    Primary DB (Writes)
         (Sub-2ms Lookups)                           |
                                            Replication Sync
                                                     |
                                                     v
                                             Read Replicas (Reads)
                                                     |
                                            (On Cache Miss Only)

                 +-----------------------------------+
                 | Click Event Stream                |
                 v                                   v
       [ Message Queue / Kafka ]          [ Key Generation Service ]
                 |                               (KGS Fleet)
                 v
         Analytics Workers
                 |
                 v
      Columnar Storage (ClickHouse)
                 |
                 v
       Analytics Dashboard

      [ Observability: Prometheus Metrics + Grafana + OpenTelemetry ]

10. The Universal System Design Framework

The core value of this lesson is not just knowing how to build TinyURL. It is mastering the universal methodology for tackling any system design challenge: