40 min read Distributed Systems Part 3 · Lesson 8 (8 / 12)
Course 11: Part 3 — Asynchronous & Distributed Systems

Lesson 11.8 — Distributed Systems & Consistency: Cooperating Machines & Reliability

Networks are inherently unreliable and nodes fail arbitrarily—yet how does a system maintain coherent state? CAP theorem, eventual consistency, distributed Sagas, and circuit breakers demystified.

Key Learning Objectives

The Reality of Distributed Systems: Network partitions, packet loss, partial failures, and designing for failure as the default environment.
Consistency Models & CAP Theorem: Deconstructing Strong vs. Eventual Consistency and navigating CP vs. AP trade-offs during network partitions.
Distributed Transactions & Sagas: Coordinating state across discrete services using compensating rollbacks instead of blocking two-phase commits.
Resilience & Cascading Failure Defense: Implementing the Circuit Breaker pattern (Closed, Open, Half-Open) and distributed locks.
1

When Computers Must Cooperate Over Networks

In previous lessons, we added load balancers, scaled application clusters, replicated databases, deployed caches, and wired message queues.

Our system is no longer a single isolated server. It is now a Distributed System spanning dozens of independent compute nodes:

// The Modern Distributed Topology
[ Global Users across Asia, Europe & Americas ]
↓ (Internet / Public Network)
[ Anycast DNS + CDN Edge Nodes ]
↓
[ Load Balancers & API Gateways ]
↓ (Internal Virtual Private Cloud)
[Order Service] [Payment Service] [Inventory Service]
↓
[Message Broker / Kafka] [Redis In-Memory Cache] [PostgreSQL Primary + Replicas]

This introduces a fundamental architectural question:

“When 50 independent computers cooperate over network cables to deliver a single web application, how does the system maintain coherent state when cables sever, servers crash, or network packets vanish?”
2

Why Multiple Computers?

If a single computer could satisfy all operational requirements, no engineering team would voluntarily embrace distributed systems complexity. We distribute architectures for three core reasons:

1. Boundless Scale No single hardware chassis on Earth possesses sufficient CPU, RAM, and disk bus bandwidth to process millions of concurrent transactions.
2. High Availability Surviving hardware failures without downtime (eliminating Single Points of Failure through active redundancy).
3. Geographic Proximity Locating infrastructure near users across multiple datacenters to overcome the speed of light in optical fiber.
3

A Network Call is NOT a Local Function Call

Calling a local function getUser() inside memory executes deterministically in nanoseconds. It never drops packets or experiences routing timeouts.

However, when Service A calls Service B over a network:

// The Unreliable Network Reality
Service A ───(HTTP POST /charge)───> [Unreliable Network] ───> Service B
  ❌ Packets drop midway at intermediary routers
  ❌ Service B is stalling due to garbage collection pauses or thread pool exhaustion
  ❌ Service B successfully processes the charge, but the response packet drops on the return route!
  ⚠️ Service A faces ambiguity: Did the charge succeed, or did it fail?
“The Golden Rule of Distributed Systems: Failure is not an edge case; failure is the default environment. Architect systems assuming remote calls can and will fail at any moment.”
4

Consistency Models: Strong vs. Eventual Consistency

When data is replicated across multiple nodes, the rules governing when nodes agree on data state define the system's Consistency Model:

Strong Consistency (Strict Correctness)

Every read request across any node is guaranteed to return the latest committed write. Stale reads are physically prevented.

✓ Ideal for: Financial balances, seat reservation, inventory checkout.
✗ Penalty: Higher latency; blocks writes during network partitions.
Eventual Consistency (Convergence)

Replicas do not synchronize immediately. Temporary read divergence is tolerated. Given sufficient time without new writes, all nodes eventually converge to the identical state.

✓ Ideal for: Video view counters, social likes, activity feeds.
✓ Advantage: Blazing speed and 100% availability.
5

The CAP Theorem: The Law of Distributed Nature

Formulated by Dr. Eric Brewer, the CAP Theorem proves that a distributed data store can guarantee at most two out of three properties simultaneously:

  • C (Consistency): Every read receives the latest write or an error.
  • A (Availability): Every non-failing node returns a response for every request (no errors, but may return stale data).
  • P (Partition Tolerance): The system continues to operate despite arbitrary message loss or network partitions between nodes.

The Architectural Reality: P is Non-Negotiable!

In physical networks, hardware routers fail, switch cables disconnect, and datacenters lose cross-connects. You cannot simply "opt out" of network partitions.

Therefore, when a partition occurs, an architect must choose between CP and AP:

CP (Consistency + Partition Tolerance) When nodes cannot communicate, writes are rejected to prevent split-brain state divergence. Sacrifices availability to preserve absolute consistency. Examples: Google Spanner, HBase, ZooKeeper, CockroachDB
AP (Availability + Partition Tolerance) Both sides of the partitioned network continue accepting writes. Nodes temporarily diverge and reconcile later. Sacrifices consistency to maximize availability. Examples: Amazon DynamoDB, Apache Cassandra, CouchDB
6

Distributed Transactions & The Saga Pattern

In a monolithic SQL database, transactions are simple: BEGIN ... COMMIT / ROLLBACK.

In microservices architectures where Orders, Payments, and Inventory live on discrete databases, traditional Two-Phase Commit (2PC) blocks threads and destroys performance.

The Saga Pattern: Orchestrated Compensating Actions

A Saga breaks a distributed transaction into a sequence of local transactions. Each step is paired with a Compensating Transaction (Rollback Action):

// The E-Commerce Saga Flow with Rollback
Step 1: Create Order [Status: PENDING] ✓
Step 2: Reserve Inventory (Stock -1) ✓
Step 3: Charge Payment → 💥 PAYMENT DECLINED!
// TRIGGERING COMPENSATING TRANSACTIONS:
└──> Compensation 1: Release Inventory (Stock +1 restored) ✓
└──> Compensation 2: Mark Order as CANCELLED ✓
✓ All services cleanly return to consistent state! Zero data corruption!
7

The Circuit Breaker: Preventing Cascading Failures

Just as electrical circuit breakers trip to protect residential appliances during power surges, software Circuit Breakers prevent failing dependencies from crashing calling services.

If an upstream recommendation service starts taking 10-second timeouts, calling it repeatedly exhausts caller connection pools and brings down the entire website (Cascading Collapse).

1. Closed (Normal Operation) Requests flow freely to the downstream service. The circuit breaker monitors error rates.
2. Open (Tripped / Short-Circuited) Repeated failures trip the breaker. Calls are blocked instantly with 0ms local fallbacks, saving caller threads.
3. Half-Open (Test Probe) After a cooldown window (e.g., 30s), a single probe request checks downstream health. If successful, the circuit resets to Closed.

Interactive Distributed Systems Workbench

Simulate network partitions, circuit breaker state machines, and Saga rollbacks

Distributed Systems Quiz

Validate your comprehension of Lesson 11.8 core concepts

1. According to the CAP Theorem, when an unavoidable network partition disconnects two datacenters, what is an architect's only operational option?

2. When a Circuit Breaker enters the "Open" state, how does it behave upon receiving an incoming request?

3. In the Saga Pattern, what is the role of a "Compensating Transaction"?

Next Step: Part 3 · Lesson 11.9

Monolith, Modular Monolith & Microservices: How to Decompose an Application

Does every project really need microservices? Why do tech giants start with monoliths, and how does the golden middle ground of "Modular Monoliths" offer the best of both worlds?

Course Syllabus Proceed to Lesson 11.9