Lesson 14.26: Probability Fundamentals

1 Introduction: Reasoning Under Uncertainty

Suppose you flip a coin. Can you predict with 100% deterministic certainty whether it will land on Heads or Tails before it hits the table? Certainly not. However, if the coin is physically balanced and fair, we can rationally state that the chance of observing Heads is exactly equal to the chance of observing Tails.

Similarly, when rolling a standard six-sided die, you cannot foresee the exact number (1 through 6) on the next roll. Yet, due to symmetry and the laws of physics, the mathematical likelihood of each individual face can be computed with exact precision.

In real life and modern technology, we constantly encounter situations where outcomes cannot be determined ahead of time:

• Meteorology: Is the probability of rain in Seattle tomorrow 75% or 10%?
• Manufacturing Quality Control: In a semiconductor fabrication plant producing 10,000 silicon wafers, what fraction is expected to be defective?
• Biomedical Diagnostics: If a patient tests positive on a diagnostic screen, what is the probability that they genuinely have the condition?
• Autonomous AI Systems: Does an autonomous vehicle’s computer vision system recognize a pedestrian crossing the road with 99% confidence or merely 52% doubt?

Rather than leaving uncertainty to subjective guesswork, mathematics provides a formal framework called Probability Theory to quantify, analyze, and make optimal decisions in the presence of randomness.

2 What is Probability? (The 0-to-1 Scale)

Probability is a numerical measure of the likelihood that a specific event will occur, mapped onto a real number between $0$ and $1$. For any arbitrary event $A$, its probability is denoted as $P(A)$.

Probabilities can also be naturally expressed as percentages:

3 Random Experiment, Outcome, and Sample Space

Probability theory uses rigorous vocabulary. Three core building blocks form the foundation of every probability problem:

Master Sample Space Catalog

4 What is an Event? (Event Classification)

An Event is any subset of the sample space $S$. Mathematically, if $E \subseteq S$, then $E$ is an event.

For instance, when rolling a six-sided die with sample space $S = \{1, 2, 3, 4, 5, 6\}$, we can define several events:

• Event of rolling an even number: $A = \{2, 4, 6\}$
• Event of rolling an odd number: $B = \{1, 3, 5\}$
• Event of rolling a prime number: $C = \{2, 3, 5\}$
• Event of rolling a number greater than 4: $D = \{5, 6\}$

When does an event "occur"?
If the actual outcome $\omega$ resulting from the experiment belongs to the subset $E$ (i.e. $\omega \in E$), we say that the event $E$ has occurred. For example, if rolling the die yields a $4$, event $A=\{2,4,6\}$ has occurred, while event $B=\{1,3,5\}$ has not occurred.

5 Classical Probability: The Principle of Equally Likely Outcomes

Formulated by Pierre-Simon Laplace in the 18th century, classical probability applies whenever a finite sample space consists of outcomes that are all Equally Likely (symmetric and equiprobable). Under this assumption:

Here, $n(A)$ denotes the cardinality (element count) of subset $A$, and $n(S)$ is the total size of sample space $S$.

6 Empirical Probability: Estimating Likelihood from Observed Data

In the real world, outcomes are rarely equally likely by physical symmetry. For instance, the chance of a server crashing or a user clicking an advertisement is not 50/50. In such scenarios, we estimate probability based on empirical observations and historical experiments—known as Empirical or Frequentist Probability.

7 Fundamental Axioms of Probability (Kolmogorov's Framework)

In 1933, Russian mathematician Andrey Kolmogorov established probability theory on an unassailable axiomatic foundation. All theorems, propositions, and operational rules in modern probability derive from these three core axioms:

Essential Derived Operational Rules

8 Union and Intersection of Events

Set operations provide the algebra for combining multiple probabilistic events:

The General Addition Rule (Inclusion-Exclusion Principle)

For any two arbitrary events $A$ and $B$, the probability of their union is given by:

9 Mutually Exclusive Events (Disjoint Events)

Two events are called Mutually Exclusive (or Disjoint) if they cannot occur at the same time. Their intersection is strictly empty:

Substituting $P(A \cap B) = 0$ into the General Addition Rule yields the simplified Special Addition Rule:

10 Connecting Probability with Combinatorics (Urn Sampling)

In Part 2 of this course, we studied counting principles, permutations, and combinations ($\binom{n}{k}$). When sample spaces contain thousands or millions of outcomes, counting techniques become essential tools for computing classical probabilities.

The Classical Urn Sampling Problem

An urn contains 4 Red ($R$) and 6 Blue ($B$) marbles (total $4 + 6 = 10$ marbles). Selecting a single marble yields:

Drawing Two Marbles: Two Fundamental Sampling Modes

11 Real-World Applications in Computer Science & AI

Probability is not merely an abstract mathematical discipline—it powers modern software architectures:

12 Practical Python Implementation

Python’s built-in `random` module and native `set` data structures provide an intuitive environment to simulate random processes and verify the Law of Large Numbers:

import random

def simulate_coin_tosses(num_tosses):
    heads_count = 0
    for _ in range(num_tosses):
        # Sample either 'H' or 'T' with uniform probability
        toss = random.choice(['H', 'T'])
        if toss == 'H':
            heads_count += 1
            
    empirical_prob = heads_count / num_tosses
    print(f"Trials: {num_tosses:>6} | Heads: {heads_count:>6} | Empirical P(H): {empirical_prob:.4f}")

print("=== Law of Large Numbers Simulation ===")
for n in [10, 50, 200, 1000, 10000, 100000]:
    simulate_coin_tosses(n)

# Sample space for a single die roll
S = {1, 2, 3, 4, 5, 6}

# Event A: Even numbers, Event B: Numbers >= 4
A = {2, 4, 6}
B = {4, 5, 6}

# Marginal probabilities
p_A = len(A) / len(S)
p_B = len(B) / len(S)

# Joint intersection (A and B)
A_and_B = A.intersection(B) # {4, 6}
p_intersection = len(A_and_B) / len(S)

# Union via General Addition Rule
p_union_formula = p_A + p_B - p_intersection

# Direct set union verification
A_or_B = A.union(B) # {2, 4, 5, 6}
p_union_direct = len(A_or_B) / len(S)

print(f"P(A) = {p_A:.3f}, P(B) = {p_B:.3f}")
print(f"P(A ∩ B) = {p_intersection:.3f}")
print(f"Addition Rule P(A ∪ B) = {p_union_formula:.3f}")
print(f"Direct Set P(A ∪ B)    = {p_union_direct:.3f}")
assert abs(p_union_formula - p_union_direct) < 1e-9, "Formula holds identically!"

13 8 Common Pitfalls & Traps to Avoid

While probability principles appear straightforward, human intuition regularly falls into cognitive traps:

14 Hands-On Interactive Workbench: The Probability Laboratory

Use the interactive laboratory below to test the Law of Large Numbers across thousands of simulated trials, explore how set overlaps dictate union probabilities in the Venn Diagram lab, and experiment with urn marble sampling.

15 Key Takeaways & Core Principles

The nine fundamental pillars of Lesson 14.26 at a glance: