Lesson 14.27: Conditional Probability & Bayes' Theorem

1 Introduction: How New Information Transforms Probability

Suppose an urn contains 5 Red marbles and 5 Blue marbles. A marble is drawn at random. What is the probability of drawing a Red marble? Classical probability gives: $\frac{5}{10} = 50\%$.

Now imagine an observer glances inside and reveals: "The marble drawn is definitely not black, and it has an even number stamped onto it." The moment this new evidence arrives, your sample space instantly shrinks. You are no longer evaluating across the original 10 marbles—the probability must be recalculated.

This dynamic scenario occurs everywhere in modern computing:

• Biomedical Diagnostics: A patient's diagnostic screening test comes back Positive (+). Even with a 99% accurate test, given that the disease is extremely rare, what is the actual probability that the patient truly has the disease?
• Automated Spam Filtering: An incoming message contains the tokens "Wire Transfer" and "Free Lottery". Knowing the presence of these suspicious phrases, what is the updated probability that the email is Spam?
• Robotics & Autonomous Navigation: A self-driving vehicle's camera perceives a blurry silhouette in the fog. How should the robot fuse sensor readings with prior map data to infer whether an obstacle is present?

Evaluating the likelihood of an event given that another event has already occurred is known as Conditional Probability. The foundational mathematical mechanism that updates prior beliefs into posterior conclusions using evidence is Bayes' Theorem.

2 What is Conditional Probability? (The Shrinking Sample Space)

The probability of an event $A$ occurring given that event $B$ has already occurred is called the Conditional Probability of $A$ given $B$, written as:

The vertical bar ($|$) denotes "Given". The formal mathematical definition is:

Breaking down the components of this equation:

• $P(A \cap B)$: The joint probability that both event $A$ and event $B$ occur simultaneously.
• $P(B)$: The marginal probability of the conditioning event $B$. It must be strictly positive ($P(B) > 0$) because conditioning on an impossible event is mathematically undefined.

3 Step-by-Step Worked Example: Rolling a Die

Consider a single roll of a fair six-sided die. The sample space is:

Let us define two events:

• Event $A$: Rolling an even number $= \{2, 4, 6\} \implies n(A) = 3$
• Event $B$: Rolling a number strictly greater than 3 $= \{4, 5, 6\} \implies n(B) = 3$

Prior to any roll, the unconditional probability of rolling an even number is: $P(A) = \frac{3}{6} = \frac{1}{2} = 50\%$.

4 Visualizing Conditional Probability with Venn Diagrams

Venn diagrams provide an immediate geometric understanding of conditioning:

5 The Multiplication Rule of Probability

Multiplying both sides of the conditional probability definition by denominator $P(B)$ yields the Multiplication Rule:

In words: The joint probability of both events occurring equals the probability of the first event multiplied by the conditional probability of the second event given that the first occurred.

6 Independent Events: When Conditioning Has No Effect

If knowing that event $B$ occurred provides zero information regarding whether event $A$ occurs, the events are called Independent:

Substituting $P(A \mid B) = P(A)$ into the multiplication rule gives the universal product criterion for independence:

7 The Law of Total Probability

In real-world systems, an outcome can occur through several distinct pathways. If events $B_1, B_2, \dots, B_n$ form a partition of sample space $S$ (mutually exclusive and exhaustive, $\sum B_i = S$), the total probability of event $A$ is the sum of its conditional probabilities across all pathways:

For the canonical binary case ($B$ occurs or its complement $B^c$ occurs):

8 Bayes' Theorem: Inverting Conditional Probabilities

We frequently know the probability of evidence given a hypothesis ($P(B \mid A)$—e.g. probability of a positive test given disease). But what we truly need is the reverse: what is the probability of the hypothesis given the evidence ($P(A \mid B)$—the test is positive, does the patient truly have the disease)?
Inverting this conditioning direction is accomplished by Bayes' Theorem.

Expanding denominator $P(B)$ via the Law of Total Probability yields the standard operational formulation:

The Four Pillars of Bayes' Theorem

9 Medical Testing & The Base Rate Fallacy

The most famous and counterintuitive application of Bayes' Theorem is medical screening. Consider a realistic diagnostic scenario:

Concrete Verification with 10,000 People

Inspecting the table: Among 594 positive test results, only 99 actually have the disease: $P(D \mid +) = \frac{99}{594} = 16.67\%$. The math is indisputable.

10 Bayes' Theorem in AI: The Naive Bayes Classifier

One of the most widely deployed probabilistic machine learning algorithms is the Naive Bayes Classifier. It is commonly used in spam detection and text categorization.

11 Probability Tree Diagrams: Visual Multi-Stage Modeling

For sequential experiments, Probability Tree Diagrams organize pathways, conditional branches, and joint outcomes cleanly:

12 Practical Python Implementation

Demonstrating Bayes' Theorem and Naive Bayes text classification in concise Python:

def bayes_disease_probability(prior_disease, sensitivity, false_positive_rate):
    """
    Computes P(Disease | Test Positive) using Bayes' Theorem.
    """
    prior_healthy = 1.0 - prior_disease
    
    # Law of Total Probability for P(Test Positive)
    p_positive = (sensitivity * prior_disease) + (false_positive_rate * prior_healthy)
    
    # Bayes' Theorem
    posterior_disease = (sensitivity * prior_disease) / p_positive
    return posterior_disease, p_positive

# Parameters from Section 9
prior_d = 0.01          # 1% prevalence
sens = 0.99             # 99% true positive rate
fpr = 0.05              # 5% false positive rate

posterior, total_pos = bayes_disease_probability(prior_d, sens, fpr)

print(f"Total Positive Test Rate: {total_pos * 100:.2f}%")
print(f"Posterior Probability of Disease Given Positive: {posterior * 100:.2f}%")

import math

class SimpleNaiveBayesSpam:
    def __init__(self, prior_spam=0.30):
        self.p_spam = prior_spam
        self.p_ham = 1.0 - prior_spam
        
        # Word likelihoods P(word | Class)
        self.likelihoods = {
            'free':    {'spam': 0.80, 'ham': 0.10},
            'lottery': {'spam': 0.70, 'ham': 0.02},
            'meeting': {'spam': 0.05, 'ham': 0.60},
            'project': {'spam': 0.02, 'ham': 0.50}
        }
        
    def classify(self, message_words):
        # Log probabilities prevent numerical underflow
        log_prob_spam = math.log(self.p_spam)
        log_prob_ham = math.log(self.p_ham)
        
        for word in message_words:
            if word in self.likelihoods:
                log_prob_spam += math.log(self.likelihoods[word]['spam'])
                log_prob_ham += math.log(self.likelihoods[word]['ham'])
                
        # Exponentiate and normalize
        prob_spam = math.exp(log_prob_spam)
        prob_ham = math.exp(log_prob_ham)
        normalized_spam = prob_spam / (prob_spam + prob_ham)
        
        decision = "SPAM" if normalized_spam > 0.50 else "HAM"
        return decision, normalized_spam

nb = SimpleNaiveBayesSpam()
words = ['free', 'lottery']
decision, p_spam = nb.classify(words)
print(f"Tokens: {words} -> Decision: {decision} (Spam Probability: {p_spam*100:.2f}%)")

13 8 Common Pitfalls & Traps to Avoid

Conditional probability and Bayesian inference are notorious for tripping up human intuition:

14 Hands-On Interactive Workbench: The Bayesian Laboratory

Explore the Base Rate Fallacy in the Medical Diagnostic Lab, manipulate conditional pathways in the Tree Diagram Explorer, and observe live spam scoring in the Naive Bayes Simulator.

15 Key Takeaways & Core Principles

The nine fundamental pillars of Lesson 14.27 at a glance: