Lesson 14.28: Random Variables & Probability Distributions

1 Introduction: Quantifying Uncertainty into Real Numbers

In the preceding lessons, we explored events as subsets of a sample space—such as rolling an odd number or testing positive for a medical condition. While foundational, raw descriptive events cannot be plugged directly into loss functions, matrix multiplications, or gradient descent steps.

Consider tossing a fair coin three consecutive times. The sample space of raw outcomes is:

As a data scientist or systems engineer, you are usually not interested in the exact chronological sequence of heads and tails. Instead, you care about a quantitative summary: "How many total Heads occurred?" Counting heads for each outcome yields: $3, 2, 2, 2, 1, 1, 1, 0$.

We have mapped an abstract set of text outcomes into real numbers. This quantification occurs in every branch of modern engineering:

• Cloud & Network Engineering: Number of incoming HTTP requests per second arriving at an API gateway ($0, 1, 2, \dots$).
• Hardware Quality Control: Defective microchips detected per batch of manufactured wafers.
• Machine Learning Inference: Latency in milliseconds required for a Large Language Model to decode a prompt ($14.85 \text{ ms}$).

2 What is a Random Variable? (A Mathematical Function)

The most common pedagogical misconception is believing that a Random Variable is a single unpredictable number or an algebraic unknown like $x$ in $2x + 1 = 5$. In rigorous mathematics:

A Random Variable is a deterministic function (mapping) from the sample space $S$ into the set of real numbers $\mathbb{R}$. It assigns a unique real numerical value $X(\omega)$ to every experimental outcome $\omega \in S$.

By convention, capital letters (such as $X, Y, Z$) denote the random variable itself (the function), while lowercase letters ($x, y, z$) denote specific realized numerical values.

Therefore, the range of possible values for the random variable $X$ is:

Important Takeaway: Multiple distinct random variables can be defined on the very same sample space. For example, on this identical 3-coin experiment, $Y$ could be "Total count of tails", or $Z$ could be "Longest run of consecutive heads".

3 Discrete vs. Continuous Random Variables

Depending on the nature and cardinality of values they can assume, random variables are classified into two fundamental domains:

4 What is a Probability Distribution?

A Probability Distribution is the complete mathematical characterization describing all allowable values a random variable can assume alongside their associated probabilities or probability densities.

Intuitively, it tells us how total probability (the budget of $1.0$) is "distributed" across the spectrum of potential outcomes.

5 Probability Mass Function (PMF) — Discrete Distributions

For a discrete random variable $X$, the function that assigns the probability that $X$ equals precisely $x$ is termed the Probability Mass Function (PMF):

Worked Example: Fair 6-Sided Die

Rolling a balanced six-sided die yields $X \in \{1, 2, 3, 4, 5, 6\}$. Uniform equiprobability implies $p_X(x) = \frac{1}{6}$ for each face.

To compute compound event probabilities, such as rolling an even number:

6 Probability Density Function (PDF) — Continuous Integrals

For continuous random variables, a single isolated real number has zero probability. Between any two real numbers, there are uncountably infinite points. Thus, we define the Probability Density Function (PDF), denoted $f_X(x)$.

Worked Example: Continuous Uniform $U(0, 4)$

Let $X$ denote a random variable uniformly distributed over $[0, 4]$. Its PDF is constant:

Find the probability that $X$ falls between $1$ and $3$ ($1 \le X \le 3$):

7 Cumulative Distribution Function (CDF)

The Cumulative Distribution Function (CDF), denoted $F_X(x)$, unifies both discrete and continuous domains. It expresses the probability that $X$ takes a value less than or equal to a threshold $x$:

4 Universal Properties of Any Valid CDF:

• Monotonically Non-decreasing: If $a \lt b$, then $F_X(a) \le F_X(b)$. A cumulative curve can never slope downward.
• Bounded Range: $0 \le F_X(x) \le 1$ for all real $x$.
• Left Asymptote: $\lim_{x \to -\infty} F_X(x) = 0$.
• Right Asymptote: $\lim_{x \to +\infty} F_X(x) = 1$.

8 Expected Value: Mathematical Expectation $\mathbb{E}[X]$

The probability-weighted average of all conceivable values of a random variable is termed its Expected Value (or expectation), denoted $\mathbb{E}[X]$ or $\mu$.

Die Roll Expectation: Why $3.5$ is Meaningful

For a fair die:

Notice that $3.5$ can never appear on a single die roll. The expected value represents the long-run theoretical center of mass: average the results of $100,000$ rolls, and the empirical mean will converge precisely toward $3.5$.

9 Variance: Measuring Spread Around the Center

While the expected value specifies the central balance point, Variance quantifies how widely dispersed the outcomes are around that mean:

Because squared deviations cannot be negative, variance is strictly non-negative ($\operatorname{Var}(X) \ge 0$).

Squaring deviations changes the unit of measurement (e.g., milliseconds become $\text{ms}^2$). Taking the positive square root restores original physical units as Standard Deviation ($\sigma$):

Coin Toss Variance Calculation

Let $X = 1$ for Heads ($p=0.5$) and $X = 0$ for Tails ($p=0.5$):

• $\mathbb{E}[X] = 1(0.5) + 0(0.5) = 0.5$
• $\mathbb{E}[X^2] = 1^2(0.5) + 0^2(0.5) = 0.5$
• $\operatorname{Var}(X) = 0.5 - (0.5)^2 = 0.5 - 0.25 = 0.25 = \frac{1}{4}$
• $\sigma = \sqrt{0.25} = 0.5$

Scaling Property: $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$. Adding a constant shifts position without altering spread; multiplying by a scalar scales variance quadratically.

10 5 Essential Probability Distributions in CS & AI

Across data science and computing, five canonical probability distributions serve as the primary mathematical building blocks:

11 Master Distribution Catalog

12 Real-World Applications in Computing & AI

13 Practical Python Implementation with SciPy

Python's `scipy.stats` module offers vectorized implementations of probability mass, density, and cumulative distributions:

import numpy as np
from scipy import stats

# 1. Binomial: Exactly 5 heads in 10 tosses (n=10, p=0.5)
n, p = 10, 0.5
p_5_heads = stats.binom.pmf(5, n, p)
print(f"P(X=5 heads) = {p_5_heads:.4f}")  # 0.2461

# 2. Poisson: Exactly 3 requests when lambda = 2
p_3_requests = stats.poisson.pmf(3, mu=2)
print(f"P(X=3 requests) = {p_3_requests:.4f}")  # 0.1804

# 3. Normal: Standard Normal Z ~ N(0, 1), P(-1 <= Z <= 1)
prob_within_1_sigma = stats.norm.cdf(1, loc=0, scale=1) - stats.norm.cdf(-1, loc=0, scale=1)
print(f"P(-1 <= Z <= 1) = {prob_within_1_sigma:.4f}")  # 0.6827 (68.3%)

# 4. Monte Carlo Verification of the Empirical Rule
samples = np.random.normal(loc=100, scale=15, size=100000)
within_1_std = np.mean((samples >= 85) & (samples <= 115))
print(f"Empirical 1-Sigma Fraction = {within_1_std * 100:.2f}%")

14 8 Common Pitfalls & Traps to Avoid

16 Key Takeaways