Lesson 14.30: Sampling, Estimation & Hypothesis Testing

1 Introduction: Inferring Big Truths from Small Data

Imagine a consumer fintech or e-commerce platform that wants to determine what percentage of its 2 million registered users prefer a newly redesigned mobile checkout flow. Polling every individual customer through a complete census would cost tens of thousands of dollars and months of delay.

Instead, if the engineering team selects a randomized sample of 1,000 customers, that compact dataset can provide remarkably precise insights into the entire 2-million-user population.

However, making rigorous inferences requires answering four fundamental statistical challenges:

• Representativeness: How should the 1,000 users be selected so that the sample is truly representative and free from systematic bias?
• Estimation: How do we compute an estimate of the true unknown population parameter from sample measurements?
• Quantifying Uncertainty: How close is our sample estimate to the actual population truth, and what is our margin of error?
• Evaluating Claims: Does an observed 3% conversion bump provide genuine evidence of superiority, or is it merely random statistical noise?

2 Population, Sample, Parameter & Statistic

Every statistical analysis operates on two conceptual planes: the target universe we wish to understand, and the observable subset we have in hand.

The numerical measurements derived from these domains carry fundamentally different terminology and notations:

3 What is Sampling and Why is it Critical?

Sampling is the scientific protocol of choosing a subset from a population such that characteristics observed in the subset can be safely generalized back to the population. A complete census is rarely viable in practical engineering and research due to four real-world constraints:

4 Core Sampling Methodologies

Sampling designs divide strictly into Probability Sampling (where every element has a known, non-zero probability of selection) and Non-probability Sampling (where selection probabilities are unknown or subjective).

5 Sampling Bias vs. Sampling Error

Discrepancies between sample statistics and population parameters stem from two entirely distinct mechanisms:

6 What is a Sampling Distribution?

Suppose you draw a random sample of $n = 30$ rows from a database and compute its mean $\bar{x}_1$. Then you draw another 30 rows and compute $\bar{x}_2$. If you repeat this process 100,000 times, you produce 100,000 distinct sample means.

The probability distribution formed by plotting all these potential sample statistics is defined as the Sampling Distribution of the Mean.

7 Central Limit Theorem (CLT)

The Central Limit Theorem is the crowning achievement of modern probability and data science.

8 Standard Deviation vs. Standard Error

Confusing Standard Deviation with Standard Error is a widespread mistake. Here is their decisive comparison:

9 Estimation: Point Estimation vs. Interval Estimation

Estimation is the discipline of using sample statistics to formulate educated guesses about unknown population parameters. It takes two forms:

10 Confidence Intervals (CI) Deep Dive

A Confidence Interval is an estimated range of values computed from sample observations, characterized by a designated confidence level (such as 90%, 95%, or 99%).

11 Confidence Intervals for Categorical Proportions

For binary outcomes (clicks vs no-clicks, pass vs fail, converts vs churns), we measure proportion $p$ rather than numerical averages:

12 What is Hypothesis Testing?

Hypothesis Testing is a formal inferential procedure that uses sample data to determine whether there is sufficient empirical evidence to reject a specified baseline claim.

It parallels the legal principle of "presumed innocent until proven guilty": we assume the baseline status quo is true unless the data demonstrates overwhelming contradictory evidence.

13 The 5-Step Hypothesis Testing Workflow

Every hypothesis test follows five standardized sequential steps:

14 Demystifying the P-value

The exact mathematical definition of P-value: "Assuming the Null Hypothesis is completely true, what is the probability of observing data at least as extreme as what we just measured?"

15 Complete Worked Example: Testing a Fair Coin

Suppose you suspect a coin is biased. You flip it 100 times and observe 58 heads. At significance level $\alpha = 0.05$, is the coin biased?

16 Type I & Type II Errors and Statistical Power

Every statistical decision faces two distinct failure modes:

17 Statistical Significance vs. Practical Significance

In the era of Big Data, confusing statistical significance with practical value is a catastrophic engineering blunder.

18 Catalog of Standard Statistical Tests

19 Practical Python Implementation: NumPy & SciPy

How to execute sampling, confidence intervals, and hypothesis tests in production Python workflows:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
population = np.arange(1, 1001)

# Simple Random Sampling without replacement
sample = rng.choice(population, size=100, replace=False)

print(f"Sample Mean: {np.mean(sample):.2f}")
print(f"Sample Variance (ddof=1): {np.var(sample, ddof=1):.2f}")
print(f"Sample Std Dev: {np.std(sample, ddof=1):.2f}")

from scipy import stats

sample = np.array([45, 48, 50, 52, 55, 47, 51, 49, 53, 50])
n = len(sample)
mean = np.mean(sample)
std_err = stats.sem(sample) # s / sqrt(n)

# 95% CI with df = n - 1
ci = stats.t.interval(0.95, df=n - 1, loc=mean, scale=std_err)
print(f"Mean: {mean:.2f}")
print(f"95% Confidence Interval: [{ci[0]:.2f}, {ci[1]:.2f}]")

# Null Hypothesis: Process mean is 50 (popmean=50)
t_stat, p_val = stats.ttest_1samp(sample, popmean=50)

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_val:.4f}")

alpha = 0.05
if p_val <= alpha:
    print("Reject H0: Significant difference from 50")
else:
    print("Fail to Reject H0: Insufficient evidence to reject 50")

20 Real-World Applications in Computer Science & AI

21 8 Common Pitfalls & Traps

22 Key Takeaways