Lesson 14.29: Descriptive Statistics, Variance & Correlation

1 Introduction: Extracting Meaning from Raw Observations

Imagine a scenario where 100 students complete an examination. After grading, you receive a raw list of scores. As a data scientist or educational administrator, you need immediate answers to critical questions:

• What was the typical performance across the cohort? What represents the central balance point?
• Are the scores tightly clustered around the average, or is there massive inequality between top and bottom performers?
• Is there an observable association between the number of hours students spent studying and their final marks?

Staring at an unorganized table of 100 numbers provides little intuition. By applying mathematical techniques to condense the dataset into a few representative parameters, we extract actionable clarity. This discipline is known as Descriptive Statistics.

In machine learning and AI, descriptive statistics constitutes the foundation of Exploratory Data Analysis (EDA), feature scaling, anomaly detection, and loss evaluation.

2 What is Descriptive Statistics? The 3 Core Pillars

Descriptive Statistics is the branch of quantitative analysis focused on summarizing, organizing, and visualizing the key empirical characteristics of a collected dataset without drawing speculative inferences beyond the data itself.

Suppose an e-commerce website tracks daily revenue (in thousands of dollars) over a 7-day period: $100, 120, 110, 150, 130, 140, 150$. To understand this performance, we analyze three universal pillars:

3 Dataset, Population & Sample (Parameters vs. Statistics)

Before computing formulas, we must establish whether our dataset represents the entire target universe or merely an observed sample:

• Population ($N$): The complete set of all possible entities under investigation (e.g., all 200 million smartphone users in a country). Metrics calculated from a population are called Parameters.
• Sample ($n$): A subset drawn via random sampling representing the broader population (e.g., 5,000 randomly selected users). Metrics calculated from a sample are called Statistics.

Crucial observation: Sample variance divides by $n - 1$ instead of $n$. This adjustment is known as Bessel's Correction, detailed in Section 8.

4 Measures of Central Tendency

4.1 Mean (Arithmetic Average)

The sum of all values divided by total count. For a sample of size $n$:

For test scores $[60, 70, 80, 90, 100]$, the sample mean is $\bar{x} = \frac{400}{5} = 80$.

4.2 Median: The Robust Central Value

The middle observation when data is sorted in ascending order.

• Odd $n$: Exactly the middle element at position $\frac{n+1}{2}$. Example: $[10, 20, 30, 40, 50] \implies \text{Median} = 30$.
• Even $n$: Average of the two central elements. Example: $[10, 20, 30, 40] \implies \text{Median} = \frac{20 + 30}{2} = 25$.

4.3 Mode (Most Frequent Value)

The value that appears with the highest frequency in the dataset.

• $[2, 3, 3, 4, 5, 5, 5, 7] \implies \text{Mode} = 5$ (Unimodal).
• $[1, 1, 2, 2, 3] \implies \text{Modes} = 1 \text{ and } 2$ (Bimodal).
• $[1, 2, 3, 4] \implies \text{No mode}$ (Uniform frequency).

Mode is the only measure of central tendency applicable to categorical/discrete text data (e.g., most purchased shoe size or dominant classification class).

5 Measures of Dispersion: Why Spread Matters

Central tendency alone fails to capture the true distribution of data. Consider two classroom cohorts:

Both cohorts share the identical mean ($50$), yet their educational dynamics are polar opposites. Dispersion metrics quantify this variation.

6. Range

The difference between highest and lowest observations: $\text{Range} = x_{\max} - x_{\min}$.

While simple, Range relies exclusively on two extremes and ignores all intermediate structure, making it notoriously fragile to noise.

7 Quartiles & Interquartile Range (IQR) — Foundation of Box Plots

Dividing an ordered dataset into four equal segments produces three division cutoffs called Quartiles:

• $Q_1$ (First Quartile / 25th Percentile): Boundary below which $25\%$ of observations fall.
• $Q_2$ (Second Quartile / 50th Percentile): Exactly identical to the Median.
• $Q_3$ (Third Quartile / 75th Percentile): Boundary below which $75\%$ of observations fall.

The span between the 75th and 25th percentiles is the Interquartile Range (IQR):

IQR represents the spread of the central 50% of the dataset and is entirely impervious to extreme outliers.

8 Variance: Average Squared Deviation & Bessel's Correction

Because unsquared deviations around the mean sum to zero ($\sum (x_i - \bar{x}) = 0$), we square each deviation before averaging:

Step-by-Step Calculation: $[2, 4, 4, 4, 5, 5, 7, 9]$

Count $n = 8$, Mean $\bar{x} = \frac{40}{8} = 5$.

• If considered a Population: $\sigma^2 = \frac{32}{8} = 4$
• If considered a Sample: $s^2 = \frac{32}{8 - 1} = \frac{32}{7} \approx 4.5714$

9 Standard Deviation: Returning to Physical Units

Variance produces squared units (e.g., $\text{ms}^2$ or $\$^2$). To restore the dispersion measurement back into original physical units, we take the non-negative square root:

From our previous problem, Population Standard Deviation is $\sigma = \sqrt{4} = 2$, and Sample Standard Deviation is $s = \sqrt{4.5714} \approx 2.138$.

10 Coefficient of Variation (CV): Relative Dispersion

When comparing datasets measured in different physical units or with vastly different mean magnitudes, standard deviation alone can be misleading. We normalize dispersion relative to the mean via the Coefficient of Variation (CV):

Machine A ($\text{Mean}=100, s=10 \implies \text{CV}=10\%$) and Machine B ($\text{Mean}=200, s=20 \implies \text{CV}=10\%$) possess identical relative variability despite differing raw spreads.

11 Covariance: Measuring Joint Variability

While variance measures the dispersion of a single feature, Covariance determines the directional linear co-movement between two random variables $X$ and $Y$:

12 Correlation: Pearson Correlation Coefficient ($r$)

Normalizing covariance by the product of both standard deviations eliminates scale dependency, yielding the Pearson Correlation Coefficient ($r$):

$r$ is strictly bounded on the closed interval $[-1, +1]$:

Worked Problem: Study Hours ($X$) vs. Score ($Y$)

5 students: $X = [1, 2, 3, 4, 5]$ and $Y = [50, 55, 65, 60, 70]$.
Means: $\bar{x} = 3, \bar{y} = 60$. Sums: $\sum (x-\bar{x})(y-\bar{y}) = 45$, $\sum (x-\bar{x})^2 = 10$, $\sum (y-\bar{y})^2 = 250$.

$r = 0.90$ confirms a strong positive linear correlation between study duration and examination score.

13 Correlation vs. Causation: The Golden Rule

The Classic Confounder: Ice cream sales and drowning rates spike simultaneously in July ($r \approx 0.85$). Does eating ice cream induce drowning? Obviously not. The true driver is a third hidden variable—Summer Outdoor Temperature (Confounder). Heat causes both higher ice cream consumption and more frequent swimming excursions.

In machine learning, models frequently latch onto spurious correlations. Establishing true causality requires controlled experimentation (A/B testing) or causal graphical modeling.

14 Pearson vs. Spearman Rank Correlation

Real-world relationships are not always linear; many are curved yet monotonically increasing or decreasing. For non-linear monotonic curves, we employ Spearman's Rank Correlation ($\rho$):

15 Practical Python Implementation with Pandas & SciPy

A standard Python EDA pipeline computing summary metrics, Tukey outlier bounds, and correlation coefficients:

import numpy as np
import pandas as pd
from scipy import stats

# Dataset creation
hours = [1, 2, 3, 4, 5, 2, 4, 3]
scores = [50, 55, 65, 60, 70, 52, 68, 62]
df = pd.DataFrame({"hours": hours, "scores": scores})

# 1. Fundamental summary statistics
mean_val = df["scores"].mean()
median_val = df["scores"].median()
std_sample = df["scores"].std(ddof=1)  # Bessel's correction n-1

# 2. Quartiles and IQR
q1 = df["scores"].quantile(0.25)
q3 = df["scores"].quantile(0.75)
iqr = q3 - q1

# 3. Tukey Outlier Filtering
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr
outliers = df[(df["scores"] < lower_bound) | (df["scores"] > upper_bound)]

# 4. Pearson & Spearman Correlation
pearson_r, _ = stats.pearsonr(df["hours"], df["scores"])
spearman_rho, _ = stats.spearmanr(df["hours"], df["scores"])

print(f"Mean: {mean_val:.2f}, Median: {median_val}, Std: {std_sample:.2f}")
print(f"IQR: {iqr:.2f} [Fences: {lower_bound} to {upper_bound}]")
print(f"Pearson r: {pearson_r:.4f} | Spearman rho: {spearman_rho:.4f}")

16 8 Common Pitfalls & Traps to Avoid

21 Key Takeaways