Naive Bayes: Probabilistic Classification

1. Introduction: From Distance to Probability

In the previous lesson, we explored K-Nearest Neighbors (KNN), where classification and regression were determined by measuring geometric distances between unlabelled query points and stored training observations. While intuitive, distance-based methods become computationally burdensome at scale ($O(N \cdot d)$ per query) and struggle in ultra-high-dimensional feature spaces due to the Curse of Dimensionality.

Now, we transition from geometric space to probabilistic hypothesis space with Naive Bayes. Instead of asking “How far is point $A$ from point $B$?”, Naive Bayes asks: “Given the exact set of features observed in this input, what is the statistical probability that it belongs to Class $y$?”

Consider an automated email spam filtering system. Every day, mail servers process millions of messages. Some contain urgent server alerts or client invoices, while others are fraudulent phishing scams or marketing spam. When an email arrives containing tokens such as:

If historical logs demonstrate that these tokens appear with overwhelming frequency inside spam messages but rarely in legitimate correspondence, our system can systematically compute the posterior probability that the new message is spam.

Naive Bayes is a generative probabilistic supervised learning algorithm rooted in Bayes' Theorem. It combines prior population baselines with observed feature evidence under a simplifying assumption of conditional independence between features.

Primary Industry Applications

2. Probability & Conditional Probability Foundations

To master Naive Bayes, we must first establish rigorous definitions for probability, joint probability, and conditional probability.

In Kolmogorov probability theory, the probability of an event $A$, denoted $P(A)$, is a real number bounded strictly between $0$ (impossible event) and $1$ (certain event):

Suppose an enterprise mail gateway inspects a sample of $N = 1,000$ emails. Historical audit reveals that $200$ emails are spam and $800$ emails are legitimate ("ham"). The unconditional (prior) probability of encountering a spam email is:

2.1 Conditional Probability: Formal Definition

Conditional probability measures the probability of event $A$ occurring given the prior knowledge that event $B$ has already occurred. Mathematically, it is denoted as $P(A \mid B)$ (read: “probability of $A$ given $B$”):

Here, $P(A \cap B)$ represents the joint probability that both event $A$ and event $B$ occur simultaneously, while $P(B)$ is the marginal probability of condition $B$.

3. Bayes' Theorem Deconstructed

The core intellectual engine of Naive Bayes is Bayes' Theorem, formulated by the English statistician and Presbyterian minister Thomas Bayes in 1763. It provides a mathematically rigorous method for updating existing beliefs in the light of new evidence.

From the multiplication rule of conditional probability, we know that the joint probability $P(A \cap B)$ can be expanded in two equivalent ways:

Dividing both sides by $P(B)$ immediately yields Bayes' Theorem:

The Four Pillars of Bayes' Theorem

3.1 Step-by-Step Spam Detection Calculation

Let us trace how Bayes' Theorem operates under our previous email scenario:

1. Prior Probability: $P(\text{Spam}) = 0.20$, and $P(\text{Ham}) = 0.80$.
2. Spam Likelihood: Among $200$ spam emails, $80$ contain "free" $\implies P(\text{free} \mid \text{Spam}) = \frac{80}{200} = 0.40$.
3. Ham Likelihood: Among $800$ ham emails, $20$ contain "free" $\implies P(\text{free} \mid \text{Ham}) = \frac{20}{800} = 0.025$.
4. Evidence Calculation (Law of Total Probability): $$\begin{aligned} P(\text{free}) &= P(\text{free} \mid \text{Spam}) \cdot P(\text{Spam}) + P(\text{free} \mid \text{Ham}) \cdot P(\text{Ham}) \\ &= (0.40 \times 0.20) + (0.025 \times 0.80) \\ &= 0.080 + 0.020 = 0.100 \end{aligned}$$
5. Posterior Computation: $$P(\text{Spam} \mid \text{free}) = \frac{P(\text{free} \mid \text{Spam}) \cdot P(\text{Spam})}{P(\text{free})} = \frac{0.40 \times 0.20}{0.100} = \frac{0.080}{0.100} = 0.80 \quad (80\%)$$

Notice how the posterior synthesizes two distinct factors: the base rate (prior) and the diagnostic power (likelihood) of the feature. Even though legitimate emails are four times more common in the inbox ($80\%$ vs $20\%$), the high diagnostic ratio of the word "free" ($\frac{0.40}{0.025} = 16$) overwhelms the prior.

4. The Naive Conditional Independence Assumption

In real-world applications, classification is rarely decided by a single feature. An email contains an arbitrary vector of $d$ words: $\mathbf{x} = (x_1, x_2, \dots, x_d)^T$.

Applying Bayes' Theorem directly to an entire feature vector yields:

Computing the joint likelihood term $P(x_1, x_2, \dots, x_d \mid y)$ presents a catastrophic obstacle known as the Combinatorial Explosion of Parameters. If we have $d = 1,000$ binary features (e.g., vocabulary words present or absent), there are $2^{1000} \approx 10^{301}$ possible feature combinations. To estimate this distribution empirically, we would require more training observations than the total number of elementary particles in the observable universe!

4.1 Is This Assumption True in Practice?

In virtually all real-world applications, the conditional independence assumption is mathematically false. In natural language, words exhibit intense semantic correlations:

• The word "prize" is strongly correlated with "claim" and "lottery".
• The word "Hong" is almost always followed by "Kong".

Yet, despite violating its foundational premise, Naive Bayes performs remarkably well in practice. Theoretical research (Domingos & Pazzani, 1997) revealed the mathematical explanation:

5. Mathematical Formulation & Log-Space Computation

Combining Bayes' Theorem with the Conditional Independence assumption yields the full Naive Bayes classification formula:

Notice that the denominator (the evidence $P(\mathbf{x})$) is constant for all candidate classes $y$ for a given input query. Therefore, to determine the most probable class, we can drop the normalizer and apply the Maximum A Posteriori (MAP) decision rule:

5.1 The Threat of Numerical Underflow & Log-Space Arithmetic

When deploying Naive Bayes to classify realistic text documents containing $d = 2,000$ words, each individual likelihood $P(x_i \mid y)$ is a tiny fraction (e.g. $10^{-3}$ or $10^{-4}$).

Multiplying hundreds of small decimal fractions causes standard 64-bit IEEE 754 floating-point numbers to exceed their minimum exponent threshold ($\approx 10^{-308}$). The values collapse catastrophically to zero ($0.0$) — a catastrophic hardware failure known as Numerical Underflow. Once the product becomes zero, all class comparisons result in $0 > 0$, causing model predictions to fail arbitrarily.

6. Complete Numerical Walkthrough: The "free win" Email

To cement our theoretical understanding, let us execute a complete manual calculation without software shortcuts on a concrete training dataset of $100$ emails:

Now, an incoming unlabelled email arrives containing the text: “claim free win”. We seek to evaluate the features $x_1 = \text{"free"}$ and $x_2 = \text{"win"}$.

7. The Five Core Variants of Naive Bayes

While the master Bayesian framework remains identical, the exact mathematical formula used to compute the likelihood term $P(x_i \mid y)$ varies depending on the statistical distribution of our input features. Scikit-Learn implements five distinct variants:

Model Selection Guide: Which Variant Should You Choose?

8. Laplace & Lidstone Additive Smoothing

A notorious structural vulnerability of Naive Bayes is the Zero-Frequency Problem (also termed the Black Swan Catastrophe).

Suppose an incoming email contains the word "lottery". If in our entire historical training dataset of $10,000$ spam messages, the word "lottery" happened to appear zero times ($N_{\text{Spam}, \text{lottery}} = 0$), the raw empirical frequency estimator yields:

Because Naive Bayes multiplies all individual likelihoods together, that single zero immediately propagates across the entire mathematical chain:

Even if the email contained twenty other unmistakable indicators of spam ("claim free money wire cash immediately"), the single unseen word completely annihilates all other evidence, rendering the spam posterior exactly zero!

8.1 The Additive Smoothing Formulation

To prevent this fatal vulnerability, Pierre-Simon Laplace formulated Additive Smoothing (pseudocount smoothing). We augment the numerator by a constant $\alpha$ and increase the denominator by $\alpha \times V$, where $V$ is the total vocabulary cardinality:

9. Production Scikit-Learn Pipelines

Now, let us construct two production-ready Python pipelines using Scikit-Learn: first, a Gaussian Naive Bayes model on continuous numerical data, and second, an industrial text classification pipeline with hyperparameter grid search.

Pipeline 1: Gaussian Naive Bayes on Numerical Data

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

# 1. Load benchmark continuous dataset
iris = load_iris()
X, y = iris.data, iris.target

# 2. Stratified Train/Test Partition (80/20)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

# 3. Instantiate GaussianNB with custom variance smoothing floor
gnb = GaussianNB(var_smoothing=1e-9)

# 4. Stratified 5-Fold Cross-Validation Audit
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(gnb, X_train, y_train, cv=cv, scoring='accuracy')
print(f"5-Fold CV Accuracy: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")

# 5. Fit on complete training split
gnb.fit(X_train, y_train)

# 6. Evaluation on held-out test set
y_pred = gnb.predict(X_test)
print(f"Test Set Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print("\nClassification Report:\n", classification_report(y_test, y_pred, target_names=iris.target_names))

Pipeline 2: Industrial Text Classification with Grid Search

In enterprise NLP pipelines, text vectorizers must always be wrapped in an immutable Scikit-Learn Pipeline to eliminate data leakage between train and test splits:

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB, ComplementNB
from sklearn.model_selection import GridSearchCV

# 1. Synthetic Enterprise Email Corpus
emails = [
    "Win a free cash prize now claim reward",
    "Claim your exclusive free bonus offer today",
    "Congratulations you won luxury vacation tickets claim now",
    "Special wire transfer offer verify bank account immediately",
    "Meeting rescheduled to Friday afternoon please confirm",
    "Review attached quarterly financial revenue report",
    "Project deployment scheduled for midnight release",
    "Invoice for client subscription attached kindly process payment"
]
labels = [1, 1, 1, 1, 0, 0, 0, 0]  # 1 = Spam, 0 = Ham

# 2. Leak-Free Pipeline Encapsulation
nlp_pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, stop_words='english')),
    ('nb', MultinomialNB())
])

# 3. Hyperparameter Tuning Grid
param_grid = {
    'tfidf__ngram_range': [(1, 1), (1, 2)],  # Unigrams vs Bigrams
    'tfidf__sublinear_tf': [True, False],     # Logarithmic term frequency
    'nb__alpha': [0.01, 0.1, 0.5, 1.0, 2.0]  # Additive smoothing search
}

grid_search = GridSearchCV(
    estimator=nlp_pipeline,
    param_grid=param_grid,
    cv=3,
    scoring='f1',
    n_jobs=-1
)
grid_search.fit(emails, labels)

print(f"Optimal Hyperparameters: {grid_search.best_params_}")
best_model = grid_search.best_estimator_

# 4. Out-of-Sample Inference
query_email = ["Urgent: claim your free cash prize today!"]
pred_class = best_model.predict(query_email)[0]
pred_probs = best_model.predict_proba(query_email)[0]

print(f"Classification: {'SPAM' if pred_class == 1 else 'HAM'}")
print(f"Probabilities -> Ham: {pred_probs[0]:.4f}, Spam: {pred_probs[1]:.4f}")

10. 9-Way Architectural Comparison Matrix

To understand where Naive Bayes fits in the wider machine learning taxonomy, let us benchmark it against its three primary peers: Logistic Regression, Support Vector Machines (SVM), and K-Nearest Neighbors (KNN):

11. Real-World Industry Best Practices