Logistic Regression: Classification from Probabilities

1. Introduction: From Numerical Estimation to Categorical Decision-Making

In the preceding lessons, we mastered Linear Regression and Polynomial Regression. Both algorithms solve regression problems where the target output $y$ is a continuous numerical quantity—such as apartment price (BDT Lakh) or vehicle braking distance (meters).

However, a massive proportion of industrial machine learning problems do not demand continuous numbers. Instead, they require making categorical decisions between discrete choices:

• E-commerce Conversion: Will a visitor browsing laptops complete a checkout or abandon the cart? ($\text{Buy} = 1, \text{No Buy} = 0$)
• Cybersecurity: Is an incoming email legitimate communication or malicious phishing? ($\text{Spam} = 1, \text{Ham} = 0$)
• Fintech Fraud Detection: Is a BDT 50,000 credit card transaction authentic or fraudulent? ($\text{Fraud} = 1, \text{Authentic} = 0$)
• Healthcare Diagnostics: Does a patient biopsy scan indicate malignant tumor pathology or benign tissue? ($\text{Malignant} = 1, \text{Benign} = 0$)

These challenges constitute Binary Classification Problems. Despite containing the historical word "Regression" in its name, Logistic Regression is fundamentally the foundational, standard classification algorithm in machine learning. It models the conditional probability $P(y = 1 \mid \mathbf{x})$ that a given observation belongs to the positive category.

2. Why Not Simple Linear Regression? The Mathematical Failure

A novice engineer might ask: "Why can't we simply encode our binary target as $0$ and $1$, and fit a standard Ordinary Least Squares linear regression line $\hat{y} = \mathbf{w}^T \mathbf{x} + b$?"

Applying standard Linear Regression to classification triggers three catastrophic mathematical and practical failures:

3. The Two-Step Engine: Linear Score & The Sigmoid Function

Logistic Regression solves classification through an elegant two-step architecture:

4. Decision Thresholds: Converting Probabilities to Hard Actions

Once the model outputs a probability estimate $\hat{p} = P(y = 1 \mid \mathbf{x})$, how does it make a binary classification decision? It compares the probability against a Decision Threshold ($\tau$):

By default, binary classifiers apply $\tau = 0.50$. If a customer's purchase probability is $52\%$, they are labeled "Buyer" ($1$); if $48\%$, "Non-buyer" ($0$).

5. Odds, Log-Odds (Logit), and Multiplicative Coefficient Interpretation

In linear regression, a weight $w_j$ directly represents the expected additive increase in $y$ per unit increase in $x_j$. In logistic regression, because the target probability is non-linear, weights are interpreted through Odds and Odds Ratios.

6. Loss Function: Binary Cross-Entropy (Log Loss) vs. Mean Squared Error

Why can't we train logistic regression by minimizing Mean Squared Error ($\sum (y_i - \hat{p}_i)^2$)?

To guarantee a convex bowl-shaped optimization surface with a unique global minimum, Logistic Regression is trained via Maximum Likelihood Estimation (MLE), formalized as Binary Cross-Entropy (BCE) or Log Loss:

7. Classification Performance Evaluation: Beyond Accuracy

In regression, evaluation is straightforward (MAE, RMSE, $R^2$). In classification, raw Accuracy is frequently misleading—particularly on imbalanced datasets.

7.1 The Confusion Matrix Blueprint

Every classification decision falls into one of four mutually exclusive quadrants:

8. Complete Scikit-Learn Python Implementation

Below is a complete, production-grade workflow modeling a synthetic imbalanced customer classification dataset (90% non-buyers, 10% buyers) with stratified partitioning, pipeline scaling, and threshold optimization.

# Step 1: Scientific & Machine Learning Imports
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score,
    f1_score, roc_auc_score, confusion_matrix, classification_report
)

# Step 2: Synthesize Imbalanced Customer Dataset (2,000 samples, 90/10 Class Ratio)
X, y = make_classification(
    n_samples=2000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    n_classes=2,
    weights=[0.90, 0.10],
    random_state=42
)

# Step 3: Stratified 70/15/15 Partitioning (Train, Validation, Test)
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, random_state=42, stratify=y_temp
)

print(f"Dataset Partitions: Train={len(X_train)}, Val={len(X_val)}, Test={len(X_test)}")

# Step 4: Construct Safe Leak-Free Preprocessing Pipeline
pipeline = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1000, random_state=42)
)

# Step 5: Fit Exclusively on Training Observations
pipeline.fit(X_train, y_train)

# Step 6: Generate Continuous Probabilities on Validation Set
y_val_probs = pipeline.predict_proba(X_val)[:, 1]

# Step 7: Hyperparameter Tuning: Optimize Decision Threshold for F1-Score
thresholds = np.arange(0.10, 0.91, 0.05)
best_thresh = 0.50
best_f1 = 0.0

for t in thresholds:
    val_preds = (y_val_probs >= t).astype(int)
    score = f1_score(y_val, val_preds, zero_division=0)
    if score > best_f1:
        best_f1 = score
        best_thresh = t

print(f"Optimized Validation Threshold: {best_thresh:.2f} (Achieved F1: {best_f1:.3f})")

# Step 8: Apply Optimal Threshold to Final Unseen Test Data
y_test_probs = pipeline.predict_proba(X_test)[:, 1]
y_test_preds = (y_test_probs >= best_thresh).astype(int)

# Step 9: Quantify Final Generalization Metrics
acc = accuracy_score(y_test, y_test_preds)
prec = precision_score(y_test, y_test_preds, zero_division=0)
rec = recall_score(y_test, y_test_preds, zero_division=0)
f1 = f1_score(y_test, y_test_preds, zero_division=0)
roc_auc = roc_auc_score(y_test, y_test_probs)

print(f"\n=== FINAL TEST EVALUATION (Threshold = {best_thresh:.2f}) ===")
print(f"Accuracy  : {acc:.4f}")
print(f"Precision : {prec:.4f}")
print(f"Recall    : {rec:.4f}")
print(f"F1-Score  : {f1:.4f}")
print(f"ROC-AUC   : {roc_auc:.4f}")

# Step 10 & 11: Confusion Matrix & Classification Report
cm = confusion_matrix(y_test, y_test_preds, labels=[0, 1])
print(f"\nConfusion Matrix:\n{cm}")
print(f"\nDetailed Classification Report:\n")
print(classification_report(y_test, y_test_preds, target_names=["Non-Buyer", "Buyer"]))

9. Production Strategies for Imbalanced Datasets

When working on severe class imbalances (e.g. 1% fraud), standard logistic regression can be optimized through three critical engineering techniques:

10. Multiclass Classification: One-vs-Rest vs. Multinomial Softmax

When expanding beyond binary choices to $K > 2$ classes (e.g. classifying user sentiment into Positive, Neutral, or Negative), Logistic Regression utilizes two primary strategies:

11. Five Core Assumptions & Architectural Limitations

12. Summary & Key Takeaways