Exploratory Data Analysis (EDA)

1. Introduction: Peering Beneath the Surface of Data

When receiving a new machine learning dataset, an inexperienced practitioner often immediately jumps to importing complex algorithms like XGBoost or Deep Neural Networks and running model.fit(). However, seasoned machine learning engineers know that blind model training inevitably invites disaster.

Imagine that Byteion operates a fast-growing e-commerce platform processing thousands of daily transactions. The company has compiled a full year of sales records, product pricing, discount rates, and order fulfillment volumes. The business objective is to forecast next month's customer demand.

Before selecting any model architecture, critical empirical questions must be answered:

• Which product categories drive the highest revenue volume? During which months do sales peak?
• Is there an inverse relationship between product unit price and quantity ordered?
• Are there missing values (nulls) in crucial predictor columns?
• Do certain orders contain extreme outlier quantities (e.g. 500 units instead of 2) caused by logging bugs or bulk wholesale transactions?

The methodical, scientific practice of answering these questions before model building is known as Exploratory Data Analysis (EDA).

The ultimate goal of EDA is not merely generating aesthetic charts; rather, it is formulating an accurate mental model of the dataset that informs intelligent Data Cleaning, principled Feature Engineering, and justifiable Model Selection.

2. Why EDA is Indispensable in Machine Learning

Raw real-world data is notoriously imperfect, noisy, and prone to corruptions. Conducting systematic EDA fulfills seven primary engineering objectives:

3. The 3 Primary Analytical Levels of EDA

Depending on the number of variables examined simultaneously, exploratory analysis is partitioned into three distinct levels:

4. Initial Inspection: Auditing Dataset Health with pandas

Upon loading any tabular dataset, the first step is running an immediate triage check. pandas provides a suite of essential inspection methods:

import pandas as pd
 
df = pd.read_csv("sales.csv")
 
# 1. Dimensions and Shape (Rows, Columns)
print("Shape:", df.shape)  # e.g., (10000, 8)
 
# 2. Previewing records
print(df.head(5))
print(df.tail(3))
 
# 3. Data types, non-null counts, and memory footprint
df.info()
 
# 4. Precise missing value audit and percentages
missing_counts = df.isna().sum()
missing_percent = (df.isna().mean() * 100).round(2)
print(pd.DataFrame({"Missing Count": missing_counts, "Percent (%)": missing_percent}))

5. Univariate Analysis: Single-Variable Anatomy & Outliers

In univariate analysis, we examine individual features isolated from others. Two core mechanisms power this investigation: Descriptive Statistics and Statistical Plots.

5.1 Descriptive Statistics

Executing df.describe() calculates the foundational summary statistics for numerical attributes:

5.2 Histograms & Distribution Shapes

A Histogram groups continuous numerical values into contiguous intervals (Bins) and displays frequencies as vertical bars:

5.3 Box Plot & Tukey's IQR Outlier Detection

Pioneered by statistician John Tukey, the Box Plot encapsulates the Five-Number Summary (Min, \(Q_1\), Median, \(Q_3\), Max) and establishes the standard for outlier detection:

6. Bivariate Analysis: Examining Pairwise Interactions

Bivariate analysis investigates how two features interact and correlate across three modal combinations:

7. Correlation Analysis: Quantifying Relationship Direction & Strength

The Pearson Correlation Coefficient (\(r\)) quantifies the strength and direction of a linear relationship between two continuous variables, bounded strictly between \(-1\) and \(+1\):

8. Multivariate Analysis: Heatmaps & Multi-Feature Interactions

In real-world predictive modeling, target variables depend on composite interactions across multiple features. For instance, house prices reflect combinations of square footage, bedroom counts, school proximity, and construction age.

To visualize all pairwise relationships across an entire feature matrix at once, we compute and render a Correlation Matrix Heatmap:

import matplotlib.pyplot as plt
 
numeric_df = df.select_dtypes(include="number")
corr_matrix = numeric_df.corr(method="pearson")
 
plt.figure(figsize=(6, 5))
plt.imshow(corr_matrix, cmap="coolwarm", vmin=-1, vmax=1)
plt.colorbar(label="Pearson r")
plt.xticks(range(len(corr_matrix.columns)), corr_matrix.columns, rotation=45, ha="right")
plt.yticks(range(len(corr_matrix.columns)), corr_matrix.columns)
plt.title("Correlation Matrix Heatmap")
plt.tight_layout()
plt.show()

9. Complete Python Practical Case Study: E-Commerce Data Audit

Let us execute an end-to-end diagnostic EDA script on an e-commerce order dataset engineered with realistic data pathologies (missing values, irregular casing, and extreme outliers):

import pandas as pd
import matplotlib.pyplot as plt
 
# Step 1: Synthesize noisy real-world dataset
data = {
    "order_id": ["O01", "O02", "O03", "O04", "O05", "O06", "O07", "O08", "O09", "O10", "O11", "O12"],
    "category": ["Electronics", "Fashion", "Electronics", "Accessories", "fashion", "Electronics",
                  "Accessories", "Fashion", "Electronics", "Fashion", "Accessories", " Fashion "],
    "price": [50000, 1500, 22000, 2500, 1800, 55000, 1200, 2000, None, 1600, 1300, 2100],
    "quantity": [2, 3, 1, 4, 2, 1, 5, 2, 1, None, 50, 1],
    "discount_pct": [5, 10, 0, 5, 15, 5, 0, 10, 0, 0, 0, 5],
    "city": ["Dhaka", "Dhaka", "Chattogram", "Dhaka", "Khulna", "Dhaka", "Rajshahi", "Chattogram", "Khulna", "Dhaka", "Dhaka", "Sylhet"]
}
df = pd.DataFrame(data)
df["order_date"] = pd.to_datetime(["2026-01-02", "2026-01-03", "2026-01-04", "2026-01-05", "2026-01-06", "2026-01-07",
                                   "2026-01-08", "2026-01-09", "2026-01-10", "2026-01-11", "2026-01-12", "2026-01-13"])
 
# Step 2: Diagnostic health check printouts
print("--- Missing Values ---")
print(df.isna().sum())
 
print("\n--- Inconsistent Categorical Strings ---")
print(df["category"].value_counts(dropna=False))
 
print("\n--- Numerical Summary (Outlier Alert) ---")
print(df[["price", "quantity", "discount_pct"]].describe())

10. 7 Critical Pitfalls Engineers Commit During EDA

11. Summary of Key Learnings