Deep Learning for Computer Vision: CNNs, Convolution & Image Classification
In previous lessons, we mastered fully connected Multi-Layer Perceptrons (MLPs) and built training pipelines using PyTorch. However, when processing spatial grid data like digital images, standard dense networks suffer from catastrophic parameter explosion, memory starvation, and spatial amnesia. In this lesson, we explore Convolutional Neural Networks (CNNs)—the bedrock of computer vision—mastering 2D discrete convolution, kernel filters, spatial dimension equations, pooling mechanics, hierarchical feature extraction, and implementing an end-to-end PyTorch image classifier.
What You Will Master
- Digital Image Geometry: Grayscale vs RGB, uint8 intensities, and PyTorch NCHW tensor layout.
- The Curse of Dense MLPs: Parameter explosion and why weight sharing and local receptive fields solve it.
- 2D Convolution Math: Cross-correlation operations, Sobel/Laplacian kernels, and multi-channel feature maps.
- Spatial Dimension Equations: Mathematical derivations of Kernel Size, Stride ($S$), Padding ($P$), and Dilation.
- Downsampling via Pooling: Max Pooling vs Average Pooling vs Strided Convolutions.
-
PyTorch CNN Architecture: Composing
nn.Conv2d,nn.MaxPool2d,nn.Dropout, and classifier heads. - End-to-End Pipeline: Training and evaluating on MNIST/Fashion-MNIST with confusion matrix diagnosis.
- Regularization & Transfer Learning: Data augmentation, BatchNorm, and pre-trained ResNet paradigms.
1. The Computer Vision Problem: Why Dense MLPs Fail on Images
Consider teaching a computer to recognize whether a photograph contains a cat, a dog, or a handwritten digit. To a human, visual comprehension is instantaneous and holistic. To a digital computer, an image is merely a two-dimensional grid of raw numerical intensities ranging from $0$ to $255$.
In previous lessons, we connected input features directly to hidden layers using dense matrix multiplications (nn.Linear). Why can we not simply flatten an image into a 1D vector and feed it to a standard Multi-Layer Perceptron?
A modest $224 \times 224$ RGB image has $224 \times 224 \times 3 = 150,528$ input features. Connecting this input to a single hidden layer with only $1,000$ neurons requires:
Over $150$ million parameters in just one layer, requiring $\gt 600$ MB VRAM before calculating activations or gradients, guaranteeing catastrophic overfitting.
Images possess fundamental spatial locality: adjacent pixels are intimately correlated, forming continuous contours, edges, and textures. Flattening an image into a 1D vector ($H \times W \times C \to D$) tears apart 2D vertical adjacency. A pixel at $(x, y)$ loses its relationship with $(x, y+1)$.
A cat in the top-left corner must be recognized as a cat just as easily as when it appears in the bottom-right corner. In an MLP, weights connected to top-left inputs learn nothing about bottom-right inputs. The network must re-learn the visual concept of a cat at every distinct coordinate!
2. How Digital Images are Represented Mathematically
A digital image is fundamentally a discrete 2D spatial function $f(x, y)$, sampled onto an integer coordinate grid of picture elements (pixels). Each pixel holds one or more numerical intensity values.
Represented as a 2D matrix of shape $(H, W)$. Each pixel stores an 8-bit unsigned integer value between $0$ (pure black) and $255$ (pure white):
Represented as a 3D volume of shape $(3, H, W)$. Every pixel position $(x, y)$ contains a 3-element vector $[R, G, B]$ representing red, green, and blue photon intensities:
2.1 The PyTorch NCHW Tensor Standard
In PyTorch, all computer vision operators (nn.Conv2d, nn.MaxPool2d) expect mini-batches formatted in NCHW ordering:
Standard image processing libraries (OpenCV, Pillow, Matplotlib, TensorFlow) store images in HWC format ($[H, W, C]$). Passing an HWC NumPy array directly into PyTorch layers results in catastrophic shape mismatch crashes!
tensor_img = torch.from_numpy(np_img).permute(2, 0, 1).unsqueeze(0).float() / 255.0
2.2 Pixel Normalization & Channel Standardization
Raw pixel values $[0, 255]$ are poorly conditioned for gradient descent. High positive inputs saturate Sigmoid/Tanh activations and cause unstable weight oscillations. We apply two standard preprocessing transformations:
-
Range Normalization: Scales unsigned 8-bit integers to floating point values in $[0.0, 1.0]$ via $x_{\text{norm}} = \frac{x}{255.0}$. PyTorch's
transforms.ToTensor()handles this automatically. - Per-Channel Standardization: Zero-centers the features to mean $\mu = 0$ and unit variance $\sigma = 1$: $$x_{\text{std}} = \frac{x_{\text{norm}} - \mu}{\sigma}$$ For ImageNet-pretrained networks, the canonical parameters are: $$\mu = [0.485, 0.456, 0.406], \quad \sigma = [0.229, 0.224, 0.225]$$
3. Understanding the 2D Convolution Operation
The central mathematical engine of a CNN is the convolutional filter (or kernel). A kernel is a small matrix of learnable weights (typically $3 \times 3$ or $5 \times 5$) that slides across the input image grid. At every valid spatial coordinate, it performs element-wise scalar multiplications with the overlapping image patch and sums them up into a single scalar in the output feature map.
Consider a single-channel $3 \times 3$ input patch $X$ and a $2 \times 2$ kernel $K$:
- Position (0, 0): $(1 \times 1) + (2 \times 0) + (4 \times 0) + (5 \times -1) = 1 - 5 = \mathbf{-4}$
- Position (0, 1): $(2 \times 1) + (3 \times 0) + (5 \times 0) + (6 \times -1) = 2 - 6 = \mathbf{-4}$
- Position (1, 0): $(4 \times 1) + (5 \times 0) + (7 \times 0) + (8 \times -1) = 4 - 8 = \mathbf{-4}$
- Position (1, 1): $(5 \times 1) + (6 \times 0) + (8 \times 0) + (9 \times -1) = 5 - 9 = \mathbf{-4}$
The resulting output feature map $Y$ has dimension $2 \times 2$:
Mathematical Note: In rigorous signal processing, convolution requires flipping the kernel by $180^\circ$ before computing the inner product: $Y[i, j] = \sum X[i-u, j-v] K[u, v]$. In deep learning libraries (including PyTorch nn.Conv2d), the kernel is not flipped; mathematically, it computes cross-correlation. Because kernel weights are learned via backpropagation, the learned filter simply reflects the unflipped representation, rendering the distinction immaterial in practice.
3.1 Multi-Channel Convolution Mechanics
In real-world networks, inputs have multiple channels (e.g., $C_{\text{in}} = 3$ for RGB, or $C_{\text{in}} = 64$ in deeper layers). A single filter is therefore not a 2D matrix, but a 3D tensor of shape $(C_{\text{in}}, K_h, K_w)$:
For each output channel $o$, 2D convolutions are computed independently across all $C_{\text{in}}$ channels and summed element-by-element, followed by the addition of a learned scalar bias $b_o$. Thus, a layer producing $C_{\text{out}}$ feature maps requires $C_{\text{out}}$ distinct 3D filters.
4. Kernel Size, Stride, Padding & Output Geometry
The spatial geometry of the output feature map is controlled by three fundamental hyperparameters:
The spatial extent of the sliding window. Modern vision architectures almost universally use $3 \times 3$ kernels. Stacking two $3 \times 3$ convolutions provides the exact same $5 \times 5$ receptive field as a single $5 \times 5$ conv, but uses $2 \times (3^2) = 18$ parameters instead of $25$ ($28\%$ fewer weights!) and injects two non-linear ReLU activations instead of one.
The step size in pixels that the kernel shifts across each axis.
- $S = 1$: Moves 1 pixel at a time, dense sampling.
- $S = 2$: Skips every other pixel, downsampling spatial dimensions by $\approx 2\times$. Often replaces pooling in modern networks (ResNet).
Border pixels added around the perimeter of the input tensor, typically filled with zeros (zero-padding).
- Valid ($P=0$): No padding; dimensions shrink every layer.
- Same ($P = \lfloor K/2 \rfloor$): Preserves exact input spatial dimensions when $S=1$. Essential for building deep networks without border truncation.
Given an input with height $H_{\text{in}}$, padding $P$, dilation $D$, kernel size $K$, and stride $S$, the output height $H_{\text{out}}$ is governed by:
For standard convolutions where dilation $D = 1$:
4.1 Parameter Counting Equation in Conv2d
The total number of learnable parameters in a standard 2D convolutional layer is determined strictly by channel count and kernel size, independent of image resolution:
For example: a layer with $C_{\text{in}} = 3$ (RGB), $C_{\text{out}} = 32$, and $K = 3 \times 3$: $$\text{Params} = (32 \times 3 \times 3 \times 3) + 32 = 864 + 32 = \mathbf{896} \text{ parameters!}$$ Whether this layer processes a tiny $28 \times 28$ digit or an ultra-HD $4096 \times 2160$ video frame, it has exactly 896 parameters.
5. Pooling Layers: Downsampling & Translation Tolerance
As we progress deeper into a CNN, we want the network to recognize broad spatial relationships without tracking exact pixel coordinates. Pooling is a fixed, non-parametric downsampling operator that reduces the spatial height and width of feature maps.
nn.MaxPool2d)
Extracts the maximum activation value in each $K \times K$ local window. By retaining only the strongest response, it provides spatial invariance: if an edge shifts by 1 pixel, the maximum activation in the window remains unchanged!
nn.AvgPool2d)
Computes the arithmetic mean of activations across each window. Smooths spatial features.
Global Average Pooling (GAP): In modern architectures (ResNet), nn.AdaptiveAvgPool2d((1, 1)) collapses each $H \times W$ feature map into a single scalar, completely replacing massive fully connected layers.
6. Anatomy of a Classic CNN Architecture
A complete Convolutional Neural Network consists of two distinct functional blocks:
Consists of alternating blocks of:
As spatial dimensions shrink ($28 \times 28 \to 14 \times 14 \to 7 \times 7$), channel depth increases ($1 \to 32 \to 64$). Spatial resolution is traded for semantic depth!
Consists of:
The 3D feature tensor is flattened into a 1D vector and mapped to class logit predictions via fully connected layers.
Hierarchical Visual Feature Extraction:
7. End-to-End PyTorch Implementation: Handwritten Digit Classification
Let us build and train an industrial-grade CNN classifier on the canonical MNIST handwritten digits dataset ($60,000$ training images, $10,000$ test images, $28 \times 28$ grayscale pixels, $10$ classes).
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, random_split
from torchvision import datasets, transforms
# 1. Device configuration
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# 2. Data transformation: convert PIL image [0, 255] to Tensor [0.0, 1.0]
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)) # MNIST global mean & std
])
# 3. Load Dataset & create Stratified 50k Train / 10k Validation split
full_train = datasets.MNIST(root="data", train=True, download=True, transform=transform)
test_dataset = datasets.MNIST(root="data", train=False, download=True, transform=transform)
train_set, val_set = random_split(full_train, [50000, 10000], generator=torch.Generator().manual_seed(42))
train_loader = DataLoader(train_set, batch_size=64, shuffle=True)
val_loader = DataLoader(val_set, batch_size=64, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)
# 4. Architecture Definition
class DigitCNN(nn.Module):
def __init__(self):
super().__init__()
self.features = nn.Sequential(
# Block 1: 1 x 28 x 28 -> 32 x 14 x 14
nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),
nn.BatchNorm2d(32),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),
# Block 2: 32 x 14 x 14 -> 64 x 7 x 7
nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1),
nn.BatchNorm2d(64),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2)
)
self.classifier = nn.Sequential(
nn.Flatten(),
nn.Linear(64 * 7 * 7, 128), # 3136 -> 128
nn.ReLU(),
nn.Dropout(p=0.25),
nn.Linear(128, 10) # 10 raw class logits
)
def forward(self, x):
return self.classifier(self.features(x))
model = DigitCNN().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
# 5. Training and Validation Loops
for epoch in range(5):
model.train()
running_loss, correct, total = 0.0, 0, 0
for imgs, targets in train_loader:
imgs, targets = imgs.to(device), targets.to(device)
optimizer.zero_grad()
logits = model(imgs)
loss = criterion(logits, targets)
loss.backward()
optimizer.step()
running_loss += loss.item() * imgs.size(0)
correct += (logits.argmax(dim=1) == targets).sum().item()
total += targets.size(0)
train_acc = correct / total
# Validation
model.eval()
val_corr, val_tot = 0, 0
with torch.no_grad():
for v_imgs, v_targs in val_loader:
v_imgs, v_targs = v_imgs.to(device), v_targs.to(device)
v_preds = model(v_imgs).argmax(dim=1)
val_corr += (v_preds == v_targs).sum().item()
val_tot += v_targs.size(0)
print(f"Epoch {epoch+1}/5 | Train Acc: {train_acc*100:.2f}% | Val Acc: {val_corr/val_tot*100:.2f}%")
# 6. Final Evaluation on Held-Out Test Set
model.eval()
test_corr, test_tot = 0, 0
with torch.no_grad():
for t_imgs, t_targs in test_loader:
t_imgs, t_targs = t_imgs.to(device), t_targs.to(device)
t_preds = model(t_imgs).argmax(dim=1)
test_corr += (t_preds == t_targs).sum().item()
test_tot += t_targs.size(0)
print(f"Final Test Accuracy: {test_corr / test_tot * 100:.2f}%") # ~99.1% Accuracy!
8. Evaluation Diagnostics: Confusion Matrices & Per-Class Metrics
While overall top-1 accuracy provides a single headline figure, it obscures systemic failure modes. In vision systems, calculating a Confusion Matrix ($K \times K$ where row $i$ is ground truth and column $j$ is prediction) exposes exact morphological ambiguities:
- 4 vs 9: Shared vertical stroke with closed top loop; distinguished only by horizontal top stroke termination.
- 3 vs 8: Double arc geometry; distinguished only by left-side contour closure.
- Shirt vs Coat: In Fashion-MNIST, shared sleeve and collar silhouettes make these classes the primary error contributors.
9. Overfitting Defenses for Computer Vision
Because deep CNNs have high representational capacity, vision models frequently memorize training samples. We employ four standard regularization strategies:
Artificially expands dataset diversity by applying random photometric and affine transformations during training:
transforms.RandomResizedCrop(28, scale=(0.9, 1.1))
nn.BatchNorm2d)
Normalizes activations across mini-batches $(N, H, W)$ for each channel, accelerating convergence by $5\times$ and regularizing gradients.
Randomly zeros activation channels or neurons with probability $p=0.25$, breaking co-adaptation of features.
Tracks validation loss after every epoch. When validation loss ceases to improve for $k$ consecutive epochs (patience), training terminates and best weights are restored.
10. Transfer Learning & Modern Pre-trained CNNs (ResNet Preview)
Training modern deep CNNs from scratch requires weeks on multi-GPU clusters and millions of labeled images. In practice, industry engineers leverage Transfer Learning: taking a network pre-trained on the massive ImageNet dataset ($1.2$ million images across $1,000$ categories) and adapting it to specialized target tasks.
Because early convolutional layers learn universal Gabor-like edge filters and color gradients, those representations transfer seamlessly across domains (from natural objects to medical histology, satellite imagery, or defect detection).
param.requires_grad = False) and train only a newly initialized linear classification head.
11. Top 10 Common CNN Pitfalls & Debugging Matrix
| # | Frequent Pitfall | Root Cause / Error Message | Correct Architectural Fix |
|---|---|---|---|
| 1 | Linear Flatten Mismatch | RuntimeError: mat1 and mat2 shapes cannot be multiplied. | Calculate $C \times H_{\text{out}} \times W_{\text{out}}$ exactly using pooling equations. |
| 2 | Channel Order Inversion | Passing NHWC tensor to nn.Conv2d. |
Permute tensor dimensions via x.permute(0, 3, 1, 2) to enforce NCHW. |
| 3 | Double Softmax with CrossEntropy | Applying Softmax before CrossEntropyLoss. |
Pass raw unbounded logits directly into CrossEntropyLoss. |
| 4 | Float Target Labels | Expected long scalar type for target. | Cast target classes to torch.int64 (.long()). |
| 5 | Augmenting Test Sets | Random rotations/crops applied during evaluation corrupt benchmark. | Keep validation and test pipelines strictly deterministic. |
| 6 | Missing model.eval() |
Dropout continues disabling 25% of features during testing. | Always invoke model.eval() before evaluation. |
| 7 | Missing torch.no_grad() |
Inference creates autograd graph tapes, exhausting VRAM. | Wrap validation in with torch.no_grad():. |
| 8 | Semantic-Breaking Augmentation | Applying HorizontalFlip on digits turns 6 into a non-digit or 9. | Tailor data augmentation strictly to domain symmetries. |
| 9 | Forgotten zero_grad() |
Gradients accumulate across batches, causing loss explosion. | Call optimizer.zero_grad() before backward pass. |
| 10 | Unnormalized Input Range | Input pixels fed as raw $[0, 255]$ uint8 values. | Divide by 255.0 to map inputs to $[0.0, 1.0]$. |
Practical Assignment: Fashion-MNIST Image Classifier
Replace handwritten digits with the more challenging Fashion-MNIST dataset (10 clothing classes: T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot):
torchvision.datasets.FashionMNIST. Split into 50k train, 10k validation, and 10k test. Apply transforms.ToTensor() and channel normalization.
CrossEntropyLoss and Adam(lr=0.001). Track validation accuracy per epoch.
Computer Vision & CNN Interactive Workbench
Explore sliding convolution filters, pooling downsampling, visual feature hierarchies, and spatial dimension math in real time.
Interactive Sliding 2D Convolution Kernel
Hover over the output feature map to inspect the receptive field patch and exact arithmetic.