Backpropagation & Gradient Descent: How Neural Networks Learn

1. How Neural Networks Correct Their Errors

In the previous lesson, we learned how an Artificial Neural Network performs Forward Propagation: input features are multiplied by synaptic weights, summed with a bias term, passed through non-linear activation functions, and aggregated to generate an output prediction $\hat{y}$.

Suppose we feed an image containing a domestic cat into our computer vision network. The network performs forward propagation and predicts:

The prediction is severely inaccurate. The loss function quantifies this error as $L = -\log(0.30) \approx 1.204$. But how does the network know which specific weights caused this failure, and by how much each parameter must be adjusted?

Modern deep learning architectures contain millions, billions, or even trillions of interconnected weights. Adjusting parameters by random trial and error would take longer than the age of the universe. To solve this monumental challenge, neural network training relies on the coordination of three mathematical pillars:

2. What is a Computational Graph?

Before deriving backpropagation mathematically, we must understand the concept of a Computational Graph. A computational graph is a directed acyclic graph (DAG) representing a mathematical expression, where:

• Nodes represent input variables, parameters, or mathematical operators ($+, \times, \exp, \sigma$).
• Edges represent the directional flow of numerical tensors.

Consider a single artificial neuron with input $x$, weight $w$, bias $b$, and sigmoid activation $\sigma$:

3. The Chain Rule: Foundation of Backpropagation

In differential calculus, the Chain Rule allows us to compute the derivative of a composite function. If variable $y$ depends on intermediate variable $u$, which in turn depends on independent variable $x$:

In intuitive terms: if variable $y$ changes $3$ times as fast as $u$, and $u$ changes $4$ times as fast as $x$, then $y$ must change $3 \times 4 = 12$ times as fast as $x$.

Multivariate Chain Rule for Branched Graphs

In neural networks, an intermediate variable often feeds into multiple downstream paths. If variable $x$ influences $y$ through two paths $u_1$ and $u_2$, the total derivative sums across all paths:

4. What is a Gradient?

A Gradient (denoted $\nabla L$) is a vector collecting all partial derivatives of a multivariable scalar function with respect to every input parameter:

5. Mathematical Derivation: Backpropagation in a Single Neuron

Let us rigorously derive the backpropagation equations for a single logistic neuron. Consider:

Our goal is to compute $\frac{\partial L}{\partial w}$ and $\frac{\partial L}{\partial b}$. By the Chain Rule:

Step 1: Derivative of Loss with respect to Probability $p$

Step 2: Derivative of Sigmoid with respect to Pre-activation $z$

Step 3: The Elegant Cancellation Theorem

Now, multiply Step 1 by Step 2 to compute the pre-activation error signal $\frac{\partial L}{\partial z}$:

Notice how the denominator $p(1-p)$ of the loss derivative perfectly cancels the $p(1-p)$ derivative of the sigmoid function! The error signal $\delta = p - y$ is simply the raw prediction residual (predicted probability minus ground truth label).

Step 4: Final Weight and Bias Gradients

Since $z = w \cdot x + b$, we have $\frac{\partial z}{\partial w} = x$ and $\frac{\partial z}{\partial b} = 1$. Therefore:

6. Step-by-Step Numerical Walkthrough

Let us perform a complete numerical forward and backward pass by hand. Suppose we have:

7. Backpropagation in Multi-Layer Perceptrons (MLP)

In a multi-layer network, gradients propagate backward layer-by-layer across dense matrix connections. Consider a 2-layer network with input $\mathbf{X} \in \mathbb{R}^{m \times d}$, hidden layer with ReLU activation, and output layer with Sigmoid:

Output Layer Gradients (Layer 2)

Starting at the output, the batch error signal is the difference between predictions and labels averaged over $m$ samples:

Hidden Layer Gradients (Layer 1)

To backpropagate through the hidden layer, we project the output error backward across $\mathbf{W}^{[2]}$ and multiply element-wise (Hadamard product $\odot$) by the local derivative of the ReLU activation:

8. Gradient Descent Variants: Batch vs. SGD vs. Mini-Batch

Once gradients are calculated, the optimizer updates parameters using the fundamental update rule:

The defining operational distinction between gradient descent algorithms is how many training samples are evaluated before executing an update:

9. Advanced Optimizers: Momentum, RMSprop, Adam & AdamW

Standard SGD struggles when navigating ill-conditioned ravines (where the surface curves much more steeply in one dimension than in another). Modern optimizers incorporate momentum and adaptive learning rates:

10. Learning Rate Scheduling

Keeping the learning rate $\eta$ constant throughout training is suboptimal. Early in training, large step sizes allow fast traversal of loss plateaus; late in training, smaller step sizes permit precise settling into deep minima:

11. Vanishing & Exploding Gradients and Gradient Clipping

In deep networks with $L$ layers, computing the gradient with respect to early weights requires chaining $L$ Jacobi matrices:

Gradient Clipping by Norm

Gradient Clipping prevents exploding gradients by capping the global L2 norm of the gradient vector at a maximum threshold $c$:

If $\|\mathbf{g}\|_2 \le c$, the gradient is unmodified. If $\|\mathbf{g}\|_2 > c$, the gradient is scaled down proportionally to exactly length $c$. Crucially, the directional vector of the gradient remains 100% unaltered; only the stride length is bounded.

12. Weight Initialization Strategies

Why can we not initialize all weights to zero?

13. Pure NumPy Implementation: Backpropagation from Scratch

Let us build a complete 2-4-1 Multi-Layer Perceptron in pure Python and NumPy to solve the non-linear XOR problem. No PyTorch, no TensorFlow—every matrix multiplication and gradient step is written explicitly:

import numpy as np

# 1. Non-linear XOR Dataset
X = np.array([
    [0.0, 0.0],
    [0.0, 1.0],
    [1.0, 0.0],
    [1.0, 1.0]
])
y = np.array([[0.0], [1.0], [1.0], [0.0]])

# 2. Reproducible Weight Initialization (He/Gaussian)
rng = np.random.default_rng(42)
W1 = rng.normal(0.0, np.sqrt(2.0 / 2), size=(2, 4))
b1 = np.zeros((1, 4))
W2 = rng.normal(0.0, np.sqrt(2.0 / 4), size=(4, 1))
b2 = np.zeros((1, 1))

def sigmoid(z):
    z = np.clip(z, -500, 500)
    return 1.0 / (1.0 + np.exp(-z))

def binary_cross_entropy(y_true, y_pred):
    eps = 1e-12
    p = np.clip(y_pred, eps, 1.0 - eps)
    return -np.mean(y_true * np.log(p) + (1.0 - y_true) * np.log(1.0 - p))

learning_rate = 0.5
epochs = 10000
m = X.shape[0]

# 3. Optimization Loop
for epoch in range(epochs):
    # Forward Pass
    Z1 = X @ W1 + b1
    A1 = np.tanh(Z1)                 # Hidden Tanh activation
    Z2 = A1 @ W2 + b2
    A2 = sigmoid(Z2)                 # Output Sigmoid activation

    loss = binary_cross_entropy(y, A2)

    # Backward Pass: Output Layer
    dZ2 = (A2 - y) / m               # Output error residual
    dW2 = A1.T @ dZ2
    db2 = np.sum(dZ2, axis=0, keepdims=True)

    # Backward Pass: Hidden Layer
    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * (1.0 - A1**2)        # Derivative of Tanh is (1 - tanh^2)
    dW1 = X.T @ dZ1
    db1 = np.sum(dZ1, axis=0, keepdims=True)

    # Gradient Descent Parameter Updates
    W1 -= learning_rate * dW1
    b1 -= learning_rate * db1
    W2 -= learning_rate * dW2
    b2 -= learning_rate * db2

    if epoch % 2000 == 0:
        print(f"Epoch {epoch:5d} | Binary Cross-Entropy Loss: {loss:.6f}")

# 4. Final Evaluation
final_preds = sigmoid(np.tanh(X @ W1 + b1) @ W2 + b2)
print("\nFinal XOR Predictions:\n", final_preds.round(4))
print("Predicted Binary Labels:", (final_preds >= 0.5).astype(int).flatten())

14. Automatic Differentiation & Modern Frameworks

While writing backpropagation by hand builds deep algorithmic intuition, manual derivations become intractable for 100-layer Transformer networks. Modern frameworks automate this via Automatic Differentiation (Autograd):

import torch

# Tracking parameter tensors on computational graph
w = torch.tensor(0.5, requires_grad=True)
x = torch.tensor(1.0)
y = torch.tensor(1.0)

# Forward pass
p = torch.sigmoid(w * x)
loss = -torch.log(p)

# Reverse-mode automatic differentiation
loss.backward()

print(f"Loss: {loss.item():.4f}")
print(f"Analytical Gradient dw: {w.grad.item():.4f}") # Matches -0.3775!

15. Gradient Checking: Numerical Verification

When writing custom layers or debugging backpropagation algorithms, we verify analytical gradients against finite-difference Numerical Approximations:

We compare analytical gradient vector $g_{\text{backprop}}$ with numerical gradient vector $g_{\text{approx}}$ using the relative Euclidean difference:

If relative difference is $\le 10^{-7}$, the analytical implementation is verified as mathematically correct.

16. The Complete 9-Step Neural Network Training Cycle

All modern deep learning systems follow an orchestrated 9-stage engineering workflow: