Lesson 14.24: Chain Rule & Backpropagation's Mathematical Foundation

1 Introduction: Multi-Step Sensitivity & Cascading Rates of Change

In previous lessons, we examined single-variable derivatives and multivariable partial derivatives and gradients. There, we studied direct functional dependencies where changing an input immediately modulated an output. However, real-world computational systems and physical phenomena rarely operate in a single leap.

Consider a mechanical processing machine that accepts a numerical input $x$ and executes three sequential operations:

1. First, it triples the input: $3x$
2. Next, it adds 2 to that product: $u = 3x + 2$
3. Finally, it squares the intermediate result: $y = u^2 = (3x + 2)^2$

The fundamental calculus question arises: If we apply an infinitesimal perturbation $\Delta x$ to the raw input, what is the resulting rate of change $\frac{dy}{dx}$ in the final output $y$?

2 What is a Composite Function? (Functions Within Functions)

When the output of one function serves as the input to a subsequent function, the resulting synthesis is called a Composite Function.

Crucially, the ultimate dependent variable $y$ is not directly connected to $x$; it influences $x$ solely through the mediation of the intermediate state $u$.

3 What is the Chain Rule? (Leibniz vs. Lagrange Notations)

The Chain Rule is a differentiation theorem that states the derivative of a composite function is the product of the local derivatives of each intermediate operational stage.

A Concrete Application: $y = (3x + 2)^4$

Let inner function $u = 3x + 2$ and outer function $y = u^4$.

4 Detailed Worked Example & Point Evaluation

Let us differentiate a higher-degree polynomial composite function and evaluate its instantaneous tangent slope at a specific coordinate.

5 Multi-Stage Chain Rule (The Long Chain)

Real deep networks do not stop at two stages; dozens or hundreds of layers are cascaded. When changes propagate across three or more sequential stages $x \to y \to z \to w$:

Universal Principle: No matter how many intermediate variables exist, the total derivative is simply the continuous product of every adjacent local rate of change along the computational path.

6 Multivariable Chain Rule (Branching Tree Derivatives)

In Lesson 23, we mastered Partial Derivatives ($\frac{\partial z}{\partial x}$, $\frac{\partial z}{\partial y}$). What if a function $z = f(x, y)$ depends on two variables $x$ and $y$, and both of them in turn depend on an overarching parameter $t$ ($x = x(t), y = y(t)$)? How does a nudge in $t$ perturb $z$?

7 What is a Computational Graph? (DAGs in Modern ML)

The universal representation used by deep learning engines (PyTorch, JAX, TensorFlow) to perform automatic differentiation is the Computational Graph. It is a Directed Acyclic Graph (DAG) where:

• Nodes: Variables ($x, w, b$), intermediate tensors, or mathematical operations ($+, \times, \text{ReLU}, \text{Square}$).
• Edges: Directional data flow representing dependencies between operations.

8 What is Backpropagation and Why is it Revolutionary?

Backpropagation (Backward Propagation of Errors) is an efficient algorithm for computing the gradient of a scalar Loss function with respect to all trainable parameters (weights and biases) of a neural network.

9 Backpropagation on a Concrete Neural Network (Step-by-Step)

Let us trace an explicit mathematical simulation on a minimal 2-layer network with 1 input, 1 hidden neuron, and 1 output neuron.

Phase 1: Forward Pass (Evaluation)

Phase 2: Backward Pass (Adjoint Chain Rule Propagation)

10 Parameter Update via Gradient Descent & Loss Reduction Verification

Having obtained the gradient vector, the optimizer adjusts parameters in the negative gradient direction: $\theta_{\text{new}} = \theta_{\text{old}} - \alpha \frac{\partial L}{\partial \theta}$.

Choosing learning rate $\alpha = 0.01$, the parameter updates yield:

11 Backpropagation Efficiency & Practical Challenges

While Backpropagation is mathematically optimal, applying the chain rule across hundreds of cascaded layers presents fundamental numerical stability issues:

12 Vectorized Backpropagation & The Jacobian Matrix

Production machine learning frameworks operate on full multidimensional tensors rather than isolated scalar values.

For vector-valued mapping $\mathbf{h} = g(\mathbf{x})$ and scalar loss $L = f(\mathbf{h})$, the multivariable chain rule assumes the matrix form:

Where $J_g(\mathbf{x})$ is the Jacobian Matrix containing all first-order partial derivatives. PyTorch and modern autodiff runtimes avoid explicitly instantiating dense $N \times M$ Jacobian matrices by evaluating Vector-Jacobian Products (VJPs) directly, conserving massive amounts of GPU memory.

13 Python & PyTorch Implementation

The executable script below implements the exact neural network from Section 9 using raw Python and verifies the computed gradients against PyTorch Autograd:

import torch

# ==========================================
# 1. Manual Backpropagation (Pure Python)
# ==========================================
x, y_true = 2.0, 4.0
w, b = 1.0, 0.5
v, c = 2.0, 0.0

# Forward Pass
z = w * x + b           # 2.5
h = max(0.0, z)         # 2.5 (ReLU)
y_hat = v * h + c       # 5.0
loss = 0.5 * (y_hat - y_true)**2  # 0.5

# Backward Pass (Chain Rule Adjoints)
dL_dyhat = y_hat - y_true       # 1.0
dL_dv = dL_dyhat * h            # 2.5
dL_dc = dL_dyhat * 1.0          # 1.0
dL_dh = dL_dyhat * v            # 2.0
dh_dz = 1.0 if z > 0 else 0.0   # 1.0 (ReLU derivative)
dL_dz = dL_dh * dh_dz           # 2.0
dL_dw = dL_dz * x               # 4.0
dL_db = dL_dz * 1.0             # 2.0

print(f"Manual Gradients: dL/dw={dL_dw}, dL/db={dL_db}, dL/dv={dL_dv}, dL/dc={dL_dc}")

# ==========================================
# 2. PyTorch Autograd Verification
# ==========================================
t_x = torch.tensor(2.0)
t_y = torch.tensor(4.0)
t_w = torch.tensor(1.0, requires_grad=True)
t_b = torch.tensor(0.5, requires_grad=True)
t_v = torch.tensor(2.0, requires_grad=True)
t_c = torch.tensor(0.0, requires_grad=True)

# PyTorch Forward
t_z = t_w * t_x + t_b
t_h = torch.relu(t_z)
t_yhat = t_v * t_h + t_c
t_loss = 0.5 * (t_yhat - t_y)**2

# PyTorch Backward
t_loss.backward()

print(f"PyTorch Gradients: dL/dw={t_w.grad.item()}, dL/db={t_b.grad.item()}, dL/dv={t_v.grad.item()}, dL/dc={t_c.grad.item()}")
# Exactly identical results: w=4.0, b=2.0, v=2.5, c=1.0!

14 Real-World Applications of Chain Rule & Backpropagation

15 8 Common Pitfalls & Traps

17 Key Takeaways