1 Introduction: The Multivariable World (Beyond Single Variables)
In the previous lesson, we analyzed single-variable functions $y = f(x)$, discovering how derivatives calculate the instantaneous rate of change. Yet in real-world science, engineering, and artificial intelligence, virtually no meaningful system depends on a solitary variable:
A software platform's net profit $P(x, y)$ is jointly dictated by subscription pricing $x$, marketing spend $y$, and infrastructure overhead.
Atmospheric temperature $T(x, y, z, t)$ fluctuates continuously across 3D coordinates and dynamic time.
A Transformer's training Loss $L(\mathbf{w})$ is a staggering landscape parameterized by billions of interconnected weights ($w_1, w_2, \dots, w_n$).
In this multivariable realm, an essential scientific question emerges: "Holding all other variables strictly constant (ceteris paribus), if I nudge one single input, exactly how sensitive is the output?"
The answer is the Partial Derivative. Furthermore, when we assemble every partial derivative into an ordered vector, we construct the most powerful directional compass in all of applied mathematics: the Gradient ($\nabla f$). The gradient does not merely compute rates of change—it points directly toward the path of steepest ascent across high-dimensional manifolds.
The Bridge from Pure Calculus to Deep Learning Optimization
Single-variable calculus taught you how to find tangent lines on curves. But Deep Learning is fundamentally multivariable optimization. Gradient Descent, Backpropagation, Adam, and loss surface analysis are all direct manifestations of partial derivatives and gradients. Mastering this lesson gives you direct control over the engine room of AI.
2 What is a Multivariable Function?
A function that maps multiple independent numerical inputs to a single scalar output is defined as a Multivariable Function ($f: \mathbb{R}^n \to \mathbb{R}$).
Where $x$ and $y$ are independent coordinate inputs ($\mathbb{R}^2$), and $z$ is the resulting scalar altitude.
Numerical Evaluation: The Circular Paraboloid
Consider the paraboloid function $f(x, y) = x^2 + y^2$. Let us evaluate its output across several coordinates:
- At $(x=0, y=0)$: $f(0, 0) = 0^2 + 0^2 = 0$ (global valley floor at the origin)
- At $(x=2, y=3)$: $f(2, 3) = 2^2 + 3^2 = 4 + 9 = 13$
- At $(x=-1, y=4)$: $f(-1, 4) = (-1)^2 + 4^2 = 1 + 16 = 17$
While a single-variable function $y = f(x)$ traces a 1D curve in 2D space, a two-variable function $z = f(x, y)$ generates a 2D Surface embedded in 3D space, resembling a bowl, mountain saddle, or rolling terrain.
3 What is a Partial Derivative? Definition & Notation
When evaluating the rate of change of a multivariable function with respect to one chosen variable while holding all other independent variables strictly constant, the resulting rate is called the Partial Derivative.
Notice: In $\frac{\partial f}{\partial x}$, $y$ remains frozen in place; only $x$ is perturbed by infinitesimal $h$. Similarly, in $\frac{\partial f}{\partial y}$, $x$ is frozen while $y$ receives increment $k$.
Notations & Symbols
To distinguish multivariable differentiation from ordinary single-variable derivatives, the rounded symbol $\partial$ (pronounced "del", "partial", or "curly d") is universally adopted:
| Notation | Originator / Form | Pronunciation | Primary Use Cases |
|---|---|---|---|
| $\frac{\partial f}{\partial x}$ | Leibniz-Jacobi Notation | "partial f by partial x" | Physics, engineering equations, backpropagation formulas |
| $f_x(x, y)$ | Subscript Notation | "f sub x" | Concise mathematical analysis, Hessian matrices |
| $\partial_x f$ | Operator Notation | "del sub x of f" | Differential geometry and field theory |
| $D_1 f$ or $D_x f$ | Coordinate Index Notation | "Derivative along 1st coordinate" | Tensor computational graphs and code algorithms |
4 How to Compute Partial Derivatives: Step-by-Step
The golden operational rule for computing partial derivatives is straightforward: Treat the targeted variable as the only active variable. Treat all other variables as static constants (like 5 or $\pi$), and apply the familiar 1D calculus rules (Power, Product, etc.).
Given: $f(x, y) = x^2 + 3xy + y^2$
• Term 1 ($x^2$): Power rule gives $2x$.
• Term 2 ($3xy = (3y)x$): $3y$ acts as a constant coefficient; derivative of $x$ is 1, yielding $3y(1) = 3y$.
• Term 3 ($y^2$): Contains no $x$ whatsoever! It is a pure constant term; derivative of a constant is zero ($0$).
• Term 1 ($x^2$): Contains no $y$; treated as a static constant, giving $0$.
• Term 2 ($3xy = (3x)y$): $3x$ acts as a constant coefficient; derivative with respect to $y$ is $3x(1) = 3x$.
• Term 3 ($y^2$): Power rule gives $2y$.
5 Evaluation at Specific Coordinates & Physical Meaning
Let us evaluate our two partial derivatives at the coordinate point $(x=1, y=2)$:
If we hold $y=2$ constant and take an infinitesimal step parallel to the east-west ($x$) axis, the surface altitude rises at a rate of 8 units per unit step.
If we hold $x=1$ constant and take an infinitesimal step parallel to the north-south ($y$) axis, the surface altitude rises at a rate of 7 units per unit step.
6 Geometric Interpretation: Cross-Section Slices & Tangent Planes
What does a partial derivative look like in 3D geometric space?
Picture a 3D mountain landscape $z = f(x, y)$. If you slice through this mountain with a vertical plane at $y = b$ (parallel to the $xz$-plane), the intersection forms a 2D cross-sectional curve. The slope of the tangent line to this curve at $(a, b)$ is precisely $\frac{\partial f}{\partial x}(a, b)$. Slicing with a vertical plane at $x = a$ generates an orthogonal curve whose tangent slope is $\frac{\partial f}{\partial y}(a, b)$.
Geometry of Slicing Planes on a 3D Surface
The green curve shows the surface cross-section parallel to $x$ (slope = $\frac{\partial f}{\partial x}$). The orange curve shows the cross-section parallel to $y$ (slope = $\frac{\partial f}{\partial y}$). Together, these two orthogonal tangent vectors uniquely span the 2D Tangent Plane grazing the surface.
7 What is a Gradient? (The Nabla Vector $\nabla f$)
Individual partial derivatives provide rates of change along isolated coordinate axes. When we gather all first-order partial derivatives together into a unified vector, we obtain the Gradient.
The inverted triangle symbol $\nabla$ is named Nabla. For 3 variables, $\nabla f = [\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z}]^T$; for $n$ variables, it forms an $n$-dimensional column vector.
For our running example $f(x, y) = x^2 + 3xy + y^2$, the general gradient vector is: $$\nabla f(x, y) = \begin{bmatrix} 2x + 3y \\ 3x + 2y \end{bmatrix}$$ Evaluating at $(1, 2)$ yields: $$\nabla f(1, 2) = \begin{bmatrix} 8 \\ 7 \end{bmatrix}$$
Crucial Distinction to Internalize:
- Partial Derivative ($\frac{\partial f}{\partial x_i}$): A scalar slope along one specific coordinate axis.
- Gradient ($\nabla f$): A full multivariable vector combining all coordinate sensitivities into a single direction and magnitude in space.
8 Magnitude & Direction of the Gradient
The gradient vector possesses two fundamental, non-negotiable geometric properties:
Standing at any point on the surface, $\nabla f$ points in the exact direction in which the function increases most rapidly. Conversely, $-\nabla f$ points in the direction of steepest descent.
The Euclidean norm $\|\nabla f\|$ gives the exact numerical value of the maximum directional slope at that coordinate: $$\|\nabla f\| = \sqrt{\left(\frac{\partial f}{\partial x}\right)^2 + \left(\frac{\partial f}{\partial y}\right)^2}$$
In our example at $(1, 2)$:
• Magnitude: $\|\nabla f(1, 2)\| = \sqrt{8^2 + 7^2} = \sqrt{64 + 49} = \sqrt{113} \approx 10.63$
• Meaning: Across all possible 360-degree directions, the absolute maximum rate of altitude increase is 10.63 units per step.
9 Directional Derivative: Rate of Change in Any Direction
What if you wish to evaluate the slope along an arbitrary direction specified by unit vector $\mathbf{u} = [u_1, u_2]^T$ ($\|\mathbf{u}\| = 1$)?
The rate of change along $\mathbf{u}$ is defined as the Directional Derivative, computed via the dot product of the gradient and the unit direction vector:
Unlocking the Gradient Mystery via the Dot Product:
From vector algebra: $\mathbf{A} \cdot \mathbf{B} = \|\mathbf{A}\| \|\mathbf{B}\| \cos(\theta)$. Since $\|\mathbf{u}\| = 1$: $$D_{\mathbf{u}}f = \|\nabla f\| \cos(\theta)$$ Where $\theta$ is the angle between the gradient vector and the chosen direction vector $\mathbf{u}$.
- Maximum Ascent ($\theta = 0^\circ$): $\cos(0) = 1 \implies D_{\mathbf{u}}f = +\|\nabla f\|$ (moving directly along $\nabla f$).
- Steepest Descent ($\theta = 180^\circ$): $\cos(180) = -1 \implies D_{\mathbf{u}}f = -\|\nabla f\|$ (moving directly opposite $-\nabla f$).
- Zero Altitude Change ($\theta = 90^\circ$): $\cos(90) = 0 \implies D_{\mathbf{u}}f = 0$ (moving perpendicular to $\nabla f$, along level curves)!
10 Gradients & Level Curves: The Orthogonality Theorem
Just as topographic hiking maps use contour lines to connect points of equal elevation, multivariable calculus defines Level Curves (or contours) as sets of coordinates where the output remains constant ($f(x, y) = c$).
For any differentiable function, the gradient vector at point $P$ is strictly orthogonal ($90^\circ$) to the tangent of the level curve passing through $P$!
Contour Lines & Perpendicular Gradient Vectors
Walking along a level curve produces zero altitude change ($df = 0$). Therefore, the direction of maximum altitude gain must point perpendicular to the curve.
11 Gradient Descent in Machine Learning
In machine learning, training an artificial model corresponds to minimizing an empirical Loss Function $L(w_1, w_2, \dots, w_n)$ to find its optimal parameters.
Where $\alpha$ (Alpha) is the Learning Rate controlling step size.
Worked Training Step Simulation
Consider a two-parameter model with quadratic loss: $$L(w_1, w_2) = w_1^2 + 2w_2^2$$
1. Compute the loss gradient:
$$\nabla L = \begin{bmatrix} \frac{\partial L}{\partial w_1} \\ \frac{\partial L}{\partial w_2} \end{bmatrix} = \begin{bmatrix} 2w_1 \\ 4w_2 \end{bmatrix}$$
2. Initial state & gradient evaluation:
Let starting weights be $\mathbf{w}_0 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$ with learning rate $\alpha = 0.1$.
Initial loss: $L(2, 1) = 2^2 + 2(1)^2 = 4 + 2 = 6$.
Initial gradient: $\nabla L(2, 1) = \begin{bmatrix} 2(2) \\ 4(1) \end{bmatrix} = \begin{bmatrix} 4 \\ 4 \end{bmatrix}$.
3. Execute first optimization update:
$$\begin{bmatrix} w_1 \\ w_2 \end{bmatrix}_{\text{new}} = \begin{bmatrix} 2 \\ 1 \end{bmatrix} - 0.1 \begin{bmatrix} 4 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 - 0.4 \\ 1 - 0.4 \end{bmatrix} = \begin{bmatrix} 1.6 \\ 0.6 \end{bmatrix}$$
4. Verify new loss value:
$$L(1.6, 0.6) = (1.6)^2 + 2(0.6)^2 = 2.56 + 2(0.36) = 2.56 + 0.72 = 3.28$$
In a single update, model error dropped from 6.00 down to 3.28!
12 Second-Order Partial Derivatives & The Hessian Matrix
Differentiating partial derivatives a second time produces Second-Order Partials. In 2D, exactly 4 permutations exist:
The Hessian Matrix ($H$)
Curvature MatrixThe matrix gathering all second-order partial derivatives is the Hessian Matrix: $$H_f = \begin{bmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} \end{bmatrix}$$
13 Practical Python Implementation: SymPy & PyTorch Autograd
Standard computational workflows for multivariable calculus in Python:
import sympy as sp
x, y = sp.symbols('x y')
f = x**2 + 3*x*y + y**2
# Partial derivatives
df_dx = sp.diff(f, x)
df_dy = sp.diff(f, y)
print(f"df/dx: {df_dx}") # 2*x + 3*y
print(f"df/dy: {df_dy}") # 3*x + 2*y
# Gradient vector
grad = sp.Matrix([df_dx, df_dy])
print(f"Gradient at (1,2): {grad.subs({x: 1, y: 2})}") # [8, 7]
# Hessian matrix
H = sp.hessian(f, (x, y))
print(f"Hessian Matrix:\n{H}") # Matrix([[2, 3], [3, 2]])import torch
# Parameter tensor w = [w1, w2]
w = torch.tensor([2.0, 1.0], requires_grad=True)
# Loss function: L = w1^2 + 2*w2^2
loss = w[0]**2 + 2 * w[1]**2
# Backpropagation (computes dL/dw1 and dL/dw2)
loss.backward()
print(f"Loss value: {loss.item():.2f}") # 6.00
print(f"Gradient vector [dL/dw1, dL/dw2]: {w.grad}") # tensor([4.0000, 4.0000])
# Single gradient descent step
lr = 0.1
with torch.no_grad():
w -= lr * w.grad
print(f"Updated weights: {w}") # tensor([1.6000, 0.6000])14 Common Pitfalls & Traps to Avoid
Key errors frequently encountered in multivariable calculus and gradient optimization:
When computing $\frac{\partial}{\partial x}(x y^2)$, mistakenly differentiating $y^2$ to $2y$. When differentiating with respect to $x$, $y^2$ is strictly a constant scalar factor!
A partial derivative is an individual scalar rate along one axis; the gradient is the complete vector combining all partials.
$\nabla f$ points towards steepest growth. To minimize error in ML, you must move along the negative gradient ($-\nabla f$).
In $D_{\mathbf{u}}f = \nabla f \cdot \mathbf{u}$, the vector $\mathbf{u}$ must have unit length ($\|\mathbf{u}\| = 1$). Failing to normalize artificially inflates the computed rate.
A zero gradient simply identifies a stationary point. It could be a local maximum or a notorious saddle point. Hessian analysis is required.
$\|\nabla f\| \ne \frac{\partial f}{\partial x} + \frac{\partial f}{\partial y}$. The magnitude is the Pythagorean length $\sqrt{(\partial_x f)^2 + (\partial_y f)^2}$.
The gradient does not flow along contour curves; it is strictly perpendicular ($90^\circ$) to level curve tangents.
Setting learning rate $\alpha$ excessively high causes parameter updates to overshoot the valley and explode toward infinity.
15 Interactive Hands-on Workbench (Multivariable Calculus Lab)
Directly test and visualize multivariable slopes, directional vectors, and optimization dynamics across 3 interactive labs:
Contour Map & Orthogonal Gradient Vector ($\nabla f$)
Click anywhere on the canvas or adjust sliders. Notice how the blue gradient vector remains strictly perpendicular ($90^\circ$) to the contour line.
16 Key Takeaways & Summary
Measures sensitivity with respect to one active variable while all other variables are held frozen as constants.
$\nabla f = [\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}]^T$ bundles all partial slopes into a single directional vector.
The gradient points in the exact direction of maximal increase; $-\nabla f$ points towards steepest descent.
$\|\nabla f\| = \sqrt{\sum (\partial f/\partial x_i)^2}$ yields the absolute maximum directional rate of change.
The rate of change along any unit direction $\mathbf{u}$ is evaluated via dot product: $D_{\mathbf{u}}f = \nabla f \cdot \mathbf{u}$.
The gradient vector is strictly perpendicular ($90^\circ$) to the tangent of the level contour curve passing through that point.
Iterative update $\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla L$ drives machine learning models from error down to convergence.
A symmetric matrix of second-order partials whose eigenvalues reveal whether stationary points are minima, maxima, or saddles.
Modern frameworks (PyTorch, JAX) traverse computational graphs to compute exact partial gradients across billions of parameters automatically.
Chain Rule & Backpropagation's Math Foundation
When one function's output feeds directly into another function's input, how do derivatives cascade through the chain? In the next lesson, we master the Chain Rule and unravel how deep neural networks propagate gradients backward across dozens of layers—the mathematical foundation of Backpropagation.