Lesson 14.20: Eigenvalues & Eigenvectors

1 Introduction: The Invariant Directions of Transformations

In earlier lessons, we learned about vectors, matrices, linear transformations, basis, dimension, and matrix rank. We discovered that multiplying a vector by an $n \times n$ matrix represents a geometric action: rotating space, stretching coordinates, reflecting axes, or shearing grids.

Imagine stretching a rubber sheet diagonally or watching air currents flow over an aircraft wing. Most smoke particles curve away from their initial course, continually shifting direction. Yet a profound mathematical question arises: are there specific directions where vectors do not rotate at all, but simply change in length?

For instance, if a matrix transformation doubles a vector's length while keeping its orientation identical, that vector is an Eigenvector with an Eigenvalue of $\lambda = 2$. If the transformation completely reverses the vector's direction ($180^\circ$) and triples its length, the Eigenvalue is $\lambda = -3$. If the vector's length is halved, $\lambda = 0.5$.

2 What are Eigenvectors and Eigenvalues?

Let $A$ be an $n \times n$ square matrix, and let $\mathbf{v}$ be a non-zero vector ($\mathbf{v} \ne \mathbf{0}$). If there exists a scalar $\lambda \in \mathbb{R}$ (or $\mathbb{C}$) such that:

Examining this equation reveals two essential concepts:

• $\mathbf{v}$ is an Eigenvector of matrix $A$: It designates an axis in space whose direction remains invariant under the transformation.
• $\lambda$ is the corresponding Eigenvalue: It indicates the factor by which the eigenvector expands, contracts, or inverts.
• Why must $\mathbf{v} \ne \mathbf{0}$? Because for the zero vector $\mathbf{0}$, the equality $A\mathbf{0} = \lambda\mathbf{0} = \mathbf{0}$ holds trivially for every possible scalar $\lambda$. It yields zero geometric information about $A$. Keep in mind: an Eigenvalue $\lambda$ can be zero, but an Eigenvector $\mathbf{v}$ can never be zero!

3 Geometric Interpretation: How Eigenvectors Transform

Geometrically, an eigenvector defines a 1-dimensional invariant subspace (a line passing through the origin). Every point on this line is mapped to another point on the exact same line. The value of $\lambda$ dictates four distinct physical behaviors:

4 How to Find Eigenvalues: The Characteristic Equation

In the equation $A\mathbf{v} = \lambda\mathbf{v}$, both the scalar $\lambda$ and the vector $\mathbf{v}$ are unknowns. How can we solve for $\lambda$ without already knowing $\mathbf{v}$? Let us follow the algebraic derivation:

This represents a homogeneous linear system $(M\mathbf{v} = \mathbf{0})$ where matrix $M = (A - \lambda I)$.

From our study of linear systems, if $(A - \lambda I)$ were an invertible matrix, the system would possess only the trivial solution: $\mathbf{v} = \mathbf{0}$. However, our definition strictly mandates that $\mathbf{v} \ne \mathbf{0}$!

To obtain non-trivial (non-zero) solutions, the matrix $(A - \lambda I)$ must be singular (non-invertible). And a square matrix is singular if and only if its determinant equals zero!

5 Step-by-Step Worked Example: Finding Eigenvalues

Let us work through a complete, concrete numerical calculation. Suppose we are given the matrix:

6 How to Find Eigenvectors: Solving the Homogeneous Systems

Having found the eigenvalues, we substitute each $\lambda$ individually into the homogeneous linear system:

7 Eigenspaces and Scale Invariance

A common misconception is that an eigenvalue corresponds to only one single, rigid vector. In reality, an eigenvector identifies an invariant direction (a 1D subspace), not an isolated point.

If $\mathbf{v}$ is an eigenvector for eigenvalue $\lambda$, then for any non-zero scalar $c \ne 0$, the scaled vector $c\mathbf{v}$ is also a valid eigenvector:

For instance, for $\lambda_1 = 5$, all of the following vectors are equally valid eigenvectors:

8 Key Properties of Eigenvalues and Eigenvectors

Eigenvalues exhibit foundational mathematical relationships that dramatically simplify matrix operations and theoretical analysis:

9 Matrix Diagonalization: $A = PDP^{-1}$

Suppose you are tasked with calculating $A^{1000}$ (multiplying a large matrix by itself 1000 times). In raw computing, performing 1000 full matrix multiplications requires $O(1000 \cdot n^3)$ floating-point operations. However, using eigenvalues and eigenvectors, this calculation becomes instantaneous via Matrix Diagonalization!

Notice what happens when taking powers of $A$:

Raising a diagonal matrix $D$ to power $k$ is trivially calculated by raising each diagonal element to power $k$:

10 Real-World Applications in Computer Science and AI

Eigenvalues and eigenvectors power many of the most foundational algorithms in data science, artificial intelligence, and physical simulation:

11 Python & NumPy Implementation: Calculating Eigenvalues

In Python, the numpy.linalg module provides robust, high-performance routines for computing eigenvalues and eigenvectors:

import numpy as np

# 1. Define square matrix A
A = np.array([
    [4.0, 1.0],
    [2.0, 3.0]
])

# 2. Compute eigenvalues and eigenvectors using numpy.linalg.eig
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues (λ):")
print(eigenvalues)  # Output: [5. 2.]

print("\nEigenvectors (Stored as column vectors):")
print(eigenvectors)
# Output:
# [[ 0.70710678 -0.4472136 ]
#  [ 0.70710678  0.89442719]]

# 3. Mathematically verify A @ v ≈ λ * v
print("\n--- Numerical Verification ---")
for i in range(len(eigenvalues)):
    lam = eigenvalues[i]
    v = eigenvectors[:, i]  # CRITICAL: extract column i!
    lhs = A @ v
    rhs = lam * v
    is_equal = np.allclose(lhs, rhs)
    print(f"Eigenpair {i+1} (λ={lam:.1f}): Av == λv -> {is_equal}")

12 8 Common Pitfalls and Traps

Avoid these classic misconceptions frequently encountered in linear algebra and machine learning examinations: