Lesson 14.15 — Vector Operations, Dot Product & Norms

1 Introduction: Computing with Multiple Vectors

Imagine a character in a 2D game whose current location is $\mathbf{p} = (3, 2)$. The player presses the movement controls, prompting a translation of 4 units to the right and 3 units upwards.

Both the current position and the translation can be represented as vectors:

• Initial position vector: $\mathbf{p} = (3, 2)$
• Displacement translation vector: $\mathbf{v} = (4, 3)$

The player's new position is computed through straightforward vector addition: $$\mathbf{p} + \mathbf{v} = (3 + 4, \, 2 + 3) = (7, 5)$$

However, modern engineering and artificial intelligence tasks require far more than basic addition:

• How far has an autonomous robot advanced along a desired trajectory? (Vector Projection).
• How semantically aligned are two text document embeddings in an AI model? (Cosine Similarity).
• How does an input feature vector interact with a neural network's weight vector? (Dot Product).
• How large or dispersed is a data point in high-dimensional space? (Vector Norms — $L_1, L_2, L_\infty$).

2 Vector Addition

Vector Addition combines two or more vectors of identical dimension by summing their corresponding components to produce a new resultant vector.

Given two 2D vectors $\mathbf{a} = (a_1, a_2)$ and $\mathbf{b} = (b_1, b_2)$, their algebraic sum is:

Numerical Example: If $\mathbf{a} = (3, 4)$ and $\mathbf{b} = (2, 1)$: $$\mathbf{a} + \mathbf{b} = (3 + 2, \, 4 + 1) = (5, 5)$$

2.1 Geometric Interpretation: The Head-to-Tail Rule

Geometrically, vector addition follows the Head-to-Tail (Triangle) rule. Placing the tail (initial point) of vector $\mathbf{b}$ at the head (terminal tip) of vector $\mathbf{a}$ produces the resultant vector $\mathbf{a} + \mathbf{b}$, which connects the origin of $\mathbf{a}$ directly to the tip of $\mathbf{b}$.

3 Vector Subtraction

Vector Subtraction subtracts corresponding components of the second vector from the first: $$\mathbf{a} - \mathbf{b} = (a_1 - b_1, \, a_2 - b_2)$$

Example: If $\mathbf{a} = (7, 5)$ and $\mathbf{b} = (3, 2)$: $$\mathbf{a} - \mathbf{b} = (7 - 3, \, 5 - 2) = (4, 3)$$

3.1 Geometric Meaning: Direction and Relative Displacement

Vector subtraction is algebraically equivalent to adding the opposite vector: $$\mathbf{a} - \mathbf{b} = \mathbf{a} + (-\mathbf{b})$$ where $-\mathbf{b}$ has identical length but points in the opposite direction ($180^\circ$ reversed).

4 Scalar Multiplication

Multiplying a vector $\mathbf{a} = (a_1, a_2)$ by a real scalar $k \in \mathbb{R}$ scales each component independently by $k$: $$k\mathbf{a} = (k a_1, \, k a_2)$$

The magnitude of the new vector scales proportionally: $$|k\mathbf{a}| = |k| \cdot |\mathbf{a}|$$

5 What is a Vector Norm?

A Vector Norm is a mathematical function that maps a vector to a non-negative real number ($\mathbb{R}_{\ge 0}$), quantifying its magnitude, size, or distance from the origin.

Denoted by $\|\mathbf{v}\|$, any valid norm must satisfy four fundamental mathematical axioms:

6 Core Vector Norms ($L_1, L_2, L_\infty$)

Different machine learning models and optimization frameworks leverage different norms:

7 Unit Vectors and Normalization

A vector whose $L_2$ norm equals exactly 1 is termed a Unit Vector (written $\hat{\mathbf{v}}$).

Dividing any non-zero vector by its Euclidean norm produces a unit vector pointing in the identical direction, a process known as Vector Normalization:

Critical Constraint: The Zero Vector has length zero, and division by zero is undefined. Therefore, the Zero Vector has no defined unit vector.

8 The Dot Product (Scalar Product)

The Dot Product takes two vectors of identical dimension and multiplies corresponding components, summing the products into a single Scalar value:

9 Geometric Meaning and Angle Determination

Beyond raw component multiplication, the dot product has an elegant geometric formulation. For any two non-zero vectors $\mathbf{a}$ and $\mathbf{b}$ separated by angle $ heta$:

Because magnitudes $\|\mathbf{a}\|_2$ and $\|\mathbf{b}\|_2$ are strictly positive, the sign of the dot product is determined entirely by $\cos heta$:

9.1 Finding the Angle Between Vectors

Rearranging the geometric formula allows us to compute the exact spatial angle $ heta$: $$\cos heta = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|_2 \|\mathbf{b}\|_2} \implies heta = \arccos\left( \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|_2 \|\mathbf{b}\|_2} ight)$$

10 Orthogonal Vectors

Two non-zero vectors $\mathbf{a}$ and $\mathbf{b}$ are called Orthogonal (perpendicular) if and only if their dot product equals zero: $$\mathbf{a} \cdot \mathbf{b} = 0 \iff \mathbf{a} \perp \mathbf{b}$$

Verification: Are $\mathbf{a} = (2, 3)$ and $\mathbf{b} = (3, -2)$ orthogonal? $$\mathbf{a} \cdot \mathbf{b} = (2 imes 3) + (3 imes -2) = 6 - 6 = 0$$ Yes! They meet at an exact $90^\circ$ angle.

11 Vector Projections

Projection measures how much of vector $\mathbf{a}$ lies directly along the directional axis of vector $\mathbf{b}$:

12 Cosine Similarity

In Natural Language Processing, Recommendation Engines, and Semantic Search, Cosine Similarity measures the directional orientation of two vectors, ignoring differences in their raw magnitude:

Cosine similarity values are bounded strictly within $[-1, +1]$:

Cosine Distance

To convert similarity into a minimization distance metric: $$\operatorname{cosine\_distance} = 1 - \operatorname{cosine\_similarity}$$

13 $L_1$ Distance vs. $L_2$ Distance

Measuring distance between two vector coordinates $\mathbf{a}$ and $\mathbf{b}$ depends on the chosen metric:

14 Vector Operations in Python & NumPy

In Python, NumPy provides vectorized C-level execution for all vector operations:

import numpy as np

a = np.array([3.0, 4.0])
b = np.array([2.0, 1.0])

# 1. Addition & Subtraction
print("a + b =", a + b)  # [5. 5.]
print("a - b =", a - b)  # [1. 3.]

# 2. Scalar Multiplication
print("2 * a =", 2 * a)  # [6. 8.]

# 3. Dot Product
dot_val = np.dot(a, b)
print("Dot Product:", dot_val)  # 3*2 + 4*1 = 10.0

# 4. Vector Norms
norm_l2 = np.linalg.norm(a, ord=2)      # Euclidean = 5.0
norm_l1 = np.linalg.norm(a, ord=1)      # Manhattan = 3 + 4 = 7.0
norm_linf = np.linalg.norm(a, ord=np.inf) # Maximum = 4.0

# 5. Unit Vector Normalization
unit_a = a / norm_l2
print("Unit Vector:", unit_a)  # [0.6 0.8]

# 6. Cosine Similarity (with zero vector guard)
def cosine_sim(u, v):
    norm_u = np.linalg.norm(u)
    norm_v = np.linalg.norm(v)
    if norm_u == 0 or norm_v == 0:
        raise ValueError("Cosine similarity is undefined for zero vectors")
    return np.dot(u, v) / (norm_u * norm_v)

print("Cosine Similarity:", cosine_sim(a, b))  # ~0.894

15 Real-World Applications (Graphics, AI, NLP, Optimization)

16 8 Common Pitfalls to Avoid

18 Key Takeaways