Linear Independence and Span

Once we have a vector space, we need a way to describe its contents efficiently. If a vector space is a “playground,” then Linear Independence and Span are the rules for how many “tools” (vectors) you actually need to build everything in that playground.

1. The Concept of Span

The Span of a set of vectors $S = \{v_1, v_2, \dots, v_n\}$ is the set of all possible linear combinations of those vectors.

$\text{span}(S) = \{ c_1 v_1 + c_2 v_2 + \dots + c_n v_n : c_i \in \mathbb{R} \}$

Intuition: If you have two non-parallel arrows in a 2D plane, their span is the entire plane because you can reach any point by scaling and adding those two arrows.

import numpy as np
import matplotlib.pyplot as plt

v1 = np.array([1, 2])
v2 = np.array([2, 1])

# Generate random linear combinations (c1*v1 + c2*v2)
points = []
for _ in range(500):
  c1, c2 = np.random.uniform(-1, 1, 2)
  points.append(c1*v1 + c2*v2)

points = np.array(points)
plt.scatter(points[:,0], points[:,1], alpha=0.5, s=10)
plt.quiver([0,0], [0,0], [v1[0], v2[0]], [v1[1], v2[1]], color=['r','b'], scale=5)
plt.title("The Span of two non-parallel vectors (fills a 2D area)")
plt.grid(True)
plt.show()

2. Linear Independence: Avoiding Redundancy

A set of vectors is linearly independent if none of the vectors can be written as a linear combination of the others. In other words, they are all “original” and provide “new directions.”

Formal Definition

Vectors $\{v_1, \dots, v_n\}$ are linearly independent if the equation $c_1 v_1 + \dots + c_n v_n = \mathbf{0}$ has only the trivial solution $c_i = 0$.

If a vector can be built from others, the set is Dependent. We can test this by checking the Rank of the matrix formed by these vectors. If $\text{Rank}(A) < \text{Number of Vectors}$, they are dependent.

import numpy as np

# Case 1: Independent vectors
v1 = np.array([1, 0, 0])
v2 = np.array([0, 1, 0])
v3 = np.array([0, 0, 1])
mat1 = np.array([v1, v2, v3])
rank1 = np.linalg.matrix_rank(mat1)
print(f"Set 1 Rank: {rank1} (Independent if rank=3)")

# Case 2: Dependent vectors
u1 = np.array([1, 2, 3])
u2 = np.array([4, 5, 6])
u3 = u1 + u2 # Explicitly dependent
mat2 = np.array([u1, u2, u3])
rank2 = np.linalg.matrix_rank(mat2)
print(f"Set 2 Rank: {rank2} (Dependent if rank < 3)")

3. Visualizing Dependency in 3D

Imagine three vectors in 3D space. If one is a combination of the others, they all lie on a single plane (2D), even though there are three vectors.

4. Why Does It Matter?

Linear independence tells us if our data is redundant. If you have 100 sensors measuring the same physical phenomenon (e.g., temperature) and they are perfectly correlated, your “feature matrix” will be rank-deficient. You have 100 numbers, but effectively only 1 “dimension” of information.

import numpy as np

# Simulate 3 redundant sensors
temp = np.random.normal(25, 1, 100) # True temperature
s1 = temp + 0.1
s2 = 2 * temp - 5
s3 = temp * 0.5 + 10

data = np.vstack([s1, s2, s3]).T
rank = np.linalg.matrix_rank(data)

print(f"Data shape: {data.shape}")
print(f"Calculated Rank: {rank}")
print("Even though we have 3 sensors, the rank is 1 because they are all linear scalings of each other.")

5. Summary Check