Eigenvalues and Eigenvectors

Most vectors change direction when a linear transformation is applied. However, some special vectors keep their direction and are only stretched or shrunk. These are eigenvectors, and their scaling factor is the eigenvalue.

1. The Stability Equation

For a linear operator $A$, a vector $v$ is an eigenvector if: $Av = \lambda v$ where $\lambda$ is a scalar (the eigenvalue).

Intuition: In a 2D rotation, no real vector keeps its direction (except the zero vector). But in a scaling transformation, the axes are eigenvectors because points on them move only along the line.

import numpy as np
import matplotlib.pyplot as plt

A = np.array([[3, 1], 
            [0, 2]])

# v is an eigenvector [1, 0] with eigenvalue 3
v = np.array([1, 0])
Av = A @ v

# u is NOT an eigenvector [0, 1]
u = np.array([0, 1])
Au = A @ u

plt.quiver([0, 0], [0, 0], [v[0], u[0]], [v[1], u[1]], color=['b', 'k'], scale=5, label='Original')
plt.quiver([0, 0], [0, 0], [Av[0], Au[0]], [Av[1], Au[1]], color=['r', 'gray'], scale=5, label='Mapped')
plt.legend()
plt.title("v (blue) stays on line, u (black) changes direction")
plt.show()

2. Finding the Spectrum

To find $\lambda$, we must solve $\det(A - \lambda I) = 0$. This gives us the Characteristic Polynomial.

3. The Power Method: Finding Eigenvectors Iteratively

In high-dimensional spaces (like Google’s PageRank), we don’t calculate determinants. Instead, we use the Power Method: repeatedly apply $A$ to a random vector until it converges to the dominant eigenvector.

import numpy as np

A = np.array([[2, 1], 
            [1, 2]])
x = np.random.rand(2) # Start with random vector

for i in range(10):
  x = A @ x
  x = x / np.linalg.norm(x) # Normalize to prevent overflow
  print(f"Iteration {i}: {x}")

print(f"
Final estimate for dominant eigenvector: {x}")
print(f"Verify: A @ x = {A @ x}")

4. Diagonalization: $A = PDP^{-1}$

If a matrix has enough eigenvectors, we can rotate our coordinate system so the transformation is just axis-aligned scaling. This is the foundation of Principal Component Analysis (PCA).

5. Summary Check