Principal Component Analysis

High-dimensional spaces suffer from the Curse of Dimensionality: points are sparse, distances become uniform, and models overfit. Dimensionality reduction compresses features, removes redundancies, and enables visualization. Principal Component Analysis (PCA) projects data onto a lower-dimensional subspace while maximizing variance preservation.

Matrix Factorization: SVD

PCA identifies the axis that accounts for the largest amount of variance in the training set. It finds principal components using Singular Value Decomposition (SVD), which decomposes the centered design matrix $X$ into three matrices:

$X = U \Sigma V^T$

The matrix $V$ contains the unit vectors that define the principal components:

$V = [\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n]$

To project the training set down to $d$ dimensions, we multiply the design matrix by the matrix $V_d$ containing the first $d$ principal components:

$X_{d\text{-proj}} = X V_d$

The Eigendecomposition Approach

Alternatively, PCA can be computed by performing an eigendecomposition on the Sample Covariance Matrix of the centered data. The covariance matrix $C$ captures the variance and correlation of features:

$C = \frac{1}{m-1} X^T X$

By solving the characteristic equation, we decompose $C$ into its eigenvalues ($\lambda$) and eigenvectors ($\mathbf{v}$):

$C \mathbf{v} = \lambda \mathbf{v}$

• The Eigenvectors represent the directions of the principal components (equivalent to the columns of $V$ from SVD). They are orthogonal to each other.
• The Eigenvalues represent the magnitude of variance captured along each corresponding principal component.

The Trace of the covariance matrix (the sum of its diagonal elements) is equal to the total variance in the dataset. Consequently, the sum of all eigenvalues is also equal to the total variance, allowing us to compute the proportion of variance captured by any single principal component $i$ as $\frac{\lambda_i}{\sum \lambda}$.

Explained Variance Ratio

The explained variance ratio indicates the proportion of the dataset’s variance that lies along each principal component. We use it to choose the number of dimensions $d$ that preserve a target percentage of variance (e.g., 95%).

Incremental PCA

For large datasets that do not fit in memory, we use Incremental PCA (IPCA). The algorithm splits the training set into mini-batches and feeds them one by one, enabling dimensionality reduction on out-of-core data.

Example: Computing PCA

The following example demonstrates projecting data using PCA and calculating the explained variance ratio:

import numpy as np
from sklearn.decomposition import PCA

# Synthetic 3D dataset
X = np.random.randn(10, 3)

# Project to 2D
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

print(f"Reduced shape: {X_2d.shape}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")

Reduced shape: (10, 2)
Explained variance ratio: [...]

Exercise

Evaluate your understanding of PCA projection logic:

Choosing the Number of Components

Instead of selecting an arbitrary number of dimensions, we look at the cumulative explained variance plot. We choose the number of principal components that capture a target percentage (typically 95%) of the total variance. This ensures that we discard noise while preserving the essential geometric structure of the original data.

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning (3rd ed.). O’Reilly Media.