Regularization Theory

Regularization restricts model complexity to prevent overfitting. It penalizes large model weights during training, forcing parameters to stay small.

Regularized Linear Regression Models

Three common regularized versions of linear regression modify the MSE cost function:

1. Ridge Regression (L2 Regularization)

Ridge Regression adds a penalty equal to the sum of squared weights to the cost function:

$J(\theta) = \text{MSE}(\theta) + \alpha \frac{1}{2} \sum_{i=1}^{n} \theta_i^2$

The hyperparameter $\alpha$ controls regularization strength. If $\alpha = 0$, it behaves like Ordinary Least Squares. Note that the bias term $\theta_0$ is not regularized.

2. Lasso Regression (L1 Regularization)

Lasso adds a penalty equal to the sum of absolute weights:

$J(\theta) = \text{MSE}(\theta) + \alpha \sum_{i=1}^{n} |\theta_i|$

Lasso regression performs feature selection by forcing less important feature weights to exactly $0$.

Geometric Intuition

L1 forms a diamond constraint boundary that tends to intersect coordinate axes at their corners, yielding sparse weights. L2 forms a spherical boundary, shrinking weights toward zero without setting them exactly to zero.

3. Elastic Net

Elastic Net combines Ridge and Lasso regularizations, controlled by a mix ratio $r$:

$J(\theta) = \text{MSE}(\theta) + r \alpha \sum_{i=1}^{n} |\theta_i| + (1 - r) \alpha \frac{1}{2} \sum_{i=1}^{n} \theta_i^2$

It acts as a compromise, stabilizing selection when features are highly correlated.

Example: Parameter Sparsity

The following example demonstrates how Lasso regularizes weights to zero compared to Ridge:

import numpy as np
from sklearn.linear_model import Ridge, Lasso

# Small continuous dataset for regression
X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([2.1, 4.0, 5.9])

# Ridge Regression (L2 regularization) shrinks weights close to zero
ridge = Ridge(alpha=1.0)
ridge.fit(X, y)

# Lasso Regression (L1 regularization) can shrink weights exactly to zero
lasso = Lasso(alpha=1.0)
lasso.fit(X, y)

print(f"Ridge weights: {ridge.coef_}")
print(f"Lasso weights: {lasso.coef_}")

Ridge weights: [0.44303797 0.44303797]
Lasso weights: [0.8 0. ]

Exercise

Test your understanding of regularized cost functions:

Early Stopping

Early stopping is a regularization technique where validation error is monitored during iterative training. We stop training as soon as the validation error reaches a minimum and starts to increase.

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd ed.). O’Reilly Media.