Foundations of Machine Learning

Machine learning allows computers to learn from experience rather than explicit developer rules. Formally, a program learns from experience $E$ with respect to a class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.

Taxonomy of Machine Learning Systems

Algorithms are structured by their learning feedback:

1. Supervised Learning: trained on labeled data $(x_i, y_i)$ to map inputs to continuous (regression) or discrete (classification) targets.
2. Unsupervised: identifies patterns in unlabeled data (e.g., clustering).
3. Semi-Supervised: combines few labeled and many unlabeled samples.
4. Reinforcement: learns optimal actions via environment rewards.

Mathematical Formulation: ERM

Learning is formulated as empirical risk minimization. Given a hypothesis space $\mathcal{H}$ and a loss function $L(y, h(x))$, we solve for a hypothesis $\hat{h} \in \mathcal{H}$ that minimizes the average loss over the training set:

$\hat{h} = \arg\min_{h \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^{n} L(y_i, h(x_i))$

Generalization and the Bias-Variance Trade-off

A model’s generalization error on unseen data is decomposed into three components:

• Bias: error due to overly simplistic assumptions, causing underfitting.
• Variance: error due to high sensitivity to small training set fluctuations, causing overfitting.
• Irreducible Noise: inherent variance in the data distribution.

Generalization is evaluated by training parameters on a training set and testing performance on an independent test set.

Example: Train-Test Splitting

The following example demonstrates splitting data and verifying shapes to ensure proper evaluation setup:

import numpy as np
from sklearn.model_selection import train_test_split

# Generate synthetic dataset
X = np.random.rand(100, 2)
y = np.random.randint(0, 2, size=100)

# 80/20 train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Train features shape: {X_train.shape}")
print(f"Test features shape: {X_test.shape}")

Train features shape: (80, 2)
Test features shape: (20, 2)

Exercise

Validate your understanding of generalization error trade-offs:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd ed.). O’Reilly Media.
• Smola, A., & Vishwanathan, S. V. N. (2008). An Introduction to Machine Learning. Cambridge University Press.