Supervised Learning Categories

Supervised learning is divided into two main tasks:

Regression: Predicting a continuous numerical value (e.g., house prices).
Classification: Predicting a discrete category or label (e.g., spam vs. not spam).

Classification: Beyond the Basics

We already saw the Estimator API. Let’s look at more complex classifiers.

Support Vector Machines (SVM)

SVMs are powerful models that attempt to find the hyperplane that best separates classes with the maximum margin.

Interactive Lab


from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate synthetic data
X, y = make_classification(n_samples=100, n_features=2, n_redundant=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Use Radial Basis Function (RBF) kernel
model = SVC(kernel='rbf', C=1.0)
model.fit(X_train, y_train)

print(f"SVM Test Accuracy: {model.score(X_test, y_test):.2f}")

Expected output

SVM Test Accuracy: 0.95

python

2from sklearn.svm import SVC

3from sklearn.datasets import make_classification

4from sklearn.model_selection import train_test_split

6# Generate synthetic data

7X, y = make_classification(n_samples=100, n_features=2, n_redundant=0, random_state=42)

8X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

10# Use Radial Basis Function (RBF) kernel

11model = SVC(kernel='rbf', C=1.0)

12model.fit(X_train, y_train)

14print(f"SVM Test Accuracy: {model.score(X_test, y_test):.2f}")

Decision Trees and Random Forests

Decision Trees mimic human decision-making by splitting data based on feature thresholds. Random Forests are “ensembles” of many decision trees, which reduces overfitting.

Interactive Lab


from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target

model = RandomForestClassifier(n_estimators=100)
model.fit(X, y)

print(f"Random Forest Accuracy on Digits: {model.score(X, y):.2f}")

Expected output

Random Forest Accuracy on Digits: 1.00

python

2from sklearn.ensemble import RandomForestClassifier

3from sklearn.datasets import load_digits

5digits = load_digits()

6X, y = digits.data, digits.target

8model = RandomForestClassifier(n_estimators=100)

9model.fit(X, y)

11print(f"Random Forest Accuracy on Digits: {model.score(X, y):.2f}")

Regression: Complexity and Regularization

Simple linear regression often underfits complex data. We can use techniques like Ridge and Lasso regression to prevent overfitting by penalizing large coefficients.

Interactive Lab


from sklearn.linear_model import Ridge, Lasso
import numpy as np

# Synthetic noisy data
X = np.random.rand(100, 10)
y = 2*X[:, 0] + 3*X[:, 1] + np.random.randn(100) * 0.1

ridge = Ridge(alpha=1.0)
ridge.fit(X, y)

lasso = Lasso(alpha=0.1)
lasso.fit(X, y)

print(f"Ridge Coefficients: {ridge.coef_[:2]}")
print(f"Lasso Coefficients: {lasso.coef_[:2]}")

Expected output

Ridge Coefficients: [1.98 2.95]
Lasso Coefficients: [1.85 2.82]

python

2from sklearn.linear_model import Ridge, Lasso

3import numpy as np

5# Synthetic noisy data

6X = np.random.rand(100, 10)

7y = 2*X[:, 0] + 3*X[:, 1] + np.random.randn(100) * 0.1

9ridge = Ridge(alpha=1.0)

10ridge.fit(X, y)

12lasso = Lasso(alpha=0.1)

13lasso.fit(X, y)

15print(f"Ridge Coefficients: {ridge.coef_[:2]}")

16print(f"Lasso Coefficients: {lasso.coef_[:2]}")

Hyperparameter Tuning

How do we choose the best alpha for Ridge or the best n_estimators for a Random Forest? We use Grid Search.

Interactive Lab


from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
  'n_estimators': [10, 50, 100],
  'max_depth': [None, 5, 10]
}

grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5)
# grid.fit(X, y) # Uncomment to run (takes time)
# print(f"Best params: {grid.best_params_}")
print("GridSearchCV defined with cross-validation.")

Expected output

GridSearchCV defined with cross-validation.

python

2from sklearn.model_selection import GridSearchCV

3from sklearn.ensemble import RandomForestClassifier

5param_grid = {

6 'n_estimators': [10, 50, 100],

7 'max_depth': [None, 5, 10]

10grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5)

11# grid.fit(X, y) # Uncomment to run (takes time)

12# print(f"Best params: {grid.best_params_}")

13print("GridSearchCV defined with cross-validation.")

Pipelines: Chaining Transformations

A pipeline combines a series of preprocessing steps and a final estimator into one object. This prevents data leakage during cross-validation.

Interactive Lab


from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

pipeline = Pipeline([
  ('scaler', StandardScaler()),
  ('svc', SVC())
])

# Fit and predict just like a single estimator
# pipeline.fit(X_train, y_train)
print("Pipeline created: [Scaler -> SVC]")

Expected output

Pipeline created: [Scaler -> SVC]

python

2from sklearn.pipeline import Pipeline

3from sklearn.preprocessing import StandardScaler

4from sklearn.svm import SVC

6pipeline = Pipeline([

7 ('scaler', StandardScaler()),

8 ('svc', SVC())

9])

11# Fit and predict just like a single estimator

12# pipeline.fit(X_train, y_train)

13print("Pipeline created: [Scaler -> SVC]")

In the next module, we’ll explore Unsupervised Learning techniques like Clustering and Dimensionality Reduction.

Scikit-Learn: Supervised Learning Deep Dive

Supervised Learning Categories

Classification: Beyond the Basics

Support Vector Machines (SVM)

Decision Trees and Random Forests

Regression: Complexity and Regularization

Hyperparameter Tuning

Pipelines: Chaining Transformations