Ensemble Learning and Random Forests

Ensemble methods combine predictions from multiple base models to build a stronger predictor with improved generalization.

Voting Classifiers

Ensemble models aggregate individual predictions using different consensus rules:

• Hard Voting: predicts the class that receives the absolute majority of votes from the base estimators.
• Soft Voting: averages the predicted class probabilities across all estimators, prioritizing confident predictions. This requires all estimators to support probability calculation.

Bagging and Pasting

Rather than using diverse algorithms, we can train multiple instances of the same base algorithm on different random subsets of the training set to construct homogeneous ensembles:

• Bagging (Bootstrap Aggregating): sampling is performed with replacement, allowing the same data point to be selected multiple times across different subsets.
• Pasting: sampling is performed without replacement, ensuring each subset contains unique data points.

Out-of-Bag (OOB) Evaluation

With bagging, statistical probability dictates that about 37% of the training instances are never sampled for any single estimator. These are known as Out-of-Bag (OOB) instances. Evaluating the ensemble’s performance on these OOB instances provides an unbiased validation score without requiring a separate validation dataset.

Random Forests

A Random Forest is an ensemble of decision trees trained via bagging. Tree splits are evaluated on random feature subsets to reduce estimator correlation and variance.

Example: Ensemble Voting Classifier

The following example demonstrates building a Voting Classifier:

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=100, noise=0.30, random_state=42)

log_clf = LogisticRegression()
tree_clf = DecisionTreeClassifier()

voting_clf = VotingClassifier(
  estimators=[('lr', log_clf), ('dt', tree_clf)],
  voting='soft'
)
voting_clf.fit(X, y)
print(f"Ensemble score: {voting_clf.score(X, y):.2f}")

Ensemble score: 0.94

Exercise

Test your understanding of bootstrap aggregation:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning (3rd ed.). O’Reilly Media.
• Smola, A., & Vishwanathan, S. V. N. (2008). An Introduction to Machine Learning. Cambridge University Press.