Model Evaluation Metrics

Evaluating model performance requires task-specific metrics. Accuracy is misleading on imbalanced datasets: if a rare disease affects 1% of the population, a dummy classifier predicting “healthy” achieves 99% accuracy. Generalization error must be measured on independent data to evaluate utility.

Classification Metrics and Confusion Matrix

Binary classification performance is tabulated in a Confusion Matrix:

In this disease-screening example with 200 samples:

• Precision: positive prediction accuracy. Out of 90 predicted positive cases, 80 were correct. $\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} = \frac{80}{80 + 10} \approx 0.889$
• Recall (Sensitivity): detection rate of actual positives. Out of 100 actual diseased patients, 80 were detected. $\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} = \frac{80}{80 + 20} = 0.800$
• F1 Score: harmonic mean of precision and recall, penalizing extreme values. $F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = 2 \times \frac{0.889 \times 0.8}{0.889 + 0.8} \approx 0.842$

ROC Curve and AUC

The ROC curve plots True Positive Rate (Recall) against False Positive Rate ($\text{FPR} = \frac{\text{FP}}{\text{TN} + \text{FP}}$) across decision thresholds. The Area Under the Curve (AUC) ranges from $0.5$ (random) to $1.0$ (perfect).

import os
import matplotlib.pyplot as plt
plt.figure(figsize=(5, 3.5))
plt.plot([0, 1], [0, 1], 'k--', label='Random (AUC=0.5)')
plt.plot([0, 0.25, 1], [0, 0.75, 1], 'b-', label='Model (AUC=0.75)')
plt.xlabel('FPR')
plt.ylabel('TPR')
plt.title('ROC Curve')
plt.legend()
plt.tight_layout()
plt.savefig(os.environ["LIBREUNI_OUTPUT"], format="svg")

Regression Metrics

While classification relies on counting correct assignments, regression predicts continuous values. Typical metrics include Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE).

A normalized evaluation metric is the Coefficient of Determination ($R^2$). The total variance in the data is captured by the Total Sum of Squares (SST). The variance explained by the model is the Sum of Squares due to Regression (SSR), and the unexplained variance is the Sum of Squared Errors (SSE).

$\text{SST} = \sum_{i=1}^n (y_i - \bar{y})^2$ $\text{SSE} = \sum_{i=1}^n (y_i - \hat{y}_i)^2$ $\text{SSR} = \sum_{i=1}^n (\hat{y}_i - \bar{y})^2$

The $R^2$ score represents the proportion of variance explained by the model:

$R^2 = 1 - \frac{\text{SSE}}{\text{SST}}$

Validation Methods

Proper validation is crucial for an iterative ML workflow to prevent overfitting and ensure model generalization.

• Train-test split: Uses a portion of data for training and a held-out test set purely for final evaluation.
• Train-val-test split: Introduces a validation set. The training set updates model parameters, the validation set is used for evaluation and hyperparameter tuning, and the test set is reserved exclusively for final evaluation.
• Stratification: When splitting non-representative training data (especially imbalanced classification datasets), stratification ensures the class distribution in all splits mirrors the original dataset.

Cross-Validation

• Cross-validation (CV): Splits the training data into $k$ folds. The model trains on $k-1$ folds and validates on the remaining one, rotating iteratively. This provides a more robust estimate of performance than a single validation set.
• Leave-one-out cross-validation (LOO CV): An extreme form of CV where $k$ equals the number of samples ($N$). The model trains on $N-1$ samples and is validated on the single remaining point. While computationally expensive, it maximizes training data usage and provides unbiased performance estimates.

Example: Calculating Metrics

The following example demonstrates calculating classification metrics using a confusion matrix in scikit-learn:

from sklearn.metrics import precision_score, recall_score, f1_score

# Ground truth and predictions
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 1, 0]

precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")

Precision: 0.80
Recall: 0.80
F1 Score: 0.80

Exercise

Validate your knowledge of classification metric trade-offs:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd ed.). O’Reilly Media.