Feature Engineering

Preprocessing transforms raw data into numerical matrices optimized for training.

Handling Missing Data and Outliers

Real-world datasets are rarely perfect. Data may be missing due to sensor failures, non-responses, or unrecorded events.

• Imputation: Replaces missing values using training statistics (e.g., mean, median, mode) or algorithms (e.g., kNN imputation).
• Removal: Dropping rows or irrelevant columns with too many missing values.

Handling Outliers

Outliers are extreme values that can significantly skew statistical measures and models like Linear Regression. Depending on the context, we can:

• Keep: If the outliers represent genuine, important anomalies (e.g., fraud detection).
• Delete: If the outliers are clearly errors or irrelevant.
• Impute: Treat outliers as missing data and impute a central value.
• Transform: Apply mathematical transformations to reduce their impact.

Boxplots are visual tools commonly used to identify outliers. They display the interquartile range (IQR) of data, with points falling outside $1.5 \times \text{IQR}$ typically flagged as outliers.

Skewed Data and Transformations

When data is highly skewed (e.g., income distribution with a long right tail), it can negatively impact model performance. We apply transformations to make the distribution more Gaussian-like:

• Logarithmic Transformation: $x' = \log(x)$ (useful for right-skewed data).
• Square Root Transformation: $x' = \sqrt{x}$.
• Box-Cox Transformation: A parameterized family of power transformations.

Handling String and Text Data

Models require numerical inputs. Categorical and textual data must be encoded:

• Ordinal Encoding: Maps categories to ordered integers (e.g., “Low”, “Medium”, “High” to 1, 2, 3).
• One-Hot Encoding: Creates separate binary columns for each category, preventing the model from assuming a false ordinal relationship between nominal categories (e.g., colors).
• Bag of Words (BoW): For free-text data, BoW creates a vocabulary of all unique words and represents each document as a vector indicating the frequency of each word.

Feature Scaling

Disparate scales bias distance-based models (KNN, SVM). We scale features via:

1. Min-Max Scaling: maps to $[0, 1]$: $x_{\text{scaled}} = \frac{x - x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}}$.
2. Standardization: centers by mean and variance: $x_{\text{scaled}} = \frac{x - \mu}{\sigma}$.

Feature Selection and Correlation

Linear relationships are analyzed using a Correlation Matrix of Pearson coefficients in the range $[-1, 1]$. Below is an example mapping features from a real estate dataset:

The diagonal $1.00$ represents self-correlation. The coefficient $0.85$ indicates a strong positive relationship between size and bedrooms, while $-0.60$ shows a negative relationship with center distance (further houses tend to be smaller).

import os, pandas as pd, numpy as np, matplotlib.pyplot as plt
d = pd.DataFrame(np.random.randn(50, 3), columns=list('ABC'))
d['B'] = d['A'] * 0.7 + np.random.randn(50) * 0.3
plt.figure(figsize=(3.5, 2.8))
plt.imshow(d.corr(), cmap='coolwarm', vmin=-1, vmax=1)
plt.colorbar()
plt.xticks(range(3), d.columns)
plt.yticks(range(3), d.columns)
plt.tight_layout()
plt.savefig(os.environ["LIBREUNI_OUTPUT"], format="svg")

Example: Scaling and Encoding

The following example demonstrates categorical one-hot encoding and standardization:

import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder

df = pd.DataFrame({'Income': [50000, 120000, 30000], 'City': ['Paris', 'Berlin', 'Paris']})
scaled = StandardScaler().fit_transform(df[['Income']])
encoded = OneHotEncoder(sparse_output=False).fit_transform(df[['City']])

print("Scaled Income:", scaled.flatten().round(2))
print("Encoded City Categories:\n", encoded)

Scaled Income: [-0.46  1.39 -0.93]
Encoded City Categories:
[[0. 1.]
[1. 0.]
[0. 1.]]

Exercise

Evaluate your understanding of data preprocessing constraints:

Data Leakage

Preprocessing parameters must fit only on training data. Fitting on all data leaks test set statistics, inflating validation scores.

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning (3rd ed.). O’Reilly Media.