Naive Bayes Classifier

Naive Bayes is a probabilistic classifier based on Bayes’ Theorem. It assumes conditional independence between features given the class.

Probabilistic Foundation: Bayes’ Theorem

We predict the class probability $P(C_k | \mathbf{x})$ for an input feature vector $\mathbf{x} = [x_1, x_2, \dots, x_d]^T$ using:

$P(C_k | \mathbf{x}) = \frac{P(\mathbf{x} | C_k) P(C_k)}{P(\mathbf{x})}$

The Conditional Independence Assumption

Estimating $P(\mathbf{x} | C_k)$ directly requires massive datasets. The Naive Bayes classifier assumes that features are conditionally independent given the class:

$P(\mathbf{x} | C_k) = \prod_{j=1}^{d} P(x_j | C_k)$

Thus, the classification rule predicts the class that maximizes the numerator:

$\hat{y} = \arg\max_{k} P(C_k) \prod_{j=1}^{d} P(x_j | C_k)$

The “Naive” Limitation

Features are rarely independent in practice, making the model “naive” (e.g., “machine” and “learning” are highly correlated), but it performs remarkably well on tasks like spam filtering.

Generative vs. Discriminative

Naive Bayes is a generative model because it models the joint distribution $P(\mathbf{x}, C_k) = P(\mathbf{x} | C_k) P(C_k)$, unlike discriminative models (e.g., Logistic Regression) which model $P(C_k | \mathbf{x})$ directly.

Laplace Smoothing

If a feature value $x_j$ never occurs with class $C_k$ in training, the probability $P(x_j | C_k)$ is $0$. Since we multiply feature probabilities, this zeroes out the entire class probability. We prevent this using Laplace Smoothing:

$P(x_j = v | C_k) = \frac{count(x_j = v, C_k) + \alpha}{count(C_k) + \alpha \cdot N_j}$

where $N_j$ is the number of possible values for $x_j$, and $\alpha$ is the smoothing parameter.

Example: Classification with Gaussian Naive Bayes

The following example demonstrates training a Gaussian Naive Bayes model on continuous features:

import numpy as np
from sklearn.naive_bayes import GaussianNB

# Continuous features: [height, weight]
X = np.array([[180, 80], [160, 55], [175, 75], [150, 50]])
y = np.array([0, 1, 0, 1]) # 0: Male, 1: Female

model = GaussianNB()
model.fit(X, y)

# Classify unknown sample
sample = [[165, 60]]
prob = model.predict_proba(sample)
print(f"Probabilities [Male, Female]: {prob[0]}")

Probabilities [Male, Female]: [0.00392348 0.99607652]

Exercise

Test your understanding of the independence assumption:

Prior Probabilities

Priors $P(C_k)$ reflect baseline class rates, acting as regularizers that bias predictions toward dominant classes.

References & Further Reading

• Smola, A., & Vishwanathan, S. V. N. (2008). An Introduction to Machine Learning. Cambridge University Press.