Logistic Regression and Probability Estimation

Logistic Regression is a classifier that computes a weighted sum of inputs and maps the output to a probability between $0$ and $1$.

Probabilistic Formulation: The Sigmoid Function

The model feeds its linear output into the Sigmoid Function $\sigma(t)$:

$\hat{p} = \sigma(\theta^T \mathbf{x}) = \frac{1}{1 + e^{-\theta^T \mathbf{x}}}$

The prediction rule for binary classes is:

$\hat{y} = \begin{cases} 1 & \text{if } \hat{p} \ge 0.5 \\ 0 & \text{if } \hat{p} < 0.5 \end{cases}$

The prediction decision boundary is linear where $\theta^T \mathbf{x} = 0$.

import os
import numpy as np
import matplotlib.pyplot as plt
t = np.linspace(-6, 6, 200)
sig = 1 / (1 + np.exp(-t))
plt.figure(figsize=(5, 3.5))
plt.plot(t, sig, "b-", linewidth=2, label=r"$\sigma(t) = \frac{1}{1 + e^{-t}}$")
plt.axhline(y=0.5, color="gray", linestyle="--")
plt.axvline(x=0, color="gray", linestyle="--")
plt.title("Sigmoid Curve")
plt.grid(True, alpha=0.3)
plt.legend()
plt.tight_layout()
plt.savefig(os.environ["LIBREUNI_OUTPUT"], format="svg")

Parameter Optimization: Log Loss

To optimize parameters $\theta$, we minimize the convex Log Loss (binary cross-entropy):

$J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(\hat{p}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{p}^{(i)}) \right]$

It penalizes confident incorrect predictions with infinite cost (if $y=1$ and $\hat{p} \to 0$, then $-\log(\hat{p}) \to \infty$).

Multiclass Classification Strategies

While Logistic Regression is inherently a binary classifier, it can be extended to handle multiple classes (e.g., classifying images of digits 0-9) using specific strategies:

• One-vs-Rest (OVR) (also called One-vs-All): Trains $N$ separate binary classifiers for an $N$-class problem. Each classifier is trained to distinguish one specific class from all the rest combined. During prediction, the classifier that outputs the highest probability wins. This is the default approach in most ML libraries for logistic regression.
• One-vs-One (OVO): Trains a binary classifier for every possible pair of classes, resulting in $\frac{N \times (N-1)}{2}$ classifiers. For a new data point, all classifiers are run, and the class that wins the most duels is selected. OVO is particularly useful for algorithms that scale poorly with dataset size (like SVMs), because each classifier is only trained on the subset of data belonging to the two classes.

Alternatively, Softmax Regression (Multinomial Logistic Regression) generalizes Logistic Regression to natively support multiple classes by normalizing raw logits into a direct probability distribution across all classes without training multiple binary models.

Example: Logistic Regression Inference

The following example demonstrates calculating predictions using a trained Logistic Regression classifier:

import numpy as np
from sklearn.linear_model import LogisticRegression

# Create toy dataset
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([0, 0, 0, 1, 1])

# Train model
clf = LogisticRegression()
clf.fit(X, y)

# Predict probabilities and classes
test_val = np.array([[3.5]])
prob = clf.predict_proba(test_val)
pred = clf.predict(test_val)

print(f"Probabilities [0, 1]: {prob[0].round(4)}")
print(f"Class: {pred[0]}")

Probabilities [0, 1]: [0.3235 0.6765]
Class: 1

Exercise

Test your knowledge of the Logistic Regression model boundary behavior:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd ed.). O’Reilly Media.