Neural Networks Foundations

Artificial Neural Networks (ANNs) are computational models inspired by biological neural networks, structured as layers of interconnected nodes.

The Perceptron

The Perceptron is the simplest ANN architecture, computing a weighted sum of inputs plus a bias, and applying a step function:

$\hat{y} = \text{step}(z) = \text{step}(w^T x + b)$

A single Perceptron is a linear classifier, meaning it cannot solve non-linearly separable problems (like the XOR problem).

Multilayer Perceptron (MLP)

An MLP consists of an input layer, one or more hidden layers, and an output layer. Hidden layers allow the model to learn non-linear representations.

Forward Propagation

For layer $l$, the activations $a^{(l)}$ are computed using the weight matrix $W^{(l)}$ and bias vector $b^{(l)}$:

$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$ $a^{(l)} = \phi(z^{(l)})$

where $\phi$ is the activation function.

Activation Functions

Non-linear activation functions prevent stacked layers from collapsing into a single linear model:

• Sigmoid: $\sigma(z) = 1 / (1 + e^{-z})$, squashing outputs to $(0, 1)$.
• Tanh: $\tanh(z) = (e^z - e^{-z}) / (e^z + e^{-z})$, squashing to $(-1, 1)$.
• ReLU: $\text{ReLU}(z) = \max(0, z)$, preventing vanishing gradients.
• ELU: $\text{ELU}(z) = \max(\alpha(e^z - 1), z)$, preventing dying neurons with small negative values.

Backpropagation

Backpropagation trains networks by computing the gradient of the loss function with respect to the weights using the chain rule of calculus, working backward from the output layer:

$\frac{\partial L}{\partial W^{(l)}} = \frac{\partial L}{\partial z^{(l)}} (a^{(l-1)})^T$

The error term $\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}}$ is computed recursively from the output error using matrix multiplications.

Example: Forward Pass Calculations

The following example demonstrates calculating the forward pass of a single neuron using NumPy:

import numpy as np

# Inputs, weights, and bias
x = np.array([1.5, 2.0])
w = np.array([0.8, -0.5])
b = -0.2

# Linear output
z = np.dot(w, x) + b

# ReLU activation
activation = max(0, z)
print(f"Pre-activation z: {z:.2f}")
print(f"Activation output: {activation:.2f}")

Pre-activation z: 0.00
Activation output: 0.00

Exercise

Test your understanding of activation functions:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning (3rd ed.). O’Reilly Media.