Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are specialized neural architectures designed to process grid-structured data, such as images, by exploiting spatial hierarchies.

Convolutional Layers

Unlike fully connected layers where neurons connect to all inputs, convolutional layers connect only to a local receptive field.

Filters and Feature Maps

A layer applies learnable filters (kernels). As a filter slides, it computes dot products, producing a feature map.

Output Dimension Formula

Given input height $H_{\text{in}}$, filter size $F$, padding $P$, and stride $S$, the output height $H_{\text{out}}$ is:

$H_{\text{out}} = \lfloor \frac{H_{\text{in}} - F + 2P}{S} \rfloor + 1$

• Padding ($P$): Adding zero-pixels around the border to preserve spatial dimensions.
• Stride ($S$): The step size of the filter as it slides across the input.

Translation Invariance

CNNs build translation invariance: if a pattern (like a cat’s ear) is learned in one part of the image, the convolutional filters can recognize it anywhere else. This parameter sharing significantly reduces model parameter counts compared to MLPs.

Pooling Layers

Pooling layers reduce the spatial size of feature maps to decrease parameter counts and build translation invariance.

• Max Pooling: Extracts the maximum value from each receptive patch.
• Average Pooling: Computes the average value of each patch.

ResNet and Residual Connections

Deep CNNs suffer from vanishing gradients. ResNet resolves this using Residual Connections (skip connections) that bypass layers:

$a^{(l)} = \phi(W^{(l)} a^{(l-1)} + b^{(l)} + a^{(l-1)})$

This allows gradients to flow directly backward, enabling the training of deep networks.

Example: Output Shapes

The following example calculates the output dimensions of a convolutional layer:

import math

# Formula for calculating the output dimension of a convolutional layer
def get_output_dim(dim_in, filter_size, padding, stride):
  return math.floor((dim_in - filter_size + 2 * padding) / stride) + 1

# Standard input: 32x32 spatial dimensions, 5x5 filter, padding of 2, stride of 1
out_w = get_output_dim(dim_in=32, filter_size=5, padding=2, stride=1)
print(f"Output width: {out_w}")

Output width: 32

Exercise

Test your understanding of CNN operations:

Receptive Fields and Hierarchies

Early layers detect simple edges, while deeper layers combine these features to recognize complex shapes and objects, building a hierarchical spatial representation.

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd ed.). O’Reilly Media.