6 Image modelling: From pixels to learned features

In ENGSCI 205, you built and trained a convolutional neural network for classifying MNIST images. You first applied a manually specified edge-detection filter and inspected the resulting feature map, then constructed a LeNet-style network containing convolutional and pooling layers. Here we briefly recall how those layers construct learned image representations before connecting them to modern object-detection models.

The MNIST images used in ENGSCI 205 were greyscale, so each image had one input channel. When you applied a $5\times5$ edge-detection filter, the filter was placed over a $5\times5$ patch of pixel values. The pixel values and filter weights were multiplied and summed to produce one output value. Repeating this calculation across the image produced a two-dimensional feature map that responded strongly around particular kinds of edges.

A colour image has several values at each pixel. For an RGB image, these are stored in separate red, green and blue input channels. In the cat-image filtering example from the ENGSCI 205 lecture notes, the same edge-detection filter was applied separately to these three channels. In a standard convolutional layer, each filter instead contains a separate set of weights for each input channel and combines all three channels when calculating each output value. One filter produces one feature map; a layer containing several filters produces several feature maps.

In the edge-detection exercise, you specified the filter weights manually. In the LeNet-style network, the filters were instead learned from the MNIST training data. They were updated using the same loss, backpropagation and optimiser steps as the other model parameters.

6.1 Activation functions

A convolution produces a weighted sum at each location, which may be positive or negative. In your ENGSCI 205 CNN, you followed convolutional layers with the ReLU activation function:

\[ \operatorname{ReLU}(z)=\max(0,z). \]

This replaces negative values with zero and leaves positive values unchanged. It is applied separately to every value in a feature map. For example,

\[ \begin{bmatrix} -2 & 3 & 1\\ 4 & -1 & 0 \end{bmatrix} \quad\longrightarrow\quad \begin{bmatrix} 0 & 3 & 1\\ 4 & 0 & 0 \end{bmatrix}. \]

Why include this operation? If we only stacked convolutional layers, each layer would compute weighted sums of the previous layer’s weighted sums. The final result would still be a weighted sum of the original inputs, possibly with an added bias, i.e. a linear model. Adding more layers would not get us beyond this kind of model.

ReLU introduces a nonlinearity: whether an input contributes to the next layer now depends on whether its value is positive. Combining convolutions with nonlinear activation functions allows the network to learn more complex relationships between the image and the prediction.

ReLU is an elementwise operation and does not change the dimensions of the feature map. Pooling, considered next, combines values from neighbouring locations and can produce a smaller feature map.

6.2 Pooling

Pooling summarises a small region of a feature map using a single value. In ENGSCI 205, you used max pooling, which takes the largest value in each region.

Consider this $4\times4$ feature map. We divide it into four $2\times2$ regions and take the maximum in each:

\[ \left[ \begin{array}{cc|cc} 1&3&2&0\\ 4&6&5&1\\ \hline 0&2&1&3\\ 1&2&4&2 \end{array} \right] \quad\longrightarrow\quad \begin{bmatrix} 6&5\\ 2&4 \end{bmatrix}. \]

For example, the upper-left region contains $1,3,4,6$, so its pooled value is $6$. Repeating this for the other three regions produces the $2\times2$ output.

Here the pooling window has size $2\times2$ and moves two positions at a time. This movement is called the stride. With a stride of 2, these windows do not overlap. The height and width are each halved, reducing 16 values to 4. This reduction in spatial resolution is called downsampling.

Pooling is applied separately to each feature map: it reduces the height and width without combining the channels. Max pooling has no learned weights; the window size and stride are settings we choose.

6.2.1 Why use pooling?

Max pooling preserves the strongest response within each region. If a filter responds strongly to an edge somewhere in that region, the pooled value retains that response without recording exactly where it occurred. In the example above, moving the value $6$ to another position within the upper-left region would still give the pooled value $6$.

This can be useful for image classification, where a small shift in the position of an image feature should not necessarily change the predicted class. The smaller feature maps also require less memory and reduce the computation needed by subsequent convolutional layers.

However, pooling discards information. From the pooled value $6$, we cannot recover the other three values or determine which position originally contained the maximum. This loss of spatial detail matters when we need precise object locations rather than only an image-level class.

6.3 Stride and padding

Convolutional layers also have a stride, specifying how far the filter moves between successive applications. A stride of 1 moves the filter one position at a time, so neighbouring input patches overlap.

Consider a $28\times28$ MNIST image and a $5\times5$ filter. If we require the filter to fit entirely inside the image, its horizontal positions run from columns $1$–$5$ through to columns $24$–$28$. There are therefore 24 possible horizontal positions and, similarly, 24 vertical positions. Applying the filter with stride 1 produces a $24\times24$ feature map.

6.3.1 Padding

In the ENGSCI 205 cat-image example, we used zero padding to obtain a patch centred at each pixel, including pixels near the boundary. For a $5\times5$ patch centred on an edge pixel, part of the patch lies outside the original image. Zero padding supplies these missing values by adding a border of zeros.

For the $5\times5$ filter, adding two rows or columns of zeros on each side turns the $28\times28$ input into a $32\times32$ padded array. There are now

\[ 32-5+1=28 \]

positions in each direction where the filter fits. With stride 1, the output feature map therefore has the same height and width as the original image.

6.3.2 Downsampling with a convolution

A larger stride skips some filter positions and produces a smaller output. For the same $28\times28$ image, $5\times5$ filter and padding of 2, changing the stride from 1 to 2 reduces the output from $28\times28$ to $14\times14$.

Thus, a convolutional layer can perform downsampling itself. Unlike max pooling, it computes a learned weighted sum at each selected location rather than taking the maximum. Both approaches are used in convolutional networks.

6.4 Combining the layers

We can combine these operations using nn.Sequential, as you did in ENGSCI 205. It applies the layers in the order listed:

import torch
from torch import nn

model = nn.Sequential(
    nn.Conv2d(in_channels=1, out_channels=6,
              kernel_size=5, stride=1, padding=2),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
)

The convolutional layer takes one input channel and applies six learned filters. Each filter has size $5\times5$. The padding of 2 and stride of 1 preserve the input’s height and width. ReLU then acts on each output value, and max pooling halves the height and width.

To check the dimensions, we can use random numbers in place of an image. The following creates a batch containing one single-channel $28\times28$ array:

x = torch.rand(1, 1, 28, 28)
print(x.shape)

for layer in model:
    x = layer(x)
    print(x.shape)

This prints:

torch.Size([1, 1, 28, 28])
torch.Size([1, 6, 28, 28])
torch.Size([1, 6, 28, 28])
torch.Size([1, 6, 14, 14])

Reading each shape as (batch, channels, height, width):

The input contains one image with one channel.
The convolution produces six feature maps, each $28\times28$.
ReLU leaves these dimensions unchanged.
Pooling reduces each feature map to $14\times14$, retaining all six channels.

The convolutional weights have been initialised but not trained; their values do not affect these output dimensions. The result is still a collection of feature maps. A classifier needs further layers to turn those features into class scores.

6.4.1 Classification output

Our pooling layer produces six $14\times14$ feature maps for each image. To use these for classification, we can flatten the channel, height and width dimensions, leaving the batch dimension unchanged. The tensor shape changes from (batch, 6, 14, 14) to (batch, 1176), since

\[ 6\times14\times14=1176. \]

A fully connected layer can then map these 1176 features to ten class scores, one for each MNIST digit from 0 to 9. Continuing with the pooled feature maps stored in x:

classifier = nn.Sequential(
    nn.Flatten(),
    nn.Linear(6 * 14 * 14, 10),
)

scores = classifier(x)
print(scores.shape)

This prints:

torch.Size([1, 10])

The first dimension is still the batch size. The second contains the ten class scores for our one image. Each score is a weighted sum of the 1176 features, with an added bias; the largest score determines the predicted class.

6.4.2 Softmax

The class scores can be positive or negative and do not sum to one. As you saw in ENGSCI 205, softmax converts these scores into the model’s probability distribution over classes.

For class scores $z_c$, softmax gives

\[ p_c=\frac{\exp(z_c)}{\sum_j\exp(z_j)}, \]

where the sum runs over all classes. Taking exponentials makes the values positive, and dividing by their sum makes the resulting probabilities sum to one.

For a small example with three classes, suppose the scores are

\[ z=(2,1,0). \]

Their exponentials are approximately $(7.389,2.718,1)$, which sum to $11.107$. Dividing each by this sum gives

\[ p\approx(0.665,\;0.245,\;0.090). \]

The first class has the largest score and the largest probability. Softmax preserves the ordering of the scores, so it does not change which class we select.

For our MNIST example, we apply softmax along the class dimension:

softmax = nn.Softmax(dim=1)
probabilities = softmax(scores)

Here dim=1 is the second dimension of (batch, classes). The output still has shape (1, 10), but the ten values now form a probability distribution for our image.

The complete calculation is therefore

\[ \text{image} \rightarrow \text{convolution} \rightarrow \text{ReLU} \rightarrow \text{pooling} \rightarrow \text{flattening} \rightarrow \text{class scores} \rightarrow \text{softmax}. \]

This defines a simple classifier, but its parameters still need to be trained on labelled images before it can recognise digits. In ENGSCI 205, you used nn.LogSoftmax, which computes log-probabilities, together with nn.NLLLoss for training. We will return to that pairing when reviewing the training loop.

We will also use softmax when studying attention. There it converts scores for different sequence positions into weights for combining information, rather than probabilities over image classes.

# Image modelling: From pixels to learned features In ENGSCI 205, you built and trained a convolutional neural network for classifying MNIST images. You first applied a manually specified edge-detection filter and inspected the resulting feature map, then constructed a LeNet-style network containing convolutional and pooling layers. Here we briefly recall how those layers construct learned image representations before connecting them to modern object-detection models. The MNIST images used in ENGSCI 205 were greyscale, so each image had one input channel. When you applied a $5\times5$ edge-detection filter, the filter was placed over a $5\times5$ patch of pixel values. The pixel values and filter weights were multiplied and summed to produce one output value. Repeating this calculation across the image produced a two-dimensional feature map that responded strongly around particular kinds of edges. A colour image has several values at each pixel. For an RGB image, these are stored in separate red, green and blue input channels. In the cat-image filtering example from the ENGSCI 205 lecture notes, the same edge-detection filter was applied separately to these three channels. In a standard convolutional layer, each filter instead contains a separate set of weights for each input channel and combines all three channels when calculating each output value. One filter produces one feature map; a layer containing several filters produces several feature maps. In the edge-detection exercise, you specified the filter weights manually. In the LeNet-style network, the filters were instead learned from the MNIST training data. They were updated using the same loss, backpropagation and optimiser steps as the other model parameters. ## Activation functions A convolution produces a weighted sum at each location, which may be positive or negative. In your ENGSCI 205 CNN, you followed convolutional layers with the **ReLU** activation function: $$ \operatorname{ReLU}(z)=\max(0,z). $$ This replaces negative values with zero and leaves positive values unchanged. It is applied separately to every value in a feature map. For example, $$ \begin{bmatrix} -2 & 3 & 1\\ 4 & -1 & 0 \end{bmatrix} \quad\longrightarrow\quad \begin{bmatrix} 0 & 3 & 1\\ 4 & 0 & 0 \end{bmatrix}. $$ Why include this operation? If we only stacked convolutional layers, each layer would compute weighted sums of the previous layer’s weighted sums. The final result would still be a weighted sum of the original inputs, possibly with an added bias, i.e. a linear model. Adding more layers would not get us beyond this kind of model. ReLU introduces a **nonlinearity**: whether an input contributes to the next layer now depends on whether its value is positive. Combining convolutions with nonlinear activation functions allows the network to learn more complex relationships between the image and the prediction. ReLU is an **elementwise** operation and does not change the dimensions of the feature map. **Pooling**, considered next, combines values from neighbouring locations and can produce a smaller feature map. ## Pooling Pooling summarises a small region of a feature map using a single value. In ENGSCI 205, you used **max pooling**, which takes the largest value in each region. Consider this $4\times4$ feature map. We divide it into four $2\times2$ regions and take the maximum in each: $$ \left[ \begin{array}{cc|cc} 1&3&2&0\\ 4&6&5&1\\ \hline 0&2&1&3\\ 1&2&4&2 \end{array} \right] \quad\longrightarrow\quad \begin{bmatrix} 6&5\\ 2&4 \end{bmatrix}. $$ For example, the upper-left region contains $1,3,4,6$, so its pooled value is $6$. Repeating this for the other three regions produces the $2\times2$ output. Here the pooling window has size $2\times2$ and moves two positions at a time. This movement is called the **stride**. With a stride of 2, these windows do not overlap. The height and width are each halved, reducing 16 values to 4. This reduction in spatial resolution is called **downsampling**. Pooling is applied separately to each feature map: it reduces the height and width without combining the channels. Max pooling has no learned weights; the window size and stride are settings we choose. ### Why use pooling? Max pooling preserves the strongest response within each region. If a filter responds strongly to an edge somewhere in that region, the pooled value retains that response without recording exactly where it occurred. In the example above, moving the value $6$ to another position within the upper-left region would still give the pooled value $6$. This can be useful for image classification, where a small shift in the position of an image feature should not necessarily change the predicted class. The smaller feature maps also require less memory and reduce the computation needed by subsequent convolutional layers. However, pooling discards information. From the pooled value $6$, we cannot recover the other three values or determine which position originally contained the maximum. This loss of spatial detail matters when we need precise object locations rather than only an image-level class. ## Stride and padding Convolutional layers also have a **stride**, specifying how far the filter moves between successive applications. A stride of 1 moves the filter one position at a time, so neighbouring input patches overlap. Consider a $28\times28$ MNIST image and a $5\times5$ filter. If we require the filter to fit entirely inside the image, its horizontal positions run from columns $1$–$5$ through to columns $24$–$28$. There are therefore 24 possible horizontal positions and, similarly, 24 vertical positions. Applying the filter with stride 1 produces a $24\times24$ feature map. ### Padding In the ENGSCI 205 cat-image example, we used **zero padding** to obtain a patch centred at each pixel, including pixels near the boundary. For a $5\times5$ patch centred on an edge pixel, part of the patch lies outside the original image. Zero padding supplies these missing values by adding a border of zeros. For the $5\times5$ filter, adding two rows or columns of zeros on each side turns the $28\times28$ input into a $32\times32$ padded array. There are now $$ 32-5+1=28 $$ positions in each direction where the filter fits. With stride 1, the output feature map therefore has the same height and width as the original image. ### Downsampling with a convolution A larger stride skips some filter positions and produces a smaller output. For the same $28\times28$ image, $5\times5$ filter and padding of 2, changing the stride from 1 to 2 reduces the output from $28\times28$ to $14\times14$. Thus, a convolutional layer can perform downsampling itself. Unlike max pooling, it computes a learned weighted sum at each selected location rather than taking the maximum. Both approaches are used in convolutional networks. ## Combining the layers We can combine these operations using `nn.Sequential`, as you did in ENGSCI 205. It applies the layers in the order listed: ```python import torch from torch import nn model = nn.Sequential( nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5, stride=1, padding=2), nn.ReLU(), nn.MaxPool2d(kernel_size=2, stride=2), ) ``` The convolutional layer takes one input channel and applies six learned filters. Each filter has size $5\times5$. The padding of 2 and stride of 1 preserve the input’s height and width. ReLU then acts on each output value, and max pooling halves the height and width. To check the dimensions, we can use random numbers in place of an image. The following creates a batch containing one single-channel $28\times28$ array: ```python x = torch.rand(1, 1, 28, 28) print(x.shape) for layer in model: x = layer(x) print(x.shape) ``` This prints: ```text torch.Size([1, 1, 28, 28]) torch.Size([1, 6, 28, 28]) torch.Size([1, 6, 28, 28]) torch.Size([1, 6, 14, 14]) ``` Reading each shape as `(batch, channels, height, width)`: - The input contains **one image with one channel**. - The convolution produces **six feature maps**, each $28\times28$. - ReLU leaves these dimensions unchanged. - Pooling reduces each feature map to $14\times14$, retaining all six channels. The convolutional weights have been initialised but not trained; their values do not affect these output dimensions. The result is still a collection of feature maps. A classifier needs further layers to turn those features into class scores. ### Classification output Our pooling layer produces six $14\times14$ feature maps for each image. To use these for classification, we can **flatten** the channel, height and width dimensions, leaving the batch dimension unchanged. The tensor shape changes from `(batch, 6, 14, 14)` to `(batch, 1176)`, since $$ 6\times14\times14=1176. $$ A fully connected layer can then map these 1176 features to ten class scores, one for each MNIST digit from 0 to 9. Continuing with the pooled feature maps stored in `x`: ```python classifier = nn.Sequential( nn.Flatten(), nn.Linear(6 * 14 * 14, 10), ) scores = classifier(x) print(scores.shape) ``` This prints: ```text torch.Size([1, 10]) ``` The first dimension is still the batch size. The second contains the ten class scores for our one image. Each score is a weighted sum of the 1176 features, with an added bias; the largest score determines the predicted class. ### Softmax The class scores can be positive or negative and do not sum to one. As you saw in ENGSCI 205, **softmax** converts these scores into the model’s probability distribution over classes. For class scores $z_c$, softmax gives $$ p_c=\frac{\exp(z_c)}{\sum_j\exp(z_j)}, $$ where the sum runs over all classes. Taking exponentials makes the values positive, and dividing by their sum makes the resulting probabilities sum to one. For a small example with three classes, suppose the scores are $$ z=(2,1,0). $$ Their exponentials are approximately $(7.389,2.718,1)$, which sum to $11.107$. Dividing each by this sum gives $$ p\approx(0.665,\;0.245,\;0.090). $$ The first class has the largest score and the largest probability. Softmax preserves the ordering of the scores, so it does not change which class we select. For our MNIST example, we apply softmax along the class dimension: ```python softmax = nn.Softmax(dim=1) probabilities = softmax(scores) ``` Here `dim=1` is the second dimension of `(batch, classes)`. The output still has shape `(1, 10)`, but the ten values now form a probability distribution for our image. The complete calculation is therefore $$ \text{image} \rightarrow \text{convolution} \rightarrow \text{ReLU} \rightarrow \text{pooling} \rightarrow \text{flattening} \rightarrow \text{class scores} \rightarrow \text{softmax}. $$ This defines a simple classifier, but its parameters still need to be trained on labelled images before it can recognise digits. In ENGSCI 205, you used **`nn.LogSoftmax`**, which computes log-probabilities, together with **`nn.NLLLoss`** for training. We will return to that pairing when reviewing the training loop. We will also use softmax when studying attention. There it converts scores for different sequence positions into weights for combining information, rather than probabilities over image classes.