02_CNN_Introduction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "private_outputs": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MatchLab-Imperial/deep-learning-course/blob/master/02_CNN_Introduction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5uoxdNjkDw2K"
      },
      "source": [
        "# Introduction to Convolutional Neural Networks\n",
        "\n",
        "This tutorial addresses the basic concepts regarding Convolutional Neural Networks and their implementation using the Keras framework.\n",
        "Convolutional Neural Networks (CNNs) are a class of feed-forward artificial neural architecture. They are applied to analyse visual 2D imagery, meaning that we can feed images directly into a CNN without the need to flatten them into a 1D vector as done in the previous tutorial.\n",
        "CNNs have revolutionised the field of computer vision in the last decade. In 2012 Alex Krizhevsky introduced the AlexNet architecture to win the ImageNet Challenge (one of the most important competitions on image classification within the Computer Vision community), by reducing the top-5 error more than 10 percentage points, which was an incredible improvement at that time. As of now, CNNs are used not only on image classification but in many other computer vision tasks.\n",
        "\n",
        "![](https://cdn-5f733ed3c1ac190fbc56ef88.closte.com/wp-content/uploads/2017/03/alexnet_small-1.png)\n",
        "\n",
        "The image above is from [cv-tricks' blog](https://cv-tricks.com/cnn/understand-resnet-alexnet-vgg-inception/) and shows the proposed AlexNet architecture. It is composed of 5 convolutional layers followed by 3 fully connected layers. Nowadays, we can find much deeper and more complex architectures, which outperform AlexNet on the ImageNet Challenge.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJ2d_oQ2UEPz"
      },
      "source": [
        "## CNN Structure\n",
        "\n",
        "The basic pipeline of common CNNs consists of an image as input and a stack of convolutional layers that extract a feature representation from the input image. The final shape of the image representation is conditioned on the type of problem/task that the architecture is facing. For instance, the output of the last layer in a classification problem is a probability vector. Each dimension of the probability vector represents how likely is that the input image belongs to a specific class. However, the architecture design is up to us, and therefore, we could code a network that outputs a single value for regression problems, or that generates a new image map for semantic segmentation. Now, let's dig in a bit into CNN and introduce some layers that are widely used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvNKve9GZL4M"
      },
      "source": [
        "### 2D Convolutional Layer\n",
        "\n",
        "The most common layer in any CNN architecture is the 2D convolutional layer. Convolutional layers are specifically designed to extract features from images or even extract features from previously extracted features. As shown in the following illustration, 2D convolutions apply the same filter to the full image. And therefore, due to its nature, 2D convolutional filters exploit the local information presented in images, making them a powerful tool for image analysis.\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/800/1*Fw-ehcNBR9byHtho-Rxbtw.gif)\n",
        "\n",
        "Image [source](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1).\n",
        "\n",
        "The latest deep learning frameworks have made possible the integration of convolutional layers easily on our architectures with only a single line of code. We will address here how 2D convolutions work since full understanding is needed to comprehend how any CNN operates. The following images and some explanations can be further explored on the original [Irhum Shafkat's blog](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1) or in the [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/) book. Both are strongly recommended.\n",
        "\n",
        "To understand 2D convolutions, we need to define first what a kernel is. Kernels are simply matrices of numbers. The numbers on the kernels are the so-called weights, and the weights on the kernels change as we train the network. Hence, network training aims to optimise the network's parameters (update the kernel weights) such that the cost function is minimised.\n",
        "\n",
        "The 2D convolution operation takes the network's kernels and \"slides\" them over the input image (alike to a sliding window) as in the following image from [PyImageSearch](https://www.pyimagesearch.com/2015/03/23/sliding-windows-for-object-detection-with-python-and-opencv/) blog:\n",
        "\n",
        "![](https://pyimagesearch.com/wp-content/uploads/2014/10/sliding_window_example.gif)\n",
        "\n",
        "In each step, the network performs an element-wise multiplication with the elements that are currently on. The results of this elementwise multiplication are added to obtain the output value of the operation. CNNs repeat previous step for all the positions of the sliding window, composing at the end the feature map. This generated feature map can go through another 2D convolutional layer and create more powerful features.\n",
        "\n",
        ">\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/800/1*Zx-ZMLKab7VOCQTxdZ1OAw.gif)\n",
        "\n",
        ">\n",
        "The previous image shows the 2D convolution operation. Thus, the new feature values are the weighted sum of all the elements in the sliding window after the elementwise multiplication between input and kernel.  The bigger the size of the kernel is, the more feature elements contribute to the final output value. In contrast to fully connected layers, where a new feature value is a weighted sum over **all** input values, as mentioned, 2D convolutions compute features based on local areas. In other words, instead of looking at every input component, they consider only features coming from close locations.\n",
        "\n",
        "In the above example, the input image on the left has a size of 5x5 and the dimension of the resulting feature map is 3x3, showing that the size of the output maps is not always equal to the input. Hence, the output size can be computed by doing:\n",
        "\n",
        "$O = W - K + 1$,\n",
        "\n",
        "where $O$ is the output height/length, $W$ is the input height/length and $K$ is the kernel size. The output size is not only conditioned on the input size but also on the kernel size. Check in the following code cell how the output feature map shape changes as you increase the kernel size. In Keras, we define the layer by using `Conv2D` from `keras.layers` (documentation [here](https://keras.io/layers/convolutional/#conv2d)).\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ez90vQEksR9b"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.layers import Conv2D, Input\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "# Generate dummy data\n",
        "input_feature = np.random.random((1, 100, 100, 1))\n",
        "\n",
        "# input: 100x100 image with 1 channels -> (100, 100, 1) tensor.\n",
        "# this applies 1 convolution filter of size 3x3 each.\n",
        "model = Sequential()\n",
        "model.add(Input(shape=(100, 100, 1)))\n",
        "model.add(Conv2D(filters=1, kernel_size=(3, 3)))\n",
        "\n",
        "output_feature = model.predict(input_feature)\n",
        "\n",
        "print('Input size: ({:}, {:})'.format(input_feature.shape[1], input_feature.shape[2]))\n",
        "print('Output size: ({:}, {:})'.format(output_feature.shape[1], output_feature.shape[2]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mNOhth4FuNob"
      },
      "source": [
        "Moreover, the kernel size and the input size are not the only parameters affecting the output size. We are going to introduce two extra elements that change the size of the output map: the padding and the stride."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mT6f_n5QiU8X"
      },
      "source": [
        "### Adding Padding to Input Features\n",
        "\n",
        "In some tasks, such as [image translation](https://arxiv.org/pdf/1611.07004.pdf), we need the output size to be equal to the input size. The solution to that is using padding, where extra edges are added to the input features so that the dimension is not reduced after the convolutional layer. Normally those pixels have $0$ value (termed zero-padding), but depending on the application other methods could be used, e.g., reflection or symmetric padding.\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/800/1*1okwhewf5KCtIPaFib4XaA.gif)\n",
        "\n",
        "If padding is used, the new output size can be computed by doing:\n",
        "\n",
        "$O = W - K + 2P+ 1$,\n",
        "\n",
        "where $P$ is the padding value. $P$ must be set in concordance with the kernel size if dimensionality wants to be preserved. Padding can be added to the `Conv2d` layer in Keras by using the padding argument and selecting one of the two settings: `valid`, which is the default value and means no padding; and `same`, which adapts the padding value to have the same output size as input size."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X7Z5G2O00Etd"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.layers import Conv2D, Input\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "# Generate dummy data\n",
        "input_feature = np.random.random((1, 100, 100, 1))\n",
        "\n",
        "# input: 100x100 image with 1 channels -> (100, 100, 1) tensor.\n",
        "# this applies 1 convolution filter of size 3x3 each.\n",
        "# attribute padding='same' applies zero-padding to the input feature map\n",
        "model = Sequential()\n",
        "model.add(Input(shape=(100, 100, 1)))\n",
        "model.add(Conv2D(filters=1, kernel_size=(3, 3), padding='same'))\n",
        "\n",
        "output_feature = model.predict(input_feature)\n",
        "\n",
        "print('Input size: ({:}, {:})'.format(input_feature.shape[1], input_feature.shape[2]))\n",
        "print('Output size: ({:}, {:})'.format(output_feature.shape[1], output_feature.shape[2]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tdB4jMeswXVA"
      },
      "source": [
        "### Using Stride in Convolutional Layers\n",
        "\n",
        "The stride operation allows the convolutional layers to skip some of the sliding windows explained above. Hence, instead of jumping one pixel apart, we can define the number of skipped elements before computing the weighting sum between the kernel's weights and input features. A stride of 1 means that features will be extracted from all windows a pixel apart, so basically, every single window is computed. A stride of 2 means that we are selecting windows 2 pixels apart, skipping every other window in the process. Strides reduce the number of computations and consequently the size of the output map. In practice, as we go deeper into the CNN, the spatial size of the feature map gets smaller while the number of channels increases. Moreover, we can further reduce the size of the feature map using pooling operations, which we introduce later in this tutorial.\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/800/1*BMngs93_rm2_BpJFH2mS0Q.gif)\n",
        "\n",
        "If strides are used, the new output size can be computed as:\n",
        "\n",
        "$O = \\dfrac{W - K + 2P}{S}+ 1$,\n",
        "\n",
        "where $S$ is the stride value. The stride is set in the layer by using the `strides` argument."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y8GjztiH0hnw"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.layers import Conv2D, Input\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "# Generate dummy data\n",
        "input_feature = np.random.random((1, 100, 100, 1))\n",
        "\n",
        "# input: 100x100 image with 1 channels -> (100, 100, 1) tensor.\n",
        "# this applies 1 convolution filter of size 3x3 each.\n",
        "# attribute padding='same' applies zero-padding to the input feature map\n",
        "# attribute strides=2 applies applies stride of 2\n",
        "model = Sequential()\n",
        "model.add(Input(shape=(100, 100, 1)))\n",
        "model.add(Conv2D(filters=1, kernel_size=(3, 3), padding='same', strides=2))\n",
        "\n",
        "output_feature = model.predict(input_feature)\n",
        "\n",
        "print('Input size: ({:}, {:})'.format(input_feature.shape[1], input_feature.shape[2]))\n",
        "print('Output size: ({:}, {:})'.format(output_feature.shape[1], output_feature.shape[2]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-fKvgJ3H2gr"
      },
      "source": [
        "## Differences Between Kernel and Filters\n",
        "\n",
        "The examples above take as input a single-channel image and compute a feature map with also one channel. However, when dealing with RGB images or feature maps, the input is no longer a single-channel map but, instead, they can have multiple channels. In the case of an RGB image, for each 2D convolution, we will need to define 3 kernels to interact with each of the image's channel colours. This group of kernels is called a filter. Thus, a filter is a collection of kernels that produces a single output.\n",
        "\n",
        "As a regular practice when defining Deep Learning models, we increase the number of filters in each convolutional layer as we go deeper into the model. Due to the element-wise multiplication, the number of kernels on each filter must be the same that the number of channels in the input feature map. Keras already deals with the number of kernels inside each filter. Keras keeps track of the input size in each convolutional layer unlike other frameworks (Pytorch or TensorFlow). Hence, in Keras, we must only decide the number of filters (output channels) in each layer.\n",
        "\n",
        "The next figure shows how the convolution is performed when having three input channels. First, one filter uses its three independent kernels to convolve with the RGB channels of the input image:\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/1000/1*8dx6nxpUh2JqvYWPadTwMQ.gif)\n",
        "\n",
        "Next, each of the processed feature maps is added together to obtain a single channel:\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/1000/1*CYB2dyR3EhFs1xNLK8ewiA.gif)\n",
        "\n",
        "Finally, we add the bias term to obtain the feature map. There is a single bias for the full output channel map. This operation is repeated for all the filters inside the convolutional layer.\n",
        "\n",
        "Now, we show how to use a `Conv2D` layer that takes an input image with 3 channels and generates an output map with 32 channels.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nqiEwpdROxga"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.layers import Conv2D, Input\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "# Generate dummy data\n",
        "input_feature = np.random.random((1, 100, 100, 3))\n",
        "\n",
        "# input: 100x100 image with 3 channels -> (100, 100, 3) tensor.\n",
        "# this applies 32 convolution filters of size 3x3 each.\n",
        "# attribute padding='same' applies zero-padding to the input feature map\n",
        "# attribute strides=2 applies applies stride of 2\n",
        "model = Sequential()\n",
        "model.add(Input(shape=(100, 100, 3)))\n",
        "model.add(Conv2D(filters=32, kernel_size=(3, 3), padding='same', strides=2))\n",
        "\n",
        "output_feature = model.predict(input_feature)\n",
        "\n",
        "print('Input size: ({:}, {:}, {:})'.format(input_feature.shape[1], input_feature.shape[2], input_feature.shape[3]))\n",
        "print('Output size: ({:}, {:}, {:})'.format(output_feature.shape[1], output_feature.shape[2], output_feature.shape[3]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S50cYwkhPqka"
      },
      "source": [
        "## Activation Functions\n",
        "\n",
        "As seen in previous tutorials, after a `Dense` layer we usually can find an activation function. We introduce here how to use them after `Conv2D` layers. Those activation functions are a set of operators that maps the feature values to a new set of values, the mapping depends on the function at hand. The main reason for using activation functions is that they add non-linearities to the network, giving more expressive power to the network, which will be able to reproduce more complex functions.\n",
        "\n",
        "\n",
        "The complete list of activation functions that Keras offers can be found [here](https://keras.io/activations/). We introduce here some of them:\n",
        "\n",
        "*  **Sigmoid Function** sets the output in the range (0, 1). The sigmoid function is widely used in binary classification problems since its output can be taken as a probability value. `keras.activations.sigmoid(x)`:\n",
        "\n",
        ">![](https://i.ibb.co/Ph8dsTv/sigmoid.png)\n",
        "\n",
        "*  **Tanh Function** is a logistic function as sigmoid, but the range of the tanh function is (-1, 1). Contrary to sigmoid function, where the values close to 0 are set around 0.5, in the tanh function they will be still mapped around the 0 value. `keras.activations.tanh(x)`:\n",
        "\n",
        ">![](https://i.ibb.co/68g7LpL/tanh.png)\n",
        "\n",
        "*  **ReLU Function** is the most common activation function you can find in any current CNN as in general works better than the rest. The range of this function is in \\[0, inf). It sets all negative values to 0 and hence is computationally easy to implement. As a drawback, during training some neurons *die*, meaning that the output is 0 for all available data points and no gradient is propagated there. `keras.activations.relu(x, alpha=0.0, max_value=None, threshold=0.0)`:\n",
        "\n",
        ">![](https://i.ibb.co/Zd9H8Z4/relu.png)\n",
        "\n",
        "*  **LeakyReLU Function** is a modified version of the ReLU activation above, which attempts to solve the problem of dying neurons that ReLU has. While ReLU does not backpropagate negative values, Leaky ReLU smooths those values without setting them to 0. That allows the gradients to backpropagate through the network even for negative values. `keras.layers.LeakyReLU(alpha=0.3)`:\n",
        "\n",
        ">![](https://i.ibb.co/dmnJ6h1/leakyrelu.png)\n",
        "\n",
        "*  **Softmax Function** is another widely activation function for multi-class classification problems and usually is employed as the last activation function in the classification model. This function sets all of the output elements to the range (0, 1). However, the softmax function does not take independently the input values to map it into its probability value. Softmax Function takes an un-normalized vector, $s$, and normalizes it into a probability distribution, $p$, following the softmax expression. As the output is a probability, the output elements add up to 1. `keras.activations.softmax(x, axis=-1)`. Thus, the output value $p_i$ is computed as:\n",
        "\n",
        "> $p_{i} = \\dfrac{e^{s_i}}{\\sum_{\\substack{j}}^{N} e^{s_j}}$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-MWo2VsYr5T2"
      },
      "source": [
        "The following example shows the feature maps before and after of the ReLU activation function. All values that are negative are set to 0 after the activation function.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lp1HH0VOqN0N"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.layers import Activation, Conv2D, Input\n",
        "from tensorflow.keras.models import Model\n",
        "\n",
        "# Generate dummy data\n",
        "input_feature = np.random.random((1, 3, 3, 1)) - 0.5\n",
        "\n",
        "# input: 3x3 image with 1 channel -> (3, 3, 1) tensor.\n",
        "# This applies a 1 convolution filter of size 3x3 each.\n",
        "# This applies a ReLU activation function\n",
        "input_tensor = Input(shape=(3, 3, 1))\n",
        "output_tensor = Conv2D(filters=1, kernel_size=(3, 3), padding='same', strides=1, name='conv_2D')(input_tensor)\n",
        "output_tensor = Activation('relu')(output_tensor)\n",
        "\n",
        "model = Model(inputs=input_tensor, outputs=output_tensor)\n",
        "model_before_ReLU = Model(inputs=model.input, outputs=model.get_layer('conv_2D').output)\n",
        "\n",
        "output_feature = model_before_ReLU.predict(input_feature)\n",
        "output_ReLU_feature = model.predict(input_feature)\n",
        "\n",
        "print('Output Network without activation function')\n",
        "print(output_feature)\n",
        "\n",
        "print('')\n",
        "print('Output Network after ReLU activation function')\n",
        "print(output_ReLU_feature)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ys6ZB1DOTLH6"
      },
      "source": [
        "## Pooling Layer\n",
        "\n",
        "It is a common practice to insert a pooling layer between convolutional layers in CNNs. In a standard CNN architecture, we set the feature sizes to become smaller progressively to reduce the computation in the networks, and to merge the information from different spatial locations. To reduce the feature map sizes, we can either use bigger stride size in the convolutional layers or we can use pooling layers. Pooling layers perform a spatial sliding window and apply an operation to reduce the spatial size. Those operations vary depending on the architecture, being the max, mean and min pooling the most typical ones. Here, we will explain the max pooling, although all the others work similarly. Max pooling keeps only the max value in a neighbourhood, where the neighbourhood is defined by the size of the kernel. Let's visualise it, the next example shows the result of a Max Pooling layer with a 2x2 kernel and a stride of 2.\n",
        "\n",
        "![](https://i.ibb.co/Xp454S4/MaxPool.png)\n",
        "\n",
        "As in convolutional layers, the final size is conditioned to the stride size of the pooling layer. However, contrary to convolutional layers, pooling layers operate independently on each of the input channels, without modifying the depth of the feature maps. To add max pooling to our model we need to import `MaxPooling2d` from `keras.layers` and define the stride and pooling size."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzoTGOwDmQm4"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.layers import Activation, Conv2D, Input, MaxPooling2D\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "# Generate dummy data\n",
        "input_feature = np.random.random((1, 100, 100, 3))\n",
        "\n",
        "# input: 100x100 image with 3 channels -> (100, 100, 3) tensor.\n",
        "# this applies 32 convolution filters of size 3x3 each.\n",
        "# attribute padding='same' applies zero-padding to the input feature map\n",
        "# attribute strides=1 applies applies stride of 1\n",
        "model = Sequential()\n",
        "model.add(Input(shape=(100, 100, 3)))\n",
        "model.add(Conv2D(filters=32, kernel_size=(3, 3), padding='same', strides=1))\n",
        "model.add(Activation('relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))\n",
        "\n",
        "output_feature = model.predict(input_feature)\n",
        "\n",
        "print('Input size: ({:}, {:}, {:})'.format(input_feature.shape[1], input_feature.shape[2], input_feature.shape[3]))\n",
        "print('Output size: ({:}, {:}, {:})'.format(output_feature.shape[1], output_feature.shape[2], output_feature.shape[3]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LKU4-GbZHw0R"
      },
      "source": [
        "# Example: Classification on MNIST\n",
        "\n",
        "As opposed to the *Introduction to Keras* tutorial, in this section, we show how to perform image classification when the input data is a 2D image instead of a flat 1D vector.\n",
        "\n",
        "As discussed above, Convolutional Neural Networks aim to extract and exploit the local relationships on 2D maps, hence, CNNs are much more convenient for images than Multi-layer Perceptron models.\n",
        "\n",
        "First of all, we load the MNIST dataset from Keras' framework. The definition of the data is almost identical than in the Keras tutorial, although this time we are not reshaping the input images into a 1D vector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3t0Wq9ArL4EZ"
      },
      "source": [
        "import numpy as np\n",
        "from tensorflow.keras.datasets import mnist\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "# the data, shuffled and split between train and test sets\n",
        "# Here we are using the official test set as our validation set, in further\n",
        "# tutorials, test and validation splits will be explained properly.\n",
        "# Hence, even though the variables are `x_test` and `y_test`, they represent our validation set\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "shape = x_train.shape\n",
        "\n",
        "# Normalize and reshape the input images\n",
        "x_train = np.expand_dims(x_train.astype('float32'), -1)\n",
        "x_test = np.expand_dims(x_test.astype('float32'), -1)\n",
        "\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "y_train_class = to_categorical(y_train, 10)\n",
        "y_test_class = to_categorical(y_test, 10)\n",
        "\n",
        "print('Image shape: {0}'.format(x_train.shape[1:]))\n",
        "print('Total number of training samples: {0}'.format(x_train.shape[0]))\n",
        "print('Total number of validation samples: {0}'.format(x_test.shape[0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Wp29GszMO-j"
      },
      "source": [
        "As illustrated in the cell, this time the input image before the model is 28x28x1.\n",
        "\n",
        "Now we can define a model composed of convolutional layers, activation functions, and maxpool operators:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B78NUAqwMeFR"
      },
      "source": [
        "from tensorflow.keras.layers import Activation, BatchNormalization, Conv2D, MaxPooling2D\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Input(shape=x_train.shape[1:]))\n",
        "model.add(Conv2D(filters=16, kernel_size=(3,3), padding='same'))\n",
        "model.add(Activation('relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "model.add(Conv2D(filters=16, kernel_size=(3,3), padding='same'))\n",
        "model.add(Activation('relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BX_KAFlBMnVZ"
      },
      "source": [
        "As explained in the previous tutorial, in a classification problem, the output of the model is a probability vector. Each dimension of the vector indicates how likely is that the input image belongs to a specific class.\n",
        "\n",
        "Up to now, the resulting feature map of the model is a map with the shape *Batch x Weight' x Height' x Channel*, and it needs to be mapped into a vector with shape *Batch x Num Classes*. A common technique to process this mapping is to add a Flatten layer that will reshape the feature map to *Batch x (Weight' * Height' * Channel)*. Following the Flatten layer, we add a dense layer which maps this new feature map to the desired output size."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3CAc1r2vOJQv"
      },
      "source": [
        "from tensorflow.keras.layers import Activation, Dense, Flatten\n",
        "\n",
        "model.add(Flatten())\n",
        "model.add(Dense(10))\n",
        "model.add(Activation('softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aHInEJ-aON5v"
      },
      "source": [
        "Let's visualize the output shape in each layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UD4NZtIxOTbY"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_F66ds58OTBu"
      },
      "source": [
        "Finally, we can train our CNN and check its performance on MNIST."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BmeolsG_Pi8A"
      },
      "source": [
        "from tensorflow.keras.optimizers import RMSprop\n",
        "\n",
        "# initiate RMSprop optimizer\n",
        "optimizer = RMSprop(learning_rate=1e-4)\n",
        "\n",
        "# Train the model\n",
        "model.compile(\n",
        "    loss='categorical_crossentropy',\n",
        "    optimizer=optimizer,\n",
        "    metrics=['accuracy'],\n",
        ")\n",
        "model.fit(x_train, y_train_class, batch_size=32, epochs=10,  verbose=0)\n",
        "score = model.evaluate(x_test, y_test_class, verbose=0)\n",
        "\n",
        "print('Validation loss:', score[0])\n",
        "print('Validation accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0ZorPBtpLlZ"
      },
      "source": [
        "# Coursework\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2gkM3dPZbNcX"
      },
      "source": [
        "## Task 1: Classification\n",
        "\n",
        "At this point, we know what is a CNN, how they work, and the components needed to design them. In this first task, we want you to create a CNN that is able to outperform the Multi-layer Perceptron model from Tutorial 1. For the first part of the coursework, we train on CIFAR10, a  classical dataset for image classification. Note that in these tutorials, we mainly use the official test sets of several standard datasets as our validation data. The reason we use the given test sets as validation data for the tutorials is that is an easy way to make sure that we all work with the same split and report results using the same data. However, in a proper machine learning setup, your validation set should be separate from the test set, so you can tune the model/parameters on the validation set and then check the final performance in the test set. Thus, even though the variables are `x_test` and `y_test`, they represent our validation set.\n",
        "\n",
        "Let's first load the dataset and visualise some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zKWvv0VwZPH3"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from tensorflow.keras.datasets import cifar10\n",
        "\n",
        "# the data, shuffled and split between train and test sets\n",
        "# Here we are using the official test set as our validation set, in further\n",
        "# tutorials, test and validation splits will be explained properly.\n",
        "# Hence, even though the variables are `x_test` and `y_test`, they represent our validation set\n",
        "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n",
        "\n",
        "print('Image shape: {0}'.format(X_train.shape[1:]))\n",
        "print('Total number of training samples: {0}'.format(X_train.shape[0]))\n",
        "print('Total number of validation samples: {0}'.format(X_test.shape[0]))\n",
        "\n",
        "# Let's visualize some examples\n",
        "N=5\n",
        "start_val = 0 # pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "items = list(range(0, 10))\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    axes[row,col].imshow(X_train[idx], cmap='gray')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    y_target = y_target = y_train[idx].item()\n",
        "    target = str(items[y_target])\n",
        "    axes[row,col].set_title(target)\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-p9uVP3pccOW"
      },
      "source": [
        "CIFAR10 contains 50,000 32x32 color training images, labeled over 10 categories and 10,000 test images. As mentioned, we use those test images as our validation set in this tutorial. We need to preprocess the images for the 1D neural network defined in the previous tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bFurVCqpclnm"
      },
      "source": [
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "# Preprocess dataset for a 1D neural network\n",
        "X_train = X_train.astype('float32')\n",
        "X_test = X_test.astype('float32')\n",
        "\n",
        "# Normalize the image\n",
        "X_train /= 255\n",
        "X_test /= 255\n",
        "\n",
        "X_train_flatten = X_train.reshape(X_train.shape[0], X_train.shape[1] * X_train.shape[2]*3)\n",
        "X_test_flatten = X_test.reshape(X_test.shape[0], X_test.shape[1] * X_test.shape[2]*3)\n",
        "\n",
        "print('New X_train shape: {0}'.format(X_train_flatten.shape))\n",
        "\n",
        "Y_train_class = to_categorical(y_train, 10)\n",
        "Y_test_class = to_categorical(y_test, 10)\n",
        "\n",
        "print('New Y_train shape: {0}'.format(Y_train_class.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mxFx2s3Fc6wM"
      },
      "source": [
        "Now, we are ready to define the Multi-layer Perceptron model and train it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KEZW-1pxbKlC"
      },
      "source": [
        "from tensorflow.keras.layers import Dense, Dropout, Input\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.optimizers import RMSprop\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Input(shape=X_train_flatten.shape[1:]))\n",
        "model.add(Dense(32, activation='relu'))\n",
        "model.add(Dense(64, activation='relu'))\n",
        "model.add(Dense(128, activation='relu'))\n",
        "model.add(Dense(256, activation='relu'))\n",
        "model.add(Dense(512, activation='relu'))\n",
        "model.add(Dense(10, activation='softmax'))\n",
        "\n",
        "# initiate RMSprop optimizer\n",
        "optimizer = RMSprop(learning_rate=1e-4)\n",
        "\n",
        "# Let's train the model using RMSprop\n",
        "model.compile(\n",
        "    loss='categorical_crossentropy',\n",
        "    optimizer=optimizer,\n",
        "    metrics=['accuracy'],\n",
        ")\n",
        "model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=20)\n",
        "score = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "\n",
        "print('Validation loss:', score[0])\n",
        "print('Validation accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CEuYNmyHlVJd"
      },
      "source": [
        "### Problem Definition\n",
        "\n",
        "In this exercise, you are asked to test several CNN architectures in the code provided below. Do not modify the optimizer, loss used or parameters related to the training such as the learning rate, they will be investigated in future tutorials. You must focus on the architecture itself: number of convolutional layers, number of filters in every layer, activation functions, pooling operators, among others. Batch Normalization and Dropout layers, which are quite used in CNN architectures, will be also investigated in a future tutorial so you do not have to discuss them.\n",
        "\n",
        "\n",
        "**Report**:\n",
        "*   Present a bar figure with the training and validation accuracies for different design choices. Discuss only the parameters that have a significant influence on the network's performance. Explain any discrepancy between training and validation accuracies.\n",
        "*   Present a sketch that introduces your best architecture. See some examples on how to display networks in [cv-tricks' blog](https://cv-tricks.com/cnn/understand-resnet-alexnet-vgg-inception/).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r8ATD38dDM1q"
      },
      "source": [
        "from tensorflow.keras.datasets import cifar10\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.optimizers import RMSprop\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "# the data, shuffled and split between train and val sets\n",
        "# Here we are using the official test set as our validation set, in further\n",
        "# tutorials, test and validation splits will be explained properly.\n",
        "# Hence, even though the variables are `x_test` and `y_test`, they represent our validation set\n",
        "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
        "\n",
        "print('Image shape: {0}'.format(x_train.shape[1:]))\n",
        "print('Total number of training samples: {0}'.format(x_train.shape[0]))\n",
        "print('Total number of validation samples: {0}'.format(x_test.shape[0]))\n",
        "\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "\n",
        "# Normalize the image\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "y_train_class = to_categorical(y_train, 10)\n",
        "y_test_class = to_categorical(y_test, 10)\n",
        "\n",
        "\n",
        "# TODO: Define your architecture here\n",
        "model = Sequential()\n",
        "# . . .\n",
        "model.summary()\n",
        "\n",
        "# initiate RMSprop optimizer\n",
        "optimizer = RMSprop(learning_rate=1e-4)\n",
        "\n",
        "model.compile(\n",
        "    loss='categorical_crossentropy',\n",
        "    optimizer=optimizer,\n",
        "    metrics=['accuracy'],\n",
        ")\n",
        "model.fit(x_train, y_train_class, batch_size=32, epochs=20)\n",
        "score = model.evaluate(x_test, y_test_class, verbose=0)\n",
        "\n",
        "print('Validation loss:', score[0])\n",
        "print('Validation accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w8eUV9RDPkL"
      },
      "source": [
        "\n",
        "---\n",
        "---\n",
        "\n",
        "## Task 2: Regression\n",
        "\n",
        "Now we face a regression task instead of a classification problem. Loss function, activations, and dataset will change in the following task. Thus, instead of having one vector with the probabilities of each class, in this regression problem, the output is a single scalar.\n",
        "\n",
        "For this second task, we chose the task of estimating house prices based on input images. To get the data run the following script, which clones Ahmed and Moustafa’s [repository](https://github.com/emanhamed/Houses-dataset) into colmap."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ibukle2ODMEp"
      },
      "source": [
        "!git clone https://github.com/emanhamed/Houses-dataset\n",
        "%cd /content/Houses-dataset/Houses\\ Dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ye6xldkVEV-t"
      },
      "source": [
        "This dataset contains four images of the house (kitchen, frontal, bedroom and bathroom), and attributes (number of bedrooms, number of bathrooms, zip code...). For our exercise, we only use the images of the house. We start with front door images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "71urud9ZHq9d"
      },
      "source": [
        "import cv2\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "house_section = 'frontal' # select between: kitchen, frontal, bedroom or bathroom\n",
        "print('We have selected {} images. You can switch to the kitchen, frontal, bedroom or bathroom images by changing house_section variable.'.format(house_section))\n",
        "images = []\n",
        "for i_im in range(1, 536):\n",
        "  image = cv2.imread(str(i_im)+'_'+house_section+'.jpg')\n",
        "  image = cv2.resize(image, (64, 64))\n",
        "  images.append(image)\n",
        "\n",
        "labels = []\n",
        "f = open('HousesInfo.txt', \"r\")\n",
        "for x in f:\n",
        "  label = (x).split(' ')[-1].split('\\n')[0]\n",
        "  labels.append(label)\n",
        "\n",
        "# Let's visualize some examples\n",
        "N=3\n",
        "start_val = 0 # pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    tmp = cv2.cvtColor(images[idx],cv2.COLOR_BGR2RGB)\n",
        "    axes[row,col].imshow(tmp, cmap='gray')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    target = int(labels[idx])\n",
        "    axes[row,col].set_title(str(target) + '$')\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scNfczb73nGS"
      },
      "source": [
        "Prepare the dataset for training the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FORNJ1QU2L-t"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "np.random.seed(1234)\n",
        "\n",
        "images = np.asarray(images).astype('float32')\n",
        "labels = np.asarray(labels).astype('float32')\n",
        "\n",
        "# Normalize the image\n",
        "max_price = labels.max()\n",
        "images /= 255.\n",
        "labels /= max_price\n",
        "\n",
        "random_idx = np.random.permutation(len(images))\n",
        "images = images[random_idx]\n",
        "labels = labels[random_idx]\n",
        "\n",
        "split_size_val = int(0.8*len(images))\n",
        "X_train, X_val = images[:split_size_val], images[split_size_val+1:]\n",
        "Y_train, Y_val = labels[:split_size_val], labels[split_size_val+1:]\n",
        "\n",
        "# Print shape of training and val images\n",
        "print('X_train image shape: {0}'.format(X_train.shape))\n",
        "print('X_val image shape: {0}'.format(X_val.shape))\n",
        "\n",
        "# Print shape of training and val labels\n",
        "print('Y_train labels shape: {0}'.format(Y_train.shape))\n",
        "print('Y_val labels shape: {0}'.format(Y_val.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGC6dXqdM2eU"
      },
      "source": [
        "### Problem Definition\n",
        "\n",
        "Similar to the previous task, you are asked to design a CNN architecture able to perform the estimation of house prices based on the `frontal` house image. Design a new model by changing parameters such as the number of convolutional layers, activation functions, strides, or pooling operators, among others."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "tf.random.set_seed(1234)"
      ],
      "metadata": {
        "id": "LqfWrQmMWjWc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YE8g42x3M2Ko"
      },
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "# TODO: Define here your architecture\n",
        "model = Sequential()\n",
        "# . . .\n",
        "model.summary()\n",
        "\n",
        "optimizer = Adam(learning_rate=1e-3)\n",
        "model.compile(loss=\"mean_absolute_percentage_error\", optimizer=optimizer, metrics=['mean_absolute_percentage_error'])\n",
        "\n",
        "# train the model\n",
        "print(\"[INFO] training model...\")\n",
        "model.fit(X_train, Y_train, validation_data=(X_val, Y_val), epochs=100, batch_size=32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JyXrXBFVQ-tL"
      },
      "source": [
        "The metric used in this problem to evaluate the performance is the same we used for training the model, the mean absolute percentage error. Mean absolute percentage error is defined as $\\frac{100}{n} \\sum_n \\frac{|\\hat{y} - y|}{|y|}$ where $y$ is the ground-truth, $\\hat{y}$ is the estimation of the model and `n` the number of elements in the set we are evaluating."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u2u2qQh90C3K"
      },
      "source": [
        "prices_predicted = model.predict(X_val).flatten()\n",
        "diff = ((prices_predicted - Y_val) / Y_val) * 100\n",
        "error_mean = np.mean(np.abs(diff))\n",
        "\n",
        "print(\"Predicting house prices - Estimation Error: {:.2f}%\".format(error_mean))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2uJs6-xG6lD"
      },
      "source": [
        "**Report**:\n",
        "\n",
        "\n",
        "*   Propose a CNN architecture that has an estimation error in the validation set below 75%.\n",
        "*   Present a figure showing the training and validation loss vs the number of training epochs for different architectural design choices. Discuss the gap between the training and validation loss depending on the proposed architecture.\n",
        "*   Report a table with results when using any of the other images from the house (kitchen, bedroom, and bathroom)."
      ]
    }
  ]
}