01_materials/labs/lab_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Neural Networks with Keras\n",
    "\n",
    "Welcome to the first practical session of the course! In this session, we will learn how to train neural networks with Keras. We will start with a simple example of a feedforward neural network for classification and then we will study the impact of the initialization of the weights on the convergence of the training algorithm.\n",
    "\n",
    "Keras is a high-level neural network API, built on top of TensorFlow 2.0. It provides a user-friendly interface to build, train and deploy deep learning models. Keras is designed to be modular, fast and easy to use.\n",
    "\n",
    "Throughout this course, we will focus on using Keras and TensorFlow for building and training neural networks. However, there are other popular deep learning frameworks such as PyTorch, MXNet, CNTK, etc. that you can also use to build and train neural networks.\n",
    "\n",
    "In order to use our code on Google Colab, we will need to ensure that any required packages are installed. We will use the following packages in this session:\n",
    "\n",
    "- `tensorflow`: an open-source library for numerical computation and large-scale machine learning.\n",
    "- `matplotlib`: a plotting library for the Python programming language and its numerical mathematics extension NumPy.\n",
    "- `numpy`: a library for scientific computing in Python.\n",
    "- `scikit-learn`: a machine learning library for the Python programming language.\n",
    "- `pandas`: a library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n",
    "\n",
    "The following cell will check if the packages are installed, and if not, it will install them. Students familiar with how pip works might note that it already checks this before installing! The reason for this code (which will also appear in subsequent notebooks) is to speed up execution if you re-run the entire notebook - it will skip the installation step if the packages are already installed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "With the packages installed, we can now get started on the practical session!\n",
    "\n",
    "Today, we will be working with the famous MNIST dataset. MNIST (Modified National Institute of Standards and Technology) is a database of low resolution images of handwritten digits. The history here is interesting - the dataset was originally created in the 1980s, when researchers from the aforementioned institute collected samples from American Census Bureau employees and high school students. The dataset was then modified in the 1990s (hence the M in MNIST), and has since become a popular benchmark for machine learning algorithms. \n",
    "\n",
    "The dataset contains images, each of which is a 28x28 grayscale image of a handwritten digit. The goal is to classify each image into one of the 10 possible classes (0-9).\n",
    "\n",
    "![MNIST](https://upload.wikimedia.org/wikipedia/commons/2/27/MnistExamples.png)\n",
    "\n",
    "The Scikit-Learn library provides a convenient function to download and load the MNIST dataset. The following cell will download the dataset. Then we will take a look at the shape of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.datasets import load_digits\n",
    "\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.images.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "This means that we have 1797 images, each of which is a 8x8 image. For basic image processing, we will need to flatten the images into a 1D array. In this case, Scikit-Learn has already provided the data in this format too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "For each image, we also have the corresponding label (or target, or class) in `digits.target`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We can take a look at some random images from the dataset. The following cell will select 9 random images and plot them in a 3x3 grid (meaning that you can rerun the cell to see different images)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Selecting 9 random indices\n",
    "random_indices = np.random.choice(len(digits.images), 9, replace=False)\n",
    "\n",
    "# Creating a 3x3 grid plot\n",
    "fig, axes = plt.subplots(3, 3, figsize=(6, 6))\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(digits.images[random_indices[i]], cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "    ax.set_title(f\"Label: {digits.target[random_indices[i]]}\")\n",
    "\n",
    "    # Removing axis labels\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "As you can see, these images are very low resolution. This is because they were originally scanned from paper forms, and then scaled down to 8x8 pixels. This is a common problem in machine learning - the quality of the data is often a limiting factor in the performance of the model. In this case, the low resolution of the images makes it difficult to distinguish between some digits, even for humans. For example, the following images are all labelled as 9, but they look very different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Selecting 9 random indices of images labelled as 9\n",
    "random_indices = np.random.choice(np.where(digits.target == 9)[0], 9, replace=False)\n",
    "\n",
    "# Creating a 3x3 grid plot\n",
    "fig, axes = plt.subplots(3, 3, figsize=(6, 6))\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(digits.images[random_indices[i]], cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "    ax.set_title(f\"Label: {digits.target[random_indices[i]]}\")\n",
    "\n",
    "    # Removing axis labels\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "While we are plotting the samples as images, remember that our model is only going to see a 1D array of numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train / Test Split\n",
    "\n",
    "In order to understand how well our model performs on _new_ data, we need to split our dataset into a training set and a test set. The training set will be used to train the model, and the test set will be used to evaluate the performance of the model.\n",
    "\n",
    "Let's keep some held-out data to be able to measure the generalization performance of our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    digits.data, \n",
    "    digits.target,\n",
    "    test_size=0.2, # 20% of the data is used for testing\n",
    "    random_state=42 # Providing a value here means getting the same \"random\" split every time\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Let's confirm that the data has been split correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(f'X_train shape: {X_train.shape}')\n",
    "print(f'y_train shape: {y_train.shape}')\n",
    "print(f'X_test shape: {X_test.shape}')\n",
    "print(f'y_test shape: {y_test.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "This is what we expected to see. It's always good to check as you go, to make sure that you haven't made a mistake somewhere - this is something that working in a notebook like this makes it easy to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing of the Target Data\n",
    "\n",
    "The labels that we have are integers between 0 and 9. However, we want to train a neural network to classify the images into one of 10 classes. It can be a little counter-intuitive because we are dealing with numbers, but our classes are not ordinal.\n",
    "\n",
    "What do we mean by that? Let's imagine we were trying to predict the height of a building (separated into classes) from images. If a given building was actually 10m tall, and our model predicted 9m, we would consider that to be a better prediction than if it predicted 1m. This is because the classes are ordinal - there is meaning in the difference between the classes.\n",
    "\n",
    "In our case, even though we are dealing with numbers, the classes are not ordinal. If a given image is actually a 9, and our model predicts 8, we would consider that to be just as bad as if it predicted 1. This is because the classes are not ordered, and the difference between the classes is not meaningful.\n",
    "\n",
    "Because of this, we need to convert our labels from an integer value into a one-hot encoded vector. This means that each label will be represented as a vector of length 10, with a 1 in the position corresponding to the class, and 0s everywhere else. For example, the label 9 would be represented as `[0, 0, 0, 0, 0, 0, 0, 0, 0, 1]`. This is a common way of representing categorical data in machine learning. By doing this, we ensure that our model is taught the correct relationship between the classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "print(f'Before one-hot encoding: {y_train[0]}')\n",
    "y_train = to_categorical(y_train, num_classes=10)\n",
    "y_test = to_categorical(y_test, num_classes=10)\n",
    "print(f'After one-hot encoding: {y_train[0]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feed Forward Neural Networks with Keras\n",
    "\n",
    "Now that we have prepared our data, it's time to build a simple neural network! In this section, we will use the Keras API to build a simple feed forward neural network. We will then train the model on the MNIST dataset, and evaluate its performance on the test set.\n",
    "\n",
    "In most modern deep learning frameworks, the process of building a model can be broken down into a few steps:\n",
    "\n",
    "- Define the model architecture: this is where we define the layers of the model, and how they are connected to each other.\n",
    "- Compile the model: this is where we define the loss function, the optimizer, and the metrics that we want to use to evaluate the model.\n",
    "- Train the model: this is where we train the model on the training data.\n",
    "\n",
    "Let's start with defining the model architecture. There are two ways to do this in Keras - the Sequential API and the Functional API. The Sequential API is the simplest way to build a model, and is suitable for most use cases. The Functional API is more flexible, and allows you to build more complex models. We will start with the Sequential API, and then we will look at the Functional API later in the course.\n",
    "\n",
    "Our simple neural network will be \"fully-connected\". This means that each neuron in a given layer is connected to every neuron in the next layer. This is also known as a \"dense\" layer. We will use the `Dense` class from Keras to define our layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "# Input layer\n",
    "model.add(Dense(64, activation='relu', input_shape=(64,))) # 64 neurons, ReLU activation, input shape of 64\n",
    "\n",
    "# Hidden layer\n",
    "model.add(Dense(64, activation='relu')) # 64 neurons, ReLU activation\n",
    "\n",
    "# Output layer\n",
    "model.add(Dense(10, activation='softmax')) # 10 neurons, softmax activation\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Congratulations! You have just built your first neural network with Keras. As we can confirm from the `model.summary()` output, our model has 3 layers. The first layer has 64 neurons, the second layer has 64 neurons, and the output layer has 10 neurons. The output layer uses the softmax activation function, which is commonly used for multi-class classification problems. The other layers use the ReLU activation function, which is commonly used for hidden layers in neural networks.\n",
    "\n",
    "Next, we need to compile the model. This is where we define the loss function, the optimizer, and the metrics that we want to use to evaluate the model. We will use the `compile` method of the model to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    loss='categorical_crossentropy', # Loss function\n",
    "    optimizer='sgd', # Optimizer\n",
    "    metrics=['accuracy'] # Metrics to evaluate the model\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Because we are predicting which class a sample belongs to, we will use the `categorical_crossentropy` function. This loss function is commonly used for multi-class classification problems. \n",
    "\n",
    "For our optimizer, we are using the standard stochastic gradient descent (SGD) algorithm. This is a simple optimizer that works well for many problems. We will look at more advanced optimizers later in the course.\n",
    "\n",
    "Finally, we are using the `accuracy` metric to evaluate the model. This is a common metric for classification problems, and it is simply the fraction of samples that are correctly classified. This is an easier metric for us to understand, but it's not quite as useful for actually training the model (for example, it doesn't tell us how \"confident\" the model is in its predictions).\n",
    "\n",
    "Now that we have (a) defined the model architecture and (b) compiled the model, we are ready to train the model. We will use the `fit` method of the model to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.fit(\n",
    "    X_train, # Training data\n",
    "    y_train, # Training labels\n",
    "    epochs=5, # Number of epochs\n",
    "    batch_size=32, # Number of samples per batch\n",
    "    validation_split=0.2 # Use 20% of the data for validation\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We have now trained our model! We can see that the model has been trained for 5 epochs, and the loss and accuracy have been printed for each epoch. We can also see that the model has been evaluated on the validation data at the end of each epoch. This is useful for us to see how the model is performing on data that it hasn't seen during training.\n",
    "\n",
    "Once the model is trained, it's time to evaluate the model on the test set. We can use the `evaluate` method of the model to do this. If you were building a model for a real-world application, this is the very last thing you would do, and the result here would be the figure you'd report in your paper or presentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "loss, accuracy = model.evaluate(X_test, y_test)\n",
    "\n",
    "print(f'Loss:     {loss:.2f}')\n",
    "print(f'Accuracy: {accuracy*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Hopefully you have achieved an accuracy of around 95%. This is pretty good, but we can do better! In the next section, we will look at how we can improve the performance of our model by using a more advanced optimizer. But before we get there, let's do one other thing - let's look at the predictions that our model is making on the test set. When you are building a model, it's often useful to have a look at some of the examples your model is getting wrong. Sometimes this can reveal problems with the data, or it can give you ideas for how to improve your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get the predictions for the test data\n",
    "predictions = model.predict(X_test)\n",
    "\n",
    "# Get the index of the largest probability (i.e. the predicted class)\n",
    "predicted_classes = np.argmax(predictions, axis=1)\n",
    "true_classes = np.argmax(y_test, axis=1)\n",
    "misclassified_indices = np.where(predicted_classes != true_classes)[0]\n",
    "\n",
    "# Get the misclassified samples themselves\n",
    "misclassified_samples = X_test[misclassified_indices]\n",
    "misclassified_labels = np.argmax(y_test[misclassified_indices], axis=1)\n",
    "\n",
    "# Pick 9 random misclassified samples\n",
    "random_indices = np.random.choice(len(misclassified_indices), 9, replace=False)\n",
    "\n",
    "fig, axes = plt.subplots(3, 3, figsize=(6, 6))\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(misclassified_samples[random_indices[i]].reshape(8, 8), cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "    ax.set_title(f\"Pred: {predicted_classes[misclassified_indices[random_indices[i]]]}, Real: {misclassified_labels[random_indices[i]]}\")\n",
    "\n",
    "    # Removing axis labels\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "What do you think? Would you have made the same mistakes as the model? Determining whether the mistakes are \"understandable\" is a rough way of seeing if you could improve the model further, or if this is the best you can do with the data you have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Exercises: Impact of the Optimizer\n",
    "\n",
    "In this section, you will play around with the optimizer and see how it affects the performance of the model. We will start with the standard SGD optimizer, and then we will look at more advanced optimizers.\n",
    "\n",
    "1. Try decreasing the learning rate of the SGD optimizer by a factor of 10, or 100. What do you observe?\n",
    "2. Try increasing the learning rate of the SGD optimizer. What happens?\n",
    "3. The SGD optimizer has a momentum parameter. In a nutshell, this parameter controls how much the gradient from the previous step affects the current step. Try enabling momentum in the SGD optimizer with a value of 0.9. What happens?\n",
    "  \n",
    "**Notes**: \n",
    "\n",
    "The keras API documentation is available at:\n",
    "\n",
    "https://www.tensorflow.org/api_docs/python/tf/keras\n",
    "\n",
    "It is also possible to learn more about the parameters of a class by using the question mark: type and evaluate:\n",
    "\n",
    "```python\n",
    "optimizers.SGD?\n",
    "```\n",
    "\n",
    "in a jupyter notebook cell.\n",
    "\n",
    "It is also possible to type the beginning of a function call / constructor and type \"shift-tab\" after the opening paren:\n",
    "\n",
    "```python\n",
    "optimizers.SGD(<shift-tab>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here - copy the relevant parts from the previous section and add more cells as needed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's try a more advanced optimizer. Adam is likely the most popular optimizer for deep learning. It is an adaptive learning rate optimizer, which means that it automatically adjusts the learning rate based on how the training is going. This can be very useful, as it means that we don't need to manually tune the learning rate. Let's see how it performs on our model.\n",
    "\n",
    "\n",
    "1. Replace the SGD optimizer by the Adam optimizer from keras and run it\n",
    "  with the default parameters.\n",
    "\n",
    "2. Add another hidden layer with ReLU activation and 64 neurons. Does it improve the model performance?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises: Forward Pass and Generalization\n",
    "\n",
    "Let's look in more detail at how the model makes predictions on the test set. We will walk through each step of making predictions, examining exactly what's going on.\n",
    "\n",
    "To start, we will apply our model to the test set, and look at what we get as output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_tf = model(X_test)\n",
    "predictions_tf[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(predictions_tf), predictions_tf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The raw output of the model is a tensor of shape `(360, 10)`. This means that we have 360 samples, and for each sample we have 10 values. Each of these values represents the probability that the sample belongs to a given class. This means that we have 10 probabilities for each sample, and the sum of these probabilities is 1. We can confirm this by summing the probabilities for each sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "tf.reduce_sum(predictions_tf, axis=1)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "...okay, there might be a small rounding error here and there. This is to do with how floating point numbers are represented in computers, and it's not something we need to worry about for now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also extract the label with the highest probability using the tensorflow API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted_labels_tf = tf.argmax(predictions_tf, axis=1)\n",
    "predicted_labels_tf[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "One helpful aspect of this approach is that we don't just get the prediction, but also a sense of how confident the model is in its prediction. To see this in practice, let's take a look at some of the predictions the model is highly confident about (i.e. a lot of the probability mass is on one class):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get the values corresponding to the predicted labels for each sample\n",
    "predicted_values_tf = tf.reduce_max(predictions_tf, axis=1)\n",
    "\n",
    "# Get the indices of the samples with the highest predicted values\n",
    "most_confident_indices_tf = tf.argsort(predicted_values_tf, direction='DESCENDING').numpy()[:9]\n",
    "\n",
    "# Get the 9 most confident samples\n",
    "most_confident_samples_tf = X_test[most_confident_indices_tf]\n",
    "\n",
    "# Get the true labels for the 9 most confident samples\n",
    "most_confident_labels_tf = np.argmax(y_test[most_confident_indices_tf], axis=1)\n",
    "\n",
    "# Plot the 9 most confident samples\n",
    "fig, axes = plt.subplots(3, 3, figsize=(6, 6))\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(most_confident_samples_tf[i].reshape(8, 8), cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "    ax.set_title(f\"{most_confident_labels_tf[i]}\")\n",
    "\n",
    "    # Removing axis labels\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Impact of Initialization\n",
    "\n",
    "Let's study the impact of a bad initialization when training\n",
    "a deep feed forward network.\n",
    "\n",
    "By default, Keras dense layers use the \"Glorot Uniform\" initialization\n",
    "strategy to initialize the weight matrices:\n",
    "\n",
    "- each weight coefficient is randomly sampled from [-scale, scale]\n",
    "- scale is proportional to $\\frac{1}{\\sqrt{n_{in} + n_{out}}}$\n",
    "\n",
    "This strategy is known to work well to initialize deep neural networks\n",
    "with \"tanh\" or \"relu\" activation functions and then trained with\n",
    "standard SGD.\n",
    "\n",
    "To assess the impact of initialization let us plug an alternative init\n",
    "scheme into a 2 hidden layers networks with \"tanh\" activations.\n",
    "For the sake of the example let's use normal distributed weights\n",
    "with a manually adjustable scale (standard deviation) and see the\n",
    "impact the scale value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import initializers\n",
    "from tensorflow.keras import optimizers\n",
    "\n",
    "input_dim = 64\n",
    "hidden_dim = 64\n",
    "output_dim = 10\n",
    "\n",
    "normal_init = initializers.TruncatedNormal(stddev=0.01, seed=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(hidden_dim, input_dim=input_dim, activation=\"tanh\",\n",
    "                kernel_initializer=normal_init))\n",
    "model.add(Dense(hidden_dim, activation=\"tanh\",\n",
    "                kernel_initializer=normal_init))\n",
    "model.add(Dense(output_dim, activation=\"softmax\",\n",
    "                kernel_initializer=normal_init))\n",
    "\n",
    "model.compile(optimizer=optimizers.SGD(learning_rate=0.1),\n",
    "              loss='categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the parameters of the first layer after initialization but before any training has happened:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers[0].weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = model.layers[0].weights[0].numpy()\n",
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = model.layers[0].weights[1].numpy()\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(X_train, y_train, epochs=15, batch_size=32)\n",
    "\n",
    "plt.figure(figsize=(12, 4))\n",
    "plt.plot(history.history['loss'], label=\"Truncated Normal init\")\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model has been fit, the weights have been updated and notably the biases are no longer 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers[0].weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions:\n",
    "\n",
    "- Try the following initialization schemes and see whether\n",
    "  the SGD algorithm can successfully train the network or\n",
    "  not:\n",
    "  \n",
    "  - a very small e.g. `stddev=1e-3`\n",
    "  - a larger scale e.g. `stddev=1` or `10`\n",
    "  - initialize all weights to 0 (constant initialization)\n",
    "  \n",
    "- What do you observe? Can you find an explanation for those\n",
    "  outcomes?\n",
    "\n",
    "- Are more advanced solvers such as SGD with momentum or Adam able\n",
    "  to deal better with such bad initializations?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 4
}