NUS SWS 2024/Deep Learning/SWS3009 Trial - Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SWS3009 Hands On - Neural Networks\n",
    "\n",
    "Although no submission is required, you should still attempt this hands on to give you some basic knowledge.\n",
    "\n",
    "\n",
    "## 1. Introduction\n",
    "\n",
    "The objectives of this lab are:\n",
    "\n",
    "    1. To familiarize you with how to create dense neural networks using Keras.\n",
    "    2. To familiarize you with how to encode input and output vectors for neural networks.\n",
    "    3. To give you some insight into how hyperparameters like learning rate and momentum affect training.\n",
    "    \n",
    "To save time we will train each experiment only for 50 epochs. This will lead to less than optimal results but is enough for you to make observations.\n",
    "\n",
    "**HINT: YOU CAN HIT SHIFT-ENTER TO RUN EACH CELL. NOTE THAT IF A CELL IS DEPENDENT ON A PREVIOUS CELL, YOU WILL NEED TO RUN THE PREVIOUS CELL(S) FIRST **\n",
    "\n",
    "\n",
    "## 2. The Irises Dataset\n",
    "\n",
    "We will now work again on the Irises Dataset, which we used in Lab 1, for classifying iris flowers into one of three possible types. As before we will consider four factors:\n",
    "\n",
    "    1. Sepal length in cm\n",
    "    2. Sepal width in cm\n",
    "    3. Petal length in cm\n",
    "    4. Petal width in cm\n",
    "\n",
    "In this dataset there are 150 sample points. The code below loads the dataset and prints the first 10 rows so we have an idea of what it looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 10 rows of data:\n",
      "[[5.1 3.5 1.4 0.2]\n",
      " [4.9 3.  1.4 0.2]\n",
      " [4.7 3.2 1.3 0.2]\n",
      " [4.6 3.1 1.5 0.2]\n",
      " [5.  3.6 1.4 0.2]\n",
      " [5.4 3.9 1.7 0.4]\n",
      " [4.6 3.4 1.4 0.3]\n",
      " [5.  3.4 1.5 0.2]\n",
      " [4.4 2.9 1.4 0.2]\n",
      " [4.9 3.1 1.5 0.1]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "iris = load_iris()\n",
    "\n",
    "print(\"First 10 rows of data:\")\n",
    "print(iris.data[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Scaling the Data\n",
    "\n",
    "We make use of the MinMaxScaler to scale the inputs to between 0 and 1.  The code below does this and prints the first 10 rows again, to show us the difference.\n",
    "\n",
    "In the next section we will investigate what happens if we use unscaled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 10 rows of SCALED data.\n",
      "[[0.22222222 0.625      0.06779661 0.04166667]\n",
      " [0.16666667 0.41666667 0.06779661 0.04166667]\n",
      " [0.11111111 0.5        0.05084746 0.04166667]\n",
      " [0.08333333 0.45833333 0.08474576 0.04166667]\n",
      " [0.19444444 0.66666667 0.06779661 0.04166667]\n",
      " [0.30555556 0.79166667 0.11864407 0.125     ]\n",
      " [0.08333333 0.58333333 0.06779661 0.08333333]\n",
      " [0.19444444 0.58333333 0.08474576 0.04166667]\n",
      " [0.02777778 0.375      0.06779661 0.04166667]\n",
      " [0.16666667 0.45833333 0.08474576 0.        ]]\n"
     ]
    }
   ],
   "source": [
    "scaler = MinMaxScaler()\n",
    "scaler.fit(iris.data)\n",
    "X = scaler.transform(iris.data)\n",
    "\n",
    "print(\"First 10 rows of SCALED data.\")\n",
    "print(X[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Encoding the Targets\n",
    "\n",
    "In Lab 1 we saw that the target values (type of iris flower) is a vector from 0 to 2. We can see the 150 labels below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "print(iris.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this to train the neural network, but we will use \"one-hot\" encoding, where we have a vector of _n_ integers consisting of 0's and 1's.  The table below shows how one-hot encoding works:\n",
    "\n",
    "|   Value    |    One-Hot Encoding    |\n",
    "|:----------:|:----------------------:|\n",
    "| 0 | \\[1 0 0\\] |\n",
    "| 1 | \\[0 1 0\\] |\n",
    "| 2 | \\[0 0 1\\] |\n",
    "\n",
    "Keras provides the to_categorical function to create one-hot vectors:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]\n",
      " [0. 0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "Y = to_categorical(y = iris.target, num_classes = 3)\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's split the data into training and testing data:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, \n",
    "                                                    random_state = 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Building our Neural Network\n",
    "\n",
    "Let's now begin building a simple neural network with a single hidden layer, using the Stochastic Gradient Descent (SGD) optimizer, ReLu transfer functions for the hidden layer and softmax for the output layer.\n",
    "\n",
    "The code to do this is shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "# Create the neural network\n",
    "nn = Sequential()\n",
    "nn.add(Dense(100, input_shape = (4, ), activation = 'relu'))\n",
    "nn.add(Dense(3, activation = 'softmax'))\n",
    "\n",
    "# Create our optimizer\n",
    "sgd = SGD(learning_rate = 0.1)\n",
    "\n",
    "# 'Compile' the network to associate it with a loss function,\n",
    "# an optimizer, and what metrics we want to track\n",
    "nn.compile(loss='categorical_crossentropy', optimizer=sgd, \n",
    "          metrics = 'accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Training the Neural Network\n",
    "\n",
    "As is usually the case, we can call the \"fit\" method to train the neural network for 50 epochs. We will shuffle the training data between epochs, and provide validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "4/4 [==============================] - 0s 16ms/step - loss: 1.0924 - accuracy: 0.3167 - val_loss: 1.0573 - val_accuracy: 0.1000\n",
      "Epoch 2/50\n",
      "4/4 [==============================] - 0s 2ms/step - loss: 1.0183 - accuracy: 0.4083 - val_loss: 1.0277 - val_accuracy: 0.3667\n",
      "Epoch 3/50\n",
      "4/4 [==============================] - 0s 2ms/step - loss: 0.9729 - accuracy: 0.6500 - val_loss: 0.9953 - val_accuracy: 0.5667\n",
      "Epoch 4/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.9344 - accuracy: 0.6917 - val_loss: 0.9708 - val_accuracy: 0.5667\n",
      "Epoch 5/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.9043 - accuracy: 0.6917 - val_loss: 0.9357 - val_accuracy: 0.5667\n",
      "Epoch 6/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.8706 - accuracy: 0.6917 - val_loss: 0.9059 - val_accuracy: 0.5667\n",
      "Epoch 7/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.8390 - accuracy: 0.6917 - val_loss: 0.8795 - val_accuracy: 0.5667\n",
      "Epoch 8/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.8098 - accuracy: 0.6917 - val_loss: 0.8527 - val_accuracy: 0.5667\n",
      "Epoch 9/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.7825 - accuracy: 0.6917 - val_loss: 0.8230 - val_accuracy: 0.5667\n",
      "Epoch 10/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.7553 - accuracy: 0.6917 - val_loss: 0.8028 - val_accuracy: 0.5667\n",
      "Epoch 11/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.7305 - accuracy: 0.6917 - val_loss: 0.7792 - val_accuracy: 0.5667\n",
      "Epoch 12/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.7061 - accuracy: 0.6917 - val_loss: 0.7557 - val_accuracy: 0.5667\n",
      "Epoch 13/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.6820 - accuracy: 0.6917 - val_loss: 0.7338 - val_accuracy: 0.5667\n",
      "Epoch 14/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.6598 - accuracy: 0.6917 - val_loss: 0.7075 - val_accuracy: 0.5667\n",
      "Epoch 15/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.6408 - accuracy: 0.6917 - val_loss: 0.6799 - val_accuracy: 0.5667\n",
      "Epoch 16/50\n",
      "4/4 [==============================] - 0s 7ms/step - loss: 0.6221 - accuracy: 0.7000 - val_loss: 0.6648 - val_accuracy: 0.5667\n",
      "Epoch 17/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.6047 - accuracy: 0.7000 - val_loss: 0.6497 - val_accuracy: 0.5667\n",
      "Epoch 18/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5851 - accuracy: 0.7083 - val_loss: 0.6323 - val_accuracy: 0.6000\n",
      "Epoch 19/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5689 - accuracy: 0.7167 - val_loss: 0.6133 - val_accuracy: 0.6000\n",
      "Epoch 20/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5582 - accuracy: 0.7083 - val_loss: 0.5864 - val_accuracy: 0.6000\n",
      "Epoch 21/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5421 - accuracy: 0.7583 - val_loss: 0.5798 - val_accuracy: 0.6000\n",
      "Epoch 22/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5280 - accuracy: 0.7917 - val_loss: 0.5750 - val_accuracy: 0.6000\n",
      "Epoch 23/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5158 - accuracy: 0.7500 - val_loss: 0.5515 - val_accuracy: 0.6000\n",
      "Epoch 24/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.5035 - accuracy: 0.7750 - val_loss: 0.5402 - val_accuracy: 0.6333\n",
      "Epoch 25/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4930 - accuracy: 0.8167 - val_loss: 0.5368 - val_accuracy: 0.6000\n",
      "Epoch 26/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4843 - accuracy: 0.7750 - val_loss: 0.5152 - val_accuracy: 0.6667\n",
      "Epoch 27/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4731 - accuracy: 0.8083 - val_loss: 0.5076 - val_accuracy: 0.6667\n",
      "Epoch 28/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4650 - accuracy: 0.7917 - val_loss: 0.4903 - val_accuracy: 0.7667\n",
      "Epoch 29/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4629 - accuracy: 0.8250 - val_loss: 0.4790 - val_accuracy: 0.7667\n",
      "Epoch 30/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4462 - accuracy: 0.8833 - val_loss: 0.4839 - val_accuracy: 0.7000\n",
      "Epoch 31/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4398 - accuracy: 0.8500 - val_loss: 0.4705 - val_accuracy: 0.7667\n",
      "Epoch 32/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4314 - accuracy: 0.8583 - val_loss: 0.4594 - val_accuracy: 0.7667\n",
      "Epoch 33/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4237 - accuracy: 0.8750 - val_loss: 0.4530 - val_accuracy: 0.7667\n",
      "Epoch 34/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4185 - accuracy: 0.8583 - val_loss: 0.4374 - val_accuracy: 0.8000\n",
      "Epoch 35/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4096 - accuracy: 0.8917 - val_loss: 0.4356 - val_accuracy: 0.8000\n",
      "Epoch 36/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.4044 - accuracy: 0.9083 - val_loss: 0.4330 - val_accuracy: 0.8000\n",
      "Epoch 37/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3977 - accuracy: 0.8917 - val_loss: 0.4312 - val_accuracy: 0.7667\n",
      "Epoch 38/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3936 - accuracy: 0.8833 - val_loss: 0.4260 - val_accuracy: 0.7667\n",
      "Epoch 39/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3853 - accuracy: 0.9083 - val_loss: 0.4204 - val_accuracy: 0.7667\n",
      "Epoch 40/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3782 - accuracy: 0.8833 - val_loss: 0.4037 - val_accuracy: 0.9000\n",
      "Epoch 41/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3777 - accuracy: 0.9167 - val_loss: 0.4087 - val_accuracy: 0.8000\n",
      "Epoch 42/50\n",
      "4/4 [==============================] - 0s 7ms/step - loss: 0.3702 - accuracy: 0.9167 - val_loss: 0.3935 - val_accuracy: 0.9000\n",
      "Epoch 43/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3656 - accuracy: 0.9000 - val_loss: 0.3744 - val_accuracy: 0.9667\n",
      "Epoch 44/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3582 - accuracy: 0.9583 - val_loss: 0.3775 - val_accuracy: 0.9333\n",
      "Epoch 45/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3525 - accuracy: 0.9500 - val_loss: 0.3832 - val_accuracy: 0.9000\n",
      "Epoch 46/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3508 - accuracy: 0.9333 - val_loss: 0.3739 - val_accuracy: 0.9000\n",
      "Epoch 47/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3421 - accuracy: 0.9417 - val_loss: 0.3683 - val_accuracy: 0.9333\n",
      "Epoch 48/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3425 - accuracy: 0.8917 - val_loss: 0.3328 - val_accuracy: 0.9667\n",
      "Epoch 49/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3350 - accuracy: 0.9667 - val_loss: 0.3496 - val_accuracy: 0.9333\n",
      "Epoch 50/50\n",
      "4/4 [==============================] - 0s 3ms/step - loss: 0.3288 - accuracy: 0.9583 - val_loss: 0.3482 - val_accuracy: 0.9333\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x2a22b73d0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn.fit(X_train, Y_train, shuffle = True, epochs = 50, \n",
    "      validation_data = (X_test, Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Question 1\n",
    "\n",
    "Run the code above. Do you see evidence of underfitting? Overfitting? Justify your answers. ***(4 MARKS)***\n",
    "\n",
    "**Answer: Type your answer here. Do not hit return to continue to the next line, just let the text wrap around **\n",
    "\n",
    "_(For TA) Marks awarded: ____ / 4_\n",
    "\n",
    "---\n",
    "\n",
    "#### Question 2a\n",
    "\n",
    "Consult the documentation for the SGD optimizer [here](https://keras.io/api/optimizers/sgd/). What does the lr parameter do? ***(1 MARK)***\n",
    "\n",
    "**Answer: Type your answer here. Do not hit return to continue to the next line, just let the text wrap around **\n",
    "\n",
    "#### Question 2b\n",
    "\n",
    "The documentation states that the momentum parameter \"accelerates gradient descent in the relevant direction and dampens oscillations\". Using Google or other means, illustrate what this means. ***(2 MARKS)***\n",
    "\n",
    "**Answer: Type your answer here. Do not hit return to continue to the next line, just let the text wrap around **\n",
    "\n",
    "_(For TA) Marks awarded: ____ / 3_\n",
    "\n",
    "----\n",
    "\n",
    "#### Question 3a\n",
    "\n",
    "We will now play with the lr parameter. Adjust the lr parameter to the following values and record the final training and validation accuracies in the respective columns. Also observe the sequence of accuracies over the training period, and place your observation in the \"remarks\" column, e.g. \"Progresses steadily\", \"some oscillation\" etc. ***(3 MARKS)***\n",
    "\n",
    "**Answer: Fill the table below **\n",
    "\n",
    "|  lr    | Training Acc. | Validation Acc. |      Remarks      |\n",
    "|:------:|---------------|-----------------|-------------------|\n",
    "|0.01    |               |                 |                   |\n",
    "|0.1     |               |                 |                   |\n",
    "|1.0     |               |                 |                   |\n",
    "|10.0    |               |                 |                   |\n",
    "|100     |               |                 |                   |\n",
    "|1000    |               |                 |                   |\n",
    "|10000   |               |                 |                   |\n",
    "| 100000 |               |                 |                   |\n",
    "\n",
    "\n",
    "#### Question 3b\n",
    "\n",
    "Based on your observations above, comment on the effect of small and very large learning rates on the learning. ***(2 MARKS)***\n",
    "\n",
    "**Answer: Type your answer here. Do not hit return to continue to the next line, just let the text wrap around **\n",
    "\n",
    "_(For TA) Marks awarded: ____ / 5_\n",
    "\n",
    "### 2.5 Using Momentum\n",
    "\n",
    "We will now experiment with the momentum term. To do this:\n",
    "\n",
    "    1. Change the learning rate to 0.1.\n",
    "    2. Set the momentum to 0.1. Note: Do not use the Nesterov parameter - Leave it as False.\n",
    "    \n",
    "Run your neural network.\n",
    "\n",
    "---\n",
    "\n",
    "#### Question 4a\n",
    "\n",
    "Keeping the learning rate at 0.1, complete the table below using the momentum values shown. Again record any observations in the \"Remarks\" column. ***(3 MARKS)***\n",
    "\n",
    "**Answer: Fill the table below**\n",
    "\n",
    "| momentum | Training Acc. | Validation Acc. |      Remarks      |\n",
    "|:--------:|---------------|-----------------|-------------------|\n",
    "|0.001     |               |                 |                   |\n",
    "|0.01      |               |                 |                   |\n",
    "|0.1       |               |                 |                   |\n",
    "|1.0       |               |                 |                   |\n",
    "\n",
    "#### Question 4b\n",
    "\n",
    "Based on your observations above, does the momentum term help in learning? ***(2 MARKS)***\n",
    "\n",
    "**Answer: Type your answer here. Do not hit return to continue to the next line, just let the text wrap around **\n",
    "\n",
    "_(For TA) Marks awarded: ____ / 5_\n",
    "\n",
    "---\n",
    "\n",
    "### 2.6 Using Raw Unscaled Data\n",
    "\n",
    "We begin by using unscaled X and Y data. The code below will create 120 training samples and 30 testing samples (20% of the total of 150 samples):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_unscaled = iris.data\n",
    "Y_raw = iris.target\n",
    "X_utrain, X_utest, Y_utrain, Y_utest = train_test_split(X_unscaled, Y,\n",
    "                                                        test_size = 0.2,\n",
    "                                                        random_state = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "#### Question 5\n",
    "\n",
    "Create a new neural network called \"nn2\" below using a single hidden layer of 100 neurons. Train using the data in X_utrain, X_utest and validate with Y_utrain and Y_utest. Again use the SGD optimizer with a learning rate of 0.1 and no momentum, and train for 50 epochs. ***(3 marks)***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nEnter your code for Question 5 below. To TA: 3 marks for code.\\n'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Enter your code for Question 5 below. To TA: 3 marks for code.\n",
    "\"\"\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(Question 5 continues)**\n",
    "\n",
    "Observe the training and validation error. Does not scaling the input affect the training? Why do you think this is so? What is the advantage of scaling? ***(5 MARKS)***\n",
    "\n",
    "**Answer: Type your answer here. Do not hit return to continue to the next line, just let the text wrap around **\n",
    "\n",
    "_(For TA) Marks awarded: ____ / 8_\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Conclusion\n",
    "\n",
    "In this lab we saw how to create a simple Dense neural network to complete the relatively simple task of learning how to classify irises according to their sepal and petal characteristics. \n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "***FOR TA ONLY***\n",
    "\n",
    "| Question |  Marks  |\n",
    "|:--------:|:-------:|\n",
    "|1         |     /4  |\n",
    "|2         |     /3  |\n",
    "|3         |     /5  |\n",
    "|4         |     /5  |\n",
    "|5         |     /8  |\n",
    "|Total:    |     /25 |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}