EXC_Validation_and_Test_Sets.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "private_outputs": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jmbanda/CSC4850-Machine-Learning/blob/main/EXC_Validation_and_Test_Sets.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4f3CKqFUqL2-"
      },
      "source": [
        "# Validation Sets and Test Sets\n",
        "\n",
        "Let's do some playing around with Tensorflow and Keras and complete the following excercise. \n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gV82DJO3kWpk"
      },
      "source": [
        "## The dataset\n",
        "\n",
        "As in the previous exercise, this exercise uses the [California Housing dataset](https://developers.google.com/machine-learning/crash-course/california-housing-data-description) to predict the `median_house_value` at the city block level.  Like many \"famous\" datasets, the California Housing Dataset actually consists of two separate datasets, each living in separate .csv files:\n",
        "\n",
        "* The training set is in `california_housing_train.csv`.\n",
        "* The test set is in `california_housing_test.csv`.\n",
        "\n",
        "You'll create the validation set by dividing the downloaded training set into two parts:\n",
        "\n",
        "* a smaller training set  \n",
        "* a validation set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8gm6BpqRRuh"
      },
      "source": [
        "## Import relevant modules\n",
        "\n",
        "This first code cell imports the necessary modules and sets a few display options."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9D8GgUovHbG0"
      },
      "source": [
        "#@title Import modules\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = \"{:.1f}\".format"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xjvrrClQeAJu"
      },
      "source": [
        "## Load the datasets from the internet\n",
        "\n",
        "The following code cell loads the separate .csv files and creates the following two pandas DataFrames:\n",
        "\n",
        "* `train_df`, which contains the training set.\n",
        "* `test_df`, which contains the test set.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zUnTc_wfd_o3"
      },
      "source": [
        "train_df = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\")\n",
        "test_df = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_test.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_KBdj2M_yjM"
      },
      "source": [
        "## Scale the label values\n",
        "\n",
        "The following code cell scales the `median_house_value`. \n",
        "See the previous Colab exercise for details."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3hc7QQhaAFXD"
      },
      "source": [
        "scale_factor = 1000.0\n",
        "\n",
        "# Scale the training set's label.\n",
        "train_df[\"median_house_value\"] /= scale_factor \n",
        "\n",
        "# Scale the test set's label\n",
        "test_df[\"median_house_value\"] /= scale_factor"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhessIIV8VPc"
      },
      "source": [
        "## Load the functions that build and train a model\n",
        "\n",
        "The following code cell defines two functions:\n",
        "\n",
        "  * `build_model`, which defines the model's topography.\n",
        "  * `train_model`, which will ultimately train the model, outputting not only the loss value for the training set but also the loss value for the validation set. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bvonhK857msj"
      },
      "source": [
        "#@title Define the functions that build and train a model\n",
        "def build_model(my_learning_rate):\n",
        "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
        "  # Most simple tf.keras models are sequential.\n",
        "  model = tf.keras.models.Sequential()\n",
        "\n",
        "  # Add one linear layer to the model to yield a simple linear regressor.\n",
        "  model.add(tf.keras.layers.Dense(units=1, input_shape=(1,)))\n",
        "\n",
        "  # Compile the model topography into code that TensorFlow can efficiently\n",
        "  # execute. Configure training to minimize the model's mean squared error. \n",
        "  model.compile(optimizer=tf.keras.optimizers.experimental.RMSprop(learning_rate=my_learning_rate),\n",
        "                loss=\"mean_squared_error\",\n",
        "                metrics=[tf.keras.metrics.RootMeanSquaredError()])\n",
        "\n",
        "  return model               \n",
        "\n",
        "\n",
        "def train_model(model, df, feature, label, my_epochs, \n",
        "                my_batch_size=None, my_validation_split=0.1):\n",
        "  \"\"\"Feed a dataset into the model in order to train it.\"\"\"\n",
        "\n",
        "  history = model.fit(x=df[feature],\n",
        "                      y=df[label],\n",
        "                      batch_size=my_batch_size,\n",
        "                      epochs=my_epochs,\n",
        "                      validation_split=my_validation_split)\n",
        "\n",
        "  # Gather the model's trained weight and bias.\n",
        "  trained_weight = model.get_weights()[0]\n",
        "  trained_bias = model.get_weights()[1]\n",
        "\n",
        "  # The list of epochs is stored separately from the \n",
        "  # rest of history.\n",
        "  epochs = history.epoch\n",
        "  \n",
        "  # Isolate the root mean squared error for each epoch.\n",
        "  hist = pd.DataFrame(history.history)\n",
        "  rmse = hist[\"root_mean_squared_error\"]\n",
        "\n",
        "  return epochs, rmse, history.history   \n",
        "\n",
        "print(\"Defined the build_model and train_model functions.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8gRu4Ri0D8tH"
      },
      "source": [
        "## Define plotting functions\n",
        "\n",
        "The `plot_the_loss_curve` function plots loss vs. epochs for both the training set and the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QA7hsqPZDvVM"
      },
      "source": [
        "#@title Define the plotting function\n",
        "\n",
        "def plot_the_loss_curve(epochs, mae_training, mae_validation):\n",
        "  \"\"\"Plot a curve of loss vs. epoch.\"\"\"\n",
        "\n",
        "  plt.figure()\n",
        "  plt.xlabel(\"Epoch\")\n",
        "  plt.ylabel(\"Root Mean Squared Error\")\n",
        "\n",
        "  plt.plot(epochs[1:], mae_training[1:], label=\"Training Loss\")\n",
        "  plt.plot(epochs[1:], mae_validation[1:], label=\"Validation Loss\")\n",
        "  plt.legend()\n",
        "  \n",
        "  # We're not going to plot the first epoch, since the loss on the first epoch\n",
        "  # is often substantially greater than the loss for other epochs.\n",
        "  merged_mae_lists = mae_training[1:] + mae_validation[1:]\n",
        "  highest_loss = max(merged_mae_lists)\n",
        "  lowest_loss = min(merged_mae_lists)\n",
        "  delta = highest_loss - lowest_loss\n",
        "  print(delta)\n",
        "\n",
        "  top_of_y_axis = highest_loss + (delta * 0.05)\n",
        "  bottom_of_y_axis = lowest_loss - (delta * 0.05)\n",
        "   \n",
        "  plt.ylim([bottom_of_y_axis, top_of_y_axis])\n",
        "  plt.show()  \n",
        "\n",
        "print(\"Defined the plot_the_loss_curve function.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jipBqEQXlsN8"
      },
      "source": [
        "## Task 1: Experiment with the validation split\n",
        "\n",
        "In the following code cell, you'll see a variable named `validation_split`, which we've initialized at 0.2.  The `validation_split` variable specifies the proportion of the original training set that will serve as the validation set. The original training set contains 17,000 examples. Therefore, a `validation_split` of 0.2 means that:\n",
        "\n",
        "* 17,000 * 0.2 ~= 3,400 examples will become the validation set.\n",
        "* 17,000 * 0.8 ~= 13,600 examples will become the new training set.\n",
        "\n",
        "The following code builds a model, trains it on the training set, and evaluates the built model on both:\n",
        "\n",
        "* The training set.\n",
        "* And the validation set.\n",
        "\n",
        "If the data in the training set is similar to the data in the validation set, then the two loss curves and the final loss values should be almost identical. However, the loss curves and final loss values are **not** almost identical. Hmm, that's odd.  \n",
        "\n",
        "Experiment with two or three different values of `validation_split`.  Do different values of `validation_split` fix the problem? \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "knP23Taoa00a"
      },
      "source": [
        "# The following variables are the hyperparameters.\n",
        "learning_rate = 0.08\n",
        "epochs = 30\n",
        "batch_size = 100\n",
        "\n",
        "# Split the original training set into a reduced training set and a\n",
        "# validation set. \n",
        "validation_split = 0.2\n",
        "\n",
        "# Identify the feature and the label.\n",
        "my_feature = \"median_income\"    # the median income on a specific city block.\n",
        "my_label = \"median_house_value\" # the median house value on a specific city block.\n",
        "# That is, you're going to create a model that predicts house value based \n",
        "# solely on the neighborhood's median income.  \n",
        "\n",
        "# Invoke the functions to build and train the model.\n",
        "### CONTINUE THE CODE HERE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKa11JK4Pm3f"
      },
      "source": [
        "## Task 2: Determine **why** the loss curves differ\n",
        "\n",
        "No matter how you split the training set and the validation set, the loss curves differ significantly. Evidently, the data in the training set isn't similar enough to the data in the validation set. Counterintuitive? Yes, but this problem is actually pretty common in machine learning. \n",
        "\n",
        "Your task is to determine **why** the loss curves aren't highly similar. As with most issues in machine learning, the problem is rooted in the data itself. To solve this mystery of why the training set and validation set aren't almost identical, write a line or two of [pandas code](https://colab.research.google.com/github/google/eng-edu/blob/main/ml/cc/exercises/pandas_dataframe_ultraquick_tutorial.ipynb?utm_source=validation-colab&utm_medium=colab&utm_campaign=colab-external&utm_content=pandas_tf2-colab&hl=en) in the following code cell.  Here are a couple of hints:\n",
        "\n",
        "  * The previous code cell split the original training set into:\n",
        "    * a reduced training set (the original training set - the validation set)\n",
        "    * the validation set \n",
        "  * By default, the pandas [`head`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html) method outputs the *first* 5 rows of the DataFrame. To see more of the training set, specify the `n` argument to `head` and assign a large positive integer to `n`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VJQcAZkwJt_p"
      },
      "source": [
        "# Write some code in this code cell."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rw4xI1ZEckI8"
      },
      "source": [
        "## Task 3. Fix the problem\n",
        "\n",
        "To fix the problem, shuffle the examples in the training set before splitting the examples into a training set and validation set. To do so, take the following steps:\n",
        "\n",
        "1. Shuffle the data in the training set by adding the following line anywhere before you call `train_model` (in the code cell associated with Task 1):\n",
        "\n",
        "```\n",
        "  shuffled_train_df = train_df.reindex(np.random.permutation(train_df.index))\n",
        "```                                    \n",
        "\n",
        "2. Pass `shuffled_train_df` (instead of `train_df`) as the second argument to `train_model` (in the code call associated with Task 1) so that the call becomes as follows:\n",
        "\n",
        "```\n",
        "  epochs, rmse, history = train_model(my_model, shuffled_train_df, my_feature, \n",
        "                                      my_label, epochs, batch_size, \n",
        "                                      validation_split)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ruvY7tPm2lZC"
      },
      "source": [
        "# Write some code in this code cell."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tKN239_miW8C"
      },
      "source": [
        "Experiment with `validation_split` to answer the following questions:\n",
        "\n",
        "* With the training set shuffled, is the final loss for the training set closer to the final loss for the validation set?  \n",
        "* At what range of values of `validation_split` do the final loss values for the training set and validation set diverge meaningfully?  Why?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QQMNfrGN2puK"
      },
      "source": [
        "# Write some code in this code cell."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1PP-O8TOZOeo"
      },
      "source": [
        "## Task 4: Use the Test Dataset to Evaluate Your Model's Performance\n",
        "\n",
        "The test set usually acts as the ultimate judge of a model's quality. The test set can serve as an impartial judge because its examples haven't been used in training the model. Run the following code cell to evaluate the model with the test set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nd_Sw2cygOip"
      },
      "source": [
        "x_test = test_df[my_feature]\n",
        "y_test = test_df[my_label]\n",
        "\n",
        "results = my_model.evaluate(x_test, y_test, batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qoyQKvsjmV_A"
      },
      "source": [
        "Compare the root mean squared error of the model when evaluated on each of the three datasets:\n",
        "\n",
        "* training set: look for `root_mean_squared_error` in the final training epoch.\n",
        "* validation set: look for `val_root_mean_squared_error` in the final training epoch.\n",
        "* test set: run the preceding code cell and examine the `root_mean_squared_error`.\n",
        "\n",
        "Ideally, the root mean squared error of all three sets should be similar. Are they?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UfQi0LGY2s0A"
      },
      "source": [
        "# Write some code in this code cell."
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}