t81_558_class_03_3_save_load.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_2_keras.ipynb)\n",
    "* **Part 3.3: Saving and Loading a Keras Neural Network** [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow.\n",
    "  Running the following code will map your GDrive to ```/content/drive```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive', force_remount=True)\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "    %tensorflow_version 2.x\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.3: Saving and Loading a Keras Neural Network\n",
    "\n",
    "Complex neural networks will take a long time to fit/train.  It is helpful to be able to save these neural networks so that they can be reloaded later.  A reloaded neural network will not require retraining.  Keras provides three formats for neural network saving.\n",
    "\n",
    "* **YAML** - Stores the neural network structure (no weights) in the [YAML file format](https://en.wikipedia.org/wiki/YAML).\n",
    "* **JSON** - Stores the neural network structure (no weights) in the [JSON file format](https://en.wikipedia.org/wiki/JSON).\n",
    "* **HDF5** - Stores the complete neural network (with weights) in the [HDF5 file format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format). Do not confuse HDF5 with [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop).  They are different.  We do not use HDFS in this class.\n",
    "\n",
    "Usually you will want to save in HDF5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 398 samples\n",
      "Epoch 1/100\n",
      "398/398 - 0s - loss: 21424.3914\n",
      "Epoch 2/100\n",
      "398/398 - 0s - loss: 5458.3984\n",
      "Epoch 3/100\n",
      "398/398 - 0s - loss: 1736.3177\n",
      "Epoch 4/100\n",
      "398/398 - 0s - loss: 1013.3963\n",
      "Epoch 5/100\n",
      "398/398 - 0s - loss: 827.7916\n",
      "Epoch 6/100\n",
      "398/398 - 0s - loss: 744.9987\n",
      "Epoch 7/100\n",
      "398/398 - 0s - loss: 724.7204\n",
      "Epoch 8/100\n",
      "398/398 - 0s - loss: 687.8266\n",
      "Epoch 9/100\n",
      "398/398 - 0s - loss: 672.5687\n",
      "Epoch 10/100\n",
      "398/398 - 0s - loss: 638.7330\n",
      "Epoch 11/100\n",
      "398/398 - 0s - loss: 614.1585\n",
      "Epoch 12/100\n",
      "398/398 - 0s - loss: 591.2568\n",
      "Epoch 13/100\n",
      "398/398 - 0s - loss: 584.8373\n",
      "Epoch 14/100\n",
      "398/398 - 0s - loss: 557.7072\n",
      "Epoch 15/100\n",
      "398/398 - 0s - loss: 559.3462\n",
      "Epoch 16/100\n",
      "398/398 - 0s - loss: 515.3038\n",
      "Epoch 17/100\n",
      "398/398 - 0s - loss: 472.8301\n",
      "Epoch 18/100\n",
      "398/398 - 0s - loss: 457.8038\n",
      "Epoch 19/100\n",
      "398/398 - 0s - loss: 431.7448\n",
      "Epoch 20/100\n",
      "398/398 - 0s - loss: 409.7951\n",
      "Epoch 21/100\n",
      "398/398 - 0s - loss: 392.0069\n",
      "Epoch 22/100\n",
      "398/398 - 0s - loss: 372.8872\n",
      "Epoch 23/100\n",
      "398/398 - 0s - loss: 345.6591\n",
      "Epoch 24/100\n",
      "398/398 - 0s - loss: 327.6393\n",
      "Epoch 25/100\n",
      "398/398 - 0s - loss: 323.3945\n",
      "Epoch 26/100\n",
      "398/398 - 0s - loss: 303.9275\n",
      "Epoch 27/100\n",
      "398/398 - 0s - loss: 282.9888\n",
      "Epoch 28/100\n",
      "398/398 - 0s - loss: 281.1166\n",
      "Epoch 29/100\n",
      "398/398 - 0s - loss: 251.1791\n",
      "Epoch 30/100\n",
      "398/398 - 0s - loss: 263.1462\n",
      "Epoch 31/100\n",
      "398/398 - 0s - loss: 224.0301\n",
      "Epoch 32/100\n",
      "398/398 - 0s - loss: 207.0310\n",
      "Epoch 33/100\n",
      "398/398 - 0s - loss: 188.7971\n",
      "Epoch 34/100\n",
      "398/398 - 0s - loss: 190.0870\n",
      "Epoch 35/100\n",
      "398/398 - 0s - loss: 168.0055\n",
      "Epoch 36/100\n",
      "398/398 - 0s - loss: 166.9712\n",
      "Epoch 37/100\n",
      "398/398 - 0s - loss: 149.4432\n",
      "Epoch 38/100\n",
      "398/398 - 0s - loss: 142.3372\n",
      "Epoch 39/100\n",
      "398/398 - 0s - loss: 132.7649\n",
      "Epoch 40/100\n",
      "398/398 - 0s - loss: 126.9563\n",
      "Epoch 41/100\n",
      "398/398 - 0s - loss: 114.3744\n",
      "Epoch 42/100\n",
      "398/398 - 0s - loss: 103.9548\n",
      "Epoch 43/100\n",
      "398/398 - 0s - loss: 98.0673\n",
      "Epoch 44/100\n",
      "398/398 - 0s - loss: 88.5079\n",
      "Epoch 45/100\n",
      "398/398 - 0s - loss: 83.6630\n",
      "Epoch 46/100\n",
      "398/398 - 0s - loss: 79.1238\n",
      "Epoch 47/100\n",
      "398/398 - 0s - loss: 75.0433\n",
      "Epoch 48/100\n",
      "398/398 - 0s - loss: 67.5129\n",
      "Epoch 49/100\n",
      "398/398 - 0s - loss: 62.4234\n",
      "Epoch 50/100\n",
      "398/398 - 0s - loss: 58.6467\n",
      "Epoch 51/100\n",
      "398/398 - 0s - loss: 55.5149\n",
      "Epoch 52/100\n",
      "398/398 - 0s - loss: 50.0830\n",
      "Epoch 53/100\n",
      "398/398 - 0s - loss: 46.6857\n",
      "Epoch 54/100\n",
      "398/398 - 0s - loss: 44.2211\n",
      "Epoch 55/100\n",
      "398/398 - 0s - loss: 40.9284\n",
      "Epoch 56/100\n",
      "398/398 - 0s - loss: 40.2299\n",
      "Epoch 57/100\n",
      "398/398 - 0s - loss: 37.2234\n",
      "Epoch 58/100\n",
      "398/398 - 0s - loss: 36.0334\n",
      "Epoch 59/100\n",
      "398/398 - 0s - loss: 36.2726\n",
      "Epoch 60/100\n",
      "398/398 - 0s - loss: 29.7832\n",
      "Epoch 61/100\n",
      "398/398 - 0s - loss: 28.6994\n",
      "Epoch 62/100\n",
      "398/398 - 0s - loss: 27.1550\n",
      "Epoch 63/100\n",
      "398/398 - 0s - loss: 25.8870\n",
      "Epoch 64/100\n",
      "398/398 - 0s - loss: 24.2626\n",
      "Epoch 65/100\n",
      "398/398 - 0s - loss: 23.8089\n",
      "Epoch 66/100\n",
      "398/398 - 0s - loss: 23.8112\n",
      "Epoch 67/100\n",
      "398/398 - 0s - loss: 21.9835\n",
      "Epoch 68/100\n",
      "398/398 - 0s - loss: 20.6542\n",
      "Epoch 69/100\n",
      "398/398 - 0s - loss: 19.3080\n",
      "Epoch 70/100\n",
      "398/398 - 0s - loss: 19.2560\n",
      "Epoch 71/100\n",
      "398/398 - 0s - loss: 18.4726\n",
      "Epoch 72/100\n",
      "398/398 - 0s - loss: 18.1178\n",
      "Epoch 73/100\n",
      "398/398 - 0s - loss: 19.0762\n",
      "Epoch 74/100\n",
      "398/398 - 0s - loss: 17.5053\n",
      "Epoch 75/100\n",
      "398/398 - 0s - loss: 16.1993\n",
      "Epoch 76/100\n",
      "398/398 - 0s - loss: 16.7693\n",
      "Epoch 77/100\n",
      "398/398 - 0s - loss: 16.5427\n",
      "Epoch 78/100\n",
      "398/398 - 0s - loss: 15.9786\n",
      "Epoch 79/100\n",
      "398/398 - 0s - loss: 15.2967\n",
      "Epoch 80/100\n",
      "398/398 - 0s - loss: 15.5345\n",
      "Epoch 81/100\n",
      "398/398 - 0s - loss: 16.0734\n",
      "Epoch 82/100\n",
      "398/398 - 0s - loss: 16.3355\n",
      "Epoch 83/100\n",
      "398/398 - 0s - loss: 14.7587\n",
      "Epoch 84/100\n",
      "398/398 - 0s - loss: 13.9227\n",
      "Epoch 85/100\n",
      "398/398 - 0s - loss: 14.1262\n",
      "Epoch 86/100\n",
      "398/398 - 0s - loss: 13.9718\n",
      "Epoch 87/100\n",
      "398/398 - 0s - loss: 14.7875\n",
      "Epoch 88/100\n",
      "398/398 - 0s - loss: 15.4308\n",
      "Epoch 89/100\n",
      "398/398 - 0s - loss: 13.4084\n",
      "Epoch 90/100\n",
      "398/398 - 0s - loss: 13.2533\n",
      "Epoch 91/100\n",
      "398/398 - 0s - loss: 14.1100\n",
      "Epoch 92/100\n",
      "398/398 - 0s - loss: 13.2194\n",
      "Epoch 93/100\n",
      "398/398 - 0s - loss: 12.9851\n",
      "Epoch 94/100\n",
      "398/398 - 0s - loss: 12.9105\n",
      "Epoch 95/100\n",
      "398/398 - 0s - loss: 13.2677\n",
      "Epoch 96/100\n",
      "398/398 - 0s - loss: 12.8689\n",
      "Epoch 97/100\n",
      "398/398 - 0s - loss: 13.1930\n",
      "Epoch 98/100\n",
      "398/398 - 0s - loss: 13.2477\n",
      "Epoch 99/100\n",
      "398/398 - 0s - loss: 12.6923\n",
      "Epoch 100/100\n",
      "398/398 - 0s - loss: 12.7204\n",
      "Before save score (RMSE): 3.6842460468319183\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "save_path = \".\"\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Before save score (RMSE): {score}\")\n",
    "\n",
    "# save neural network structure to JSON (no weights)\n",
    "model_json = model.to_json()\n",
    "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "\n",
    "# save neural network structure to YAML (no weights)\n",
    "model_yaml = model.to_yaml()\n",
    "with open(os.path.join(save_path,\"network.yaml\"), \"w\") as yaml_file:\n",
    "    yaml_file.write(model_yaml)\n",
    "\n",
    "# save entire network to HDF5 (save everything, suggested)\n",
    "model.save(os.path.join(save_path,\"network.h5\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network.  The weights from the previous fit are used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we reload the network and perform another prediction.  The RMSE should match the previous one exactly if the neural network was really saved and reloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After load score (RMSE): 3.6842460468319183\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import load_model\n",
    "model2 = load_model(os.path.join(save_path,\"network.h5\"))\n",
    "pred = model2.predict(x)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"After load score (RMSE): {score}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}