t81_558_class_05_5_bootstrap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_5_bootstrap.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 5: Regularization and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 5 Material\n",
    "\n",
    "* Part 5.1: Part 5.1: Introduction to Regularization: Ridge and Lasso [[Video]](https://www.youtube.com/watch?v=jfgRtCYjoBs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_1_reg_ridge_lasso.ipynb)\n",
    "* Part 5.2: Using K-Fold Cross Validation with Keras [[Video]](https://www.youtube.com/watch?v=maiQf8ray_s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_2_kfold.ipynb)\n",
    "* Part 5.3: Using L1 and L2 Regularization with Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=JEWzWv1fBFQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_3_keras_l1_l2.ipynb)\n",
    "* Part 5.4: Drop Out for Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=bRyOi0L6Rs8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_4_dropout.ipynb)\n",
    "* **Part 5.5: Benchmarking Keras Deep Learning Regularization Techniques** [[Video]](https://www.youtube.com/watch?v=1NLBwPumUAs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_5_bootstrap.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5.5: Benchmarking Keras Deep Learning Regularization Techniques\n",
    "\n",
    "Quite a few hyperparameters have been introduced so far.  Tweaking each of these values can have an effect on the score obtained by your neural networks.  Some of the hyperparameters seen so far include:\n",
    "\n",
    "* Number of layers in the neural network\n",
    "* How many neurons in each layer\n",
    "* What activation functions to use on each layer\n",
    "* Dropout percent on each layer\n",
    "* L1 and L2 values on each layer\n",
    "\n",
    "To try out each of these hyperparameters you will need to run train neural networks with multiple settings for each hyperparameter.  However, you may have noticed that neural networks often produce somewhat different results when trained multiple times.  This is because the neural networks start with random weights.  Because of this it is necessary to fit and evaluate a neural network times to ensure that one set of hyperparameters are actually better than another.  Bootstrapping can be an effective means of benchmarking (comparing) two sets of hyperparameters.  \n",
    "\n",
    "Bootstrapping is similar to cross-validation.  Both go through a number of cycles/folds providing validation and training sets.  However, bootstrapping can have an unlimited number of cycles.  Bootstrapping chooses a new train and validation split each cycle, with replacement.  The fact that each cycle is chosen with replacement means that, unlike cross validation, there will often be repeated rows selected between cycles.  If you run the bootstrap for enough cycles, there will be duplicate cycles.\n",
    "\n",
    "In this part we will use bootstrapping for hyperparameter benchmarking.  We will train a neural network for a specified number of splits (denoted by the SPLITS constant).  For these examples we use 100.  We will compare the average score at the end of the 100.  By the end of the cycles the mean score will have converged somewhat.  This ending score will be a much better basis of comparison than a single cross-validation.  Additionally, the average number of epochs will be tracked to give an idea of a possible optimal value.  Because the early stopping validation set is also used to evaluate the the neural network as well, it might be slightly inflated.  This is because we are both stopping and evaluating on the same sample.  However, we are using the scores only as relative measures to determine the superiority of one set of hyperparameters to another, so this slight inflation should not present too much of a problem.\n",
    "\n",
    "Because we are benchmarking, we will display the amount of time taken for each cycle.  The following function can be used to nicely format a time span."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Reading on Hyperparameter Tuning\n",
    "\n",
    "I will add more here as I encounter additional good sources:\n",
    "    \n",
    "* [A Recipe for Training Neural Networks](http://karpathy.github.io/2019/04/25/recipe/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrapping for Regression\n",
    "\n",
    "Regression bootstrapping uses the **ShuffleSplit** object to perform the splits.  This is similar to **KFold** for cross validation, no balancing takes place.  To demonstrate this technique we will attempt to predict the age column for the jh-simple-dataset this data is loaded by the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code performs the bootstrap.  The architecture of the neural network can be adjusted to compare many different configurations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#1: score=0.713101, mean score=0.713101, stdev=0.000000, epochs=101, mean epochs=101, time=0:00:14.91\n",
      "#2: score=0.843020, mean score=0.778061, stdev=0.064960, epochs=123, mean epochs=112, time=0:00:17.20\n",
      "#3: score=0.793505, mean score=0.783209, stdev=0.053537, epochs=147, mean epochs=123, time=0:00:22.03\n",
      "#4: score=0.685434, mean score=0.758765, stdev=0.062786, epochs=114, mean epochs=121, time=0:00:16.94\n",
      "#5: score=0.514140, mean score=0.709840, stdev=0.112820, epochs=140, mean epochs=125, time=0:00:20.09\n",
      "#6: score=0.702493, mean score=0.708616, stdev=0.103026, epochs=116, mean epochs=123, time=0:00:16.76\n",
      "#7: score=0.602510, mean score=0.693458, stdev=0.102356, epochs=91, mean epochs=118, time=0:00:13.23\n",
      "#8: score=0.612711, mean score=0.683364, stdev=0.099399, epochs=114, mean epochs=118, time=0:00:16.77\n",
      "#9: score=0.698976, mean score=0.685099, stdev=0.093843, epochs=117, mean epochs=118, time=0:00:16.98\n",
      "#10: score=1.115985, mean score=0.728188, stdev=0.156957, epochs=123, mean epochs=118, time=0:00:17.61\n",
      "#11: score=0.910434, mean score=0.744755, stdev=0.158559, epochs=117, mean epochs=118, time=0:00:16.95\n",
      "#12: score=0.810006, mean score=0.750193, stdev=0.152876, epochs=119, mean epochs=118, time=0:00:17.02\n",
      "#13: score=0.571687, mean score=0.736462, stdev=0.154388, epochs=124, mean epochs=118, time=0:00:17.99\n",
      "#14: score=1.174216, mean score=0.767730, stdev=0.186664, epochs=89, mean epochs=116, time=0:00:13.03\n",
      "#15: score=0.692769, mean score=0.762732, stdev=0.181301, epochs=110, mean epochs=116, time=0:00:15.89\n",
      "#16: score=0.819567, mean score=0.766285, stdev=0.176082, epochs=126, mean epochs=116, time=0:00:18.35\n",
      "#17: score=0.750111, mean score=0.765333, stdev=0.170867, epochs=96, mean epochs=115, time=0:00:14.12\n",
      "#18: score=0.964762, mean score=0.776413, stdev=0.172222, epochs=107, mean epochs=115, time=0:00:15.51\n",
      "#19: score=0.662127, mean score=0.770398, stdev=0.169560, epochs=122, mean epochs=115, time=0:00:17.84\n",
      "#20: score=0.801966, mean score=0.771976, stdev=0.165410, epochs=133, mean epochs=116, time=0:00:19.00\n",
      "#21: score=0.914573, mean score=0.778766, stdev=0.164255, epochs=102, mean epochs=115, time=0:00:14.81\n",
      "#22: score=0.541062, mean score=0.767962, stdev=0.167943, epochs=97, mean epochs=114, time=0:00:14.25\n",
      "#23: score=0.748295, mean score=0.767107, stdev=0.164301, epochs=103, mean epochs=114, time=0:00:15.05\n",
      "#24: score=0.655802, mean score=0.762469, stdev=0.162372, epochs=125, mean epochs=114, time=0:00:19.60\n",
      "#25: score=0.572725, mean score=0.754879, stdev=0.163379, epochs=149, mean epochs=116, time=0:00:21.31\n",
      "#26: score=0.641570, mean score=0.750521, stdev=0.161681, epochs=90, mean epochs=115, time=0:00:14.43\n",
      "#27: score=0.767380, mean score=0.751145, stdev=0.158691, epochs=129, mean epochs=115, time=0:00:18.74\n",
      "#28: score=0.835364, mean score=0.754153, stdev=0.156613, epochs=99, mean epochs=115, time=0:00:14.37\n",
      "#29: score=0.586128, mean score=0.748359, stdev=0.156913, epochs=93, mean epochs=114, time=0:00:13.81\n",
      "#30: score=0.562052, mean score=0.742149, stdev=0.157859, epochs=141, mean epochs=115, time=0:00:20.07\n",
      "#31: score=0.590155, mean score=0.737246, stdev=0.157597, epochs=119, mean epochs=115, time=0:00:17.04\n",
      "#32: score=0.508154, mean score=0.730087, stdev=0.160155, epochs=153, mean epochs=116, time=0:00:22.14\n",
      "#33: score=0.704053, mean score=0.729298, stdev=0.157772, epochs=156, mean epochs=117, time=0:00:22.14\n",
      "#34: score=1.209816, mean score=0.743431, stdev=0.175361, epochs=105, mean epochs=117, time=0:00:15.28\n",
      "#35: score=0.675447, mean score=0.741488, stdev=0.173208, epochs=150, mean epochs=118, time=0:00:21.76\n",
      "#36: score=0.642438, mean score=0.738737, stdev=0.171559, epochs=93, mean epochs=117, time=0:00:13.70\n",
      "#37: score=1.205389, mean score=0.751349, stdev=0.185374, epochs=92, mean epochs=116, time=0:00:14.20\n",
      "#38: score=0.613412, mean score=0.747719, stdev=0.184247, epochs=139, mean epochs=117, time=0:00:20.07\n",
      "#39: score=0.576269, mean score=0.743323, stdev=0.183877, epochs=140, mean epochs=118, time=0:00:20.18\n",
      "#40: score=0.619933, mean score=0.740238, stdev=0.182583, epochs=139, mean epochs=118, time=0:00:19.99\n",
      "#41: score=1.266249, mean score=0.753068, stdev=0.197756, epochs=103, mean epochs=118, time=0:00:15.85\n",
      "#42: score=0.608243, mean score=0.749620, stdev=0.196631, epochs=127, mean epochs=118, time=0:00:19.50\n",
      "#43: score=0.613483, mean score=0.746454, stdev=0.195411, epochs=142, mean epochs=118, time=0:00:25.32\n",
      "#44: score=0.659772, mean score=0.744484, stdev=0.193610, epochs=112, mean epochs=118, time=0:00:20.00\n",
      "#45: score=1.157535, mean score=0.753663, stdev=0.200895, epochs=88, mean epochs=118, time=0:00:16.34\n",
      "#46: score=0.706465, mean score=0.752637, stdev=0.198818, epochs=110, mean epochs=117, time=0:00:19.56\n",
      "#47: score=0.653439, mean score=0.750526, stdev=0.197212, epochs=102, mean epochs=117, time=0:00:16.68\n",
      "#48: score=0.606111, mean score=0.747517, stdev=0.196234, epochs=83, mean epochs=116, time=0:00:14.03\n",
      "#49: score=1.007754, mean score=0.752828, stdev=0.197676, epochs=108, mean epochs=116, time=0:00:19.17\n",
      "#50: score=0.529380, mean score=0.748359, stdev=0.198174, epochs=172, mean epochs=117, time=0:00:27.94\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "import time\n",
    "import statistics\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "SPLITS = 50\n",
    "\n",
    "# Bootstrap\n",
    "boot = ShuffleSplit(n_splits=SPLITS, test_size=0.1, random_state=42)\n",
    "\n",
    "# Track progress\n",
    "mean_benchmark = []\n",
    "epochs_needed = []\n",
    "num = 0\n",
    "\n",
    "# Loop through samples\n",
    "for train, test in boot.split(x):\n",
    "    start_time = time.time()\n",
    "    num+=1\n",
    "\n",
    "    # Split train and test\n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "\n",
    "    # Construct neural network\n",
    "    model = Sequential()\n",
    "    model.add(Dense(20, input_dim=x_train.shape[1], activation='relu'))\n",
    "    model.add(Dense(10, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    \n",
    "    monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "        patience=5, verbose=0, mode='auto', restore_best_weights=True)\n",
    "\n",
    "    # Train on the bootstrap sample\n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "              callbacks=[monitor],verbose=0,epochs=1000)\n",
    "    epochs = monitor.stopped_epoch\n",
    "    epochs_needed.append(epochs)\n",
    "    \n",
    "    # Predict on the out of boot (validation)\n",
    "    pred = model.predict(x_test)\n",
    "  \n",
    "    # Measure this bootstrap's log loss\n",
    "    score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "    mean_benchmark.append(score)\n",
    "    m1 = statistics.mean(mean_benchmark)\n",
    "    m2 = statistics.mean(epochs_needed)\n",
    "    mdev = statistics.pstdev(mean_benchmark)\n",
    "    \n",
    "    # Record this iteration\n",
    "    time_took = time.time() - start_time\n",
    "    print(f\"#{num}: score={score:.6f}, mean score={m1:.6f}, stdev={mdev:.6f}, \n",
    "          epochs={epochs}, mean epochs={int(m2)}, \n",
    "          time={hms_string(time_took)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bootstrapping process for classification is similar and is presented in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrapping for Classification\n",
    "\n",
    "Regression bootstrapping uses the **StratifiedShuffleSplit** object to perform the splits.  This is similar to **StratifiedKFold** for cross validation, as the classes are balanced so that the sampling has no effect on proportions.  To demonstrate this technique we will attempt to predict the product column for the jh-simple-dataset this data is loaded by the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#1: score=0.632018, mean score=0.632018, stdev=0.000000, epochs=100, mean epochs=100, time=0:00:18.69\n",
      "#2: score=0.655166, mean score=0.643592, stdev=0.011574, epochs=67, mean epochs=83, time=0:00:13.48\n",
      "#3: score=0.683039, mean score=0.656741, stdev=0.020859, epochs=45, mean epochs=70, time=0:00:09.34\n",
      "#4: score=0.655106, mean score=0.656332, stdev=0.018078, epochs=102, mean epochs=78, time=0:00:19.19\n",
      "#5: score=0.681267, mean score=0.661319, stdev=0.018998, epochs=57, mean epochs=74, time=0:00:11.37\n",
      "#6: score=0.687970, mean score=0.665761, stdev=0.019986, epochs=55, mean epochs=71, time=0:00:10.77\n",
      "#7: score=0.692665, mean score=0.669604, stdev=0.020761, epochs=62, mean epochs=69, time=0:00:12.14\n",
      "#8: score=0.744507, mean score=0.678967, stdev=0.031476, epochs=48, mean epochs=67, time=0:00:09.72\n",
      "#9: score=0.593017, mean score=0.669417, stdev=0.040128, epochs=79, mean epochs=68, time=0:00:15.52\n",
      "#10: score=0.650762, mean score=0.667552, stdev=0.038478, epochs=84, mean epochs=69, time=0:00:16.44\n",
      "#11: score=0.689548, mean score=0.669551, stdev=0.037229, epochs=66, mean epochs=69, time=0:00:13.05\n",
      "#12: score=0.705157, mean score=0.672519, stdev=0.036977, epochs=61, mean epochs=68, time=0:00:12.14\n",
      "#13: score=0.702199, mean score=0.674802, stdev=0.036396, epochs=109, mean epochs=71, time=0:00:20.86\n",
      "#14: score=0.746486, mean score=0.679922, stdev=0.039635, epochs=42, mean epochs=69, time=0:00:08.58\n",
      "#15: score=0.680792, mean score=0.679980, stdev=0.038291, epochs=76, mean epochs=70, time=0:00:14.97\n",
      "#16: score=0.747207, mean score=0.684182, stdev=0.040489, epochs=50, mean epochs=68, time=0:00:10.15\n",
      "#17: score=0.650663, mean score=0.682210, stdev=0.040064, epochs=71, mean epochs=69, time=0:00:14.37\n",
      "#18: score=0.644407, mean score=0.680110, stdev=0.039887, epochs=50, mean epochs=68, time=0:00:10.15\n",
      "#19: score=0.617795, mean score=0.676830, stdev=0.041241, epochs=49, mean epochs=67, time=0:00:10.03\n",
      "#20: score=0.672280, mean score=0.676603, stdev=0.040209, epochs=46, mean epochs=65, time=0:00:09.59\n",
      "#21: score=0.648994, mean score=0.675288, stdev=0.039678, epochs=49, mean epochs=65, time=0:00:10.28\n",
      "#22: score=0.712242, mean score=0.676968, stdev=0.039523, epochs=42, mean epochs=64, time=0:00:08.86\n",
      "#23: score=0.620194, mean score=0.674499, stdev=0.040351, epochs=65, mean epochs=64, time=0:00:12.72\n",
      "#24: score=0.721169, mean score=0.676444, stdev=0.040587, epochs=112, mean epochs=66, time=0:00:21.58\n",
      "#25: score=0.589143, mean score=0.672952, stdev=0.043291, epochs=69, mean epochs=66, time=0:00:13.48\n",
      "#26: score=0.646644, mean score=0.671940, stdev=0.042750, epochs=92, mean epochs=67, time=0:00:18.03\n",
      "#27: score=0.697591, mean score=0.672890, stdev=0.042230, epochs=71, mean epochs=67, time=0:00:13.79\n",
      "#28: score=0.704431, mean score=0.674016, stdev=0.041880, epochs=103, mean epochs=68, time=0:00:19.51\n",
      "#29: score=0.649851, mean score=0.673183, stdev=0.041387, epochs=58, mean epochs=68, time=0:00:11.50\n",
      "#30: score=0.731638, mean score=0.675132, stdev=0.042023, epochs=47, mean epochs=67, time=0:00:09.46\n",
      "#31: score=0.716883, mean score=0.676479, stdev=0.041993, epochs=49, mean epochs=66, time=0:00:10.40\n",
      "#32: score=0.683501, mean score=0.676698, stdev=0.041349, epochs=49, mean epochs=66, time=0:00:09.90\n",
      "#33: score=0.628763, mean score=0.675245, stdev=0.041539, epochs=54, mean epochs=66, time=0:00:10.79\n",
      "#34: score=0.571066, mean score=0.672181, stdev=0.044548, epochs=79, mean epochs=66, time=0:00:15.36\n",
      "#35: score=0.514747, mean score=0.667683, stdev=0.051145, epochs=96, mean epochs=67, time=0:00:18.83\n",
      "#36: score=0.644418, mean score=0.667037, stdev=0.050574, epochs=57, mean epochs=66, time=0:00:11.80\n",
      "#37: score=0.668499, mean score=0.667076, stdev=0.049886, epochs=61, mean epochs=66, time=0:00:12.49\n",
      "#38: score=0.680755, mean score=0.667436, stdev=0.049274, epochs=80, mean epochs=67, time=0:00:15.78\n",
      "#39: score=0.795418, mean score=0.670718, stdev=0.052677, epochs=58, mean epochs=66, time=0:00:11.76\n",
      "#40: score=0.692155, mean score=0.671254, stdev=0.052122, epochs=51, mean epochs=66, time=0:00:10.33\n",
      "#41: score=0.749775, mean score=0.673169, stdev=0.052889, epochs=38, mean epochs=65, time=0:00:08.70\n",
      "#42: score=0.637172, mean score=0.672312, stdev=0.052543, epochs=64, mean epochs=65, time=0:00:13.08\n",
      "#43: score=0.676773, mean score=0.672416, stdev=0.051932, epochs=64, mean epochs=65, time=0:00:12.82\n",
      "#44: score=0.781751, mean score=0.674901, stdev=0.053863, epochs=65, mean epochs=65, time=0:00:13.03\n",
      "#45: score=0.688358, mean score=0.675200, stdev=0.053298, epochs=60, mean epochs=65, time=0:00:12.04\n",
      "#46: score=0.676594, mean score=0.675230, stdev=0.052716, epochs=48, mean epochs=65, time=0:00:10.15\n",
      "#47: score=0.683254, mean score=0.675401, stdev=0.052165, epochs=62, mean epochs=65, time=0:00:13.14\n",
      "#48: score=0.748860, mean score=0.676931, stdev=0.052674, epochs=64, mean epochs=65, time=0:00:13.92\n",
      "#49: score=0.656282, mean score=0.676510, stdev=0.052215, epochs=60, mean epochs=65, time=0:00:12.04\n",
      "#50: score=0.696442, mean score=0.676908, stdev=0.051766, epochs=42, mean epochs=64, time=0:00:08.78\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "import time\n",
    "import statistics\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "SPLITS = 50\n",
    "\n",
    "# Bootstrap\n",
    "boot = StratifiedShuffleSplit(n_splits=SPLITS, test_size=0.1, \n",
    "                                random_state=42)\n",
    "\n",
    "# Track progress\n",
    "mean_benchmark = []\n",
    "epochs_needed = []\n",
    "num = 0\n",
    "\n",
    "# Loop through samples\n",
    "for train, test in boot.split(x,df['product']):\n",
    "    start_time = time.time()\n",
    "    num+=1\n",
    "\n",
    "    # Split train and test\n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "\n",
    "    # Construct neural network\n",
    "    model = Sequential()\n",
    "    model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "    model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "    model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "    monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "        patience=25, verbose=0, mode='auto', restore_best_weights=True)\n",
    "\n",
    "    # Train on the bootstrap sample\n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "              callbacks=[monitor],verbose=0,epochs=1000)\n",
    "    epochs = monitor.stopped_epoch\n",
    "    epochs_needed.append(epochs)\n",
    "    \n",
    "    # Predict on the out of boot (validation)\n",
    "    pred = model.predict(x_test)\n",
    "  \n",
    "    # Measure this bootstrap's log loss\n",
    "    y_compare = np.argmax(y_test,axis=1) # For log loss calculation\n",
    "    score = metrics.log_loss(y_compare, pred)\n",
    "    mean_benchmark.append(score)\n",
    "    m1 = statistics.mean(mean_benchmark)\n",
    "    m2 = statistics.mean(epochs_needed)\n",
    "    mdev = statistics.pstdev(mean_benchmark)\n",
    "    \n",
    "    # Record this iteration\n",
    "    time_took = time.time() - start_time\n",
    "    print(f\"#{num}: score={score:.6f}, mean score={m1:.6f}, stdev={mdev:.6f}, epochs={epochs}, mean epochs={int(m2)}, time={hms_string(time_took)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmarking\n",
    "\n",
    "Now that we've seen how to bootstrap with both classification and regression we can start to try to optimize the hyperparameters for the jh-simple-dataset data.  For this example we will encode for classification of the product column.  Evaluation will be in log loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],\n",
    "               axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I performed some optimization and the code is currently set to the best settings that I came up with. Later in this course we will see how we can use an automatic process to optimize the hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#1: score=0.667172, mean score=0.667172, stdev=0.000000, epochs=236, mean epochs=236, time=0:01:41.02\n",
      "#2: score=0.672654, mean score=0.669913, stdev=0.002741, epochs=166, mean epochs=201, time=0:01:12.49\n",
      "#3: score=0.638013, mean score=0.659280, stdev=0.015203, epochs=235, mean epochs=212, time=0:01:37.79\n",
      "#4: score=0.608298, mean score=0.646534, stdev=0.025704, epochs=236, mean epochs=218, time=0:01:37.65\n",
      "#5: score=0.713477, mean score=0.659923, stdev=0.035293, epochs=140, mean epochs=202, time=0:00:59.98\n",
      "#6: score=0.583086, mean score=0.647117, stdev=0.043104, epochs=224, mean epochs=206, time=0:01:33.00\n",
      "#7: score=0.662984, mean score=0.649383, stdev=0.040291, epochs=199, mean epochs=205, time=0:01:22.04\n",
      "#8: score=0.636290, mean score=0.647747, stdev=0.037937, epochs=264, mean epochs=212, time=0:01:48.70\n",
      "#9: score=0.667210, mean score=0.649909, stdev=0.036287, epochs=155, mean epochs=206, time=0:01:04.21\n",
      "#10: score=0.615860, mean score=0.646504, stdev=0.035908, epochs=163, mean epochs=201, time=0:01:06.02\n",
      "#11: score=0.648626, mean score=0.646697, stdev=0.034242, epochs=175, mean epochs=199, time=0:01:10.91\n",
      "#12: score=0.665190, mean score=0.648238, stdev=0.033181, epochs=188, mean epochs=198, time=0:01:16.27\n",
      "#13: score=0.598302, mean score=0.644397, stdev=0.034545, epochs=219, mean epochs=200, time=0:01:27.04\n",
      "#14: score=0.611978, mean score=0.642081, stdev=0.034319, epochs=225, mean epochs=201, time=0:01:31.19\n",
      "#15: score=0.663114, mean score=0.643484, stdev=0.033568, epochs=153, mean epochs=198, time=0:01:03.48\n",
      "#16: score=0.624269, mean score=0.642283, stdev=0.032833, epochs=166, mean epochs=196, time=0:01:07.49\n",
      "#17: score=0.717895, mean score=0.646730, stdev=0.036485, epochs=292, mean epochs=202, time=0:02:03.20\n",
      "#18: score=0.588706, mean score=0.643507, stdev=0.037866, epochs=266, mean epochs=205, time=0:01:46.75\n",
      "#19: score=0.638096, mean score=0.643222, stdev=0.036876, epochs=235, mean epochs=207, time=0:01:34.03\n",
      "#20: score=0.594079, mean score=0.640765, stdev=0.037504, epochs=164, mean epochs=205, time=0:01:05.31\n",
      "#21: score=0.549027, mean score=0.636396, stdev=0.041488, epochs=257, mean epochs=207, time=0:01:42.09\n",
      "#22: score=0.684634, mean score=0.638589, stdev=0.041761, epochs=166, mean epochs=205, time=0:01:06.69\n",
      "#23: score=0.664673, mean score=0.639723, stdev=0.041188, epochs=138, mean epochs=202, time=0:00:56.99\n",
      "#24: score=0.664802, mean score=0.640768, stdev=0.040631, epochs=196, mean epochs=202, time=0:01:17.70\n",
      "#25: score=0.688673, mean score=0.642684, stdev=0.040902, epochs=140, mean epochs=199, time=0:00:55.92\n",
      "#26: score=0.557733, mean score=0.639417, stdev=0.043307, epochs=258, mean epochs=202, time=0:01:42.33\n",
      "#27: score=0.699693, mean score=0.641649, stdev=0.043996, epochs=144, mean epochs=200, time=0:00:58.39\n",
      "#28: score=0.675730, mean score=0.642867, stdev=0.043663, epochs=180, mean epochs=199, time=0:01:14.50\n",
      "#29: score=0.753875, mean score=0.646694, stdev=0.047445, epochs=213, mean epochs=199, time=0:01:31.28\n",
      "#30: score=0.658910, mean score=0.647102, stdev=0.046699, epochs=204, mean epochs=199, time=0:01:27.94\n",
      "#31: score=0.612882, mean score=0.645998, stdev=0.046336, epochs=147, mean epochs=198, time=0:01:07.93\n",
      "#32: score=0.658263, mean score=0.646381, stdev=0.045656, epochs=205, mean epochs=198, time=0:01:28.53\n",
      "#33: score=0.615351, mean score=0.645441, stdev=0.045272, epochs=181, mean epochs=197, time=0:01:20.98\n",
      "#34: score=0.637761, mean score=0.645215, stdev=0.044621, epochs=213, mean epochs=198, time=0:01:32.42\n",
      "#35: score=0.642996, mean score=0.645151, stdev=0.043980, epochs=198, mean epochs=198, time=0:01:27.50\n",
      "#36: score=0.671460, mean score=0.645882, stdev=0.043580, epochs=146, mean epochs=196, time=0:01:05.71\n",
      "#37: score=0.662100, mean score=0.646321, stdev=0.043067, epochs=264, mean epochs=198, time=0:01:51.06\n",
      "#38: score=0.659598, mean score=0.646670, stdev=0.042550, epochs=193, mean epochs=198, time=0:01:29.80\n",
      "#39: score=0.722732, mean score=0.648620, stdev=0.043688, epochs=189, mean epochs=198, time=0:01:28.61\n",
      "#40: score=0.644408, mean score=0.648515, stdev=0.043143, epochs=168, mean epochs=197, time=0:01:22.21\n",
      "#41: score=0.680791, mean score=0.649302, stdev=0.042904, epochs=168, mean epochs=196, time=0:01:15.22\n",
      "#42: score=0.640282, mean score=0.649087, stdev=0.042412, epochs=233, mean epochs=197, time=0:01:39.00\n",
      "#43: score=0.621055, mean score=0.648436, stdev=0.042128, epochs=288, mean epochs=199, time=0:02:00.59\n",
      "#44: score=0.621844, mean score=0.647831, stdev=0.041835, epochs=264, mean epochs=201, time=0:01:58.32\n",
      "#45: score=0.652669, mean score=0.647939, stdev=0.041374, epochs=168, mean epochs=200, time=0:01:13.73\n",
      "#46: score=0.648629, mean score=0.647954, stdev=0.040922, epochs=148, mean epochs=199, time=0:01:17.06\n",
      "#47: score=0.672136, mean score=0.648468, stdev=0.040634, epochs=135, mean epochs=197, time=0:01:11.74\n",
      "#48: score=0.726245, mean score=0.650089, stdev=0.041715, epochs=147, mean epochs=196, time=0:01:02.88\n",
      "#49: score=0.686220, mean score=0.650826, stdev=0.041602, epochs=199, mean epochs=196, time=0:01:22.78\n",
      "#50: score=0.646327, mean score=0.650736, stdev=0.041189, epochs=271, mean epochs=198, time=0:01:56.09\n",
      "#51: score=0.722010, mean score=0.652133, stdev=0.041963, epochs=135, mean epochs=197, time=0:00:57.01\n",
      "#52: score=0.670448, mean score=0.652486, stdev=0.041634, epochs=148, mean epochs=196, time=0:01:02.20\n",
      "#53: score=0.677840, mean score=0.652964, stdev=0.041383, epochs=133, mean epochs=195, time=0:00:56.96\n",
      "#54: score=0.620244, mean score=0.652358, stdev=0.041235, epochs=184, mean epochs=194, time=0:01:18.71\n",
      "#55: score=0.666669, mean score=0.652618, stdev=0.040903, epochs=206, mean epochs=195, time=0:01:38.92\n",
      "#56: score=0.661755, mean score=0.652782, stdev=0.040554, epochs=194, mean epochs=195, time=0:01:29.09\n",
      "#57: score=0.679234, mean score=0.653246, stdev=0.040346, epochs=152, mean epochs=194, time=0:01:13.84\n",
      "#58: score=0.707453, mean score=0.654180, stdev=0.040615, epochs=139, mean epochs=193, time=0:01:00.65\n",
      "#59: score=0.682023, mean score=0.654652, stdev=0.040429, epochs=174, mean epochs=193, time=0:01:14.70\n",
      "#60: score=0.520656, mean score=0.652419, stdev=0.043607, epochs=349, mean epochs=195, time=0:02:33.93\n",
      "#61: score=0.656989, mean score=0.652494, stdev=0.043252, epochs=187, mean epochs=195, time=0:01:21.39\n",
      "#62: score=0.616703, mean score=0.651916, stdev=0.043138, epochs=266, mean epochs=196, time=0:01:51.97\n",
      "#63: score=0.509017, mean score=0.649648, stdev=0.046371, epochs=372, mean epochs=199, time=0:03:02.13\n",
      "#64: score=0.693095, mean score=0.650327, stdev=0.046322, epochs=151, mean epochs=198, time=0:01:16.62\n",
      "#65: score=0.642250, mean score=0.650203, stdev=0.045975, epochs=160, mean epochs=198, time=0:01:09.11\n",
      "#66: score=0.612865, mean score=0.649637, stdev=0.045853, epochs=141, mean epochs=197, time=0:01:02.69\n",
      "#67: score=0.659263, mean score=0.649781, stdev=0.045524, epochs=160, mean epochs=196, time=0:01:09.38\n",
      "#68: score=0.585105, mean score=0.648830, stdev=0.045854, epochs=203, mean epochs=196, time=0:01:34.32\n",
      "#69: score=0.598647, mean score=0.648102, stdev=0.045914, epochs=268, mean epochs=197, time=0:02:07.36\n",
      "#70: score=0.654746, mean score=0.648197, stdev=0.045592, epochs=193, mean epochs=197, time=0:01:25.96\n",
      "#71: score=0.637213, mean score=0.648043, stdev=0.045288, epochs=256, mean epochs=198, time=0:01:52.72\n",
      "#72: score=0.642632, mean score=0.647967, stdev=0.044977, epochs=193, mean epochs=198, time=0:01:26.73\n",
      "#73: score=0.667972, mean score=0.648241, stdev=0.044728, epochs=162, mean epochs=197, time=0:01:13.67\n",
      "#74: score=0.639040, mean score=0.648117, stdev=0.044438, epochs=192, mean epochs=197, time=0:01:28.22\n",
      "#75: score=0.584020, mean score=0.647262, stdev=0.044749, epochs=284, mean epochs=198, time=0:02:00.98\n",
      "#76: score=0.615059, mean score=0.646839, stdev=0.044604, epochs=251, mean epochs=199, time=0:01:45.83\n",
      "#77: score=0.643729, mean score=0.646798, stdev=0.044315, epochs=274, mean epochs=200, time=0:02:02.35\n",
      "#78: score=0.654714, mean score=0.646900, stdev=0.044039, epochs=257, mean epochs=201, time=0:02:01.56\n",
      "#79: score=0.728335, mean score=0.647931, stdev=0.044697, epochs=141, mean epochs=200, time=0:01:07.08\n",
      "#80: score=0.597498, mean score=0.647300, stdev=0.044768, epochs=238, mean epochs=201, time=0:01:55.90\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#81: score=0.633083, mean score=0.647125, stdev=0.044519, epochs=304, mean epochs=202, time=0:02:24.00\n",
      "#82: score=0.671827, mean score=0.647426, stdev=0.044330, epochs=168, mean epochs=201, time=0:01:23.74\n",
      "#83: score=0.718810, mean score=0.648286, stdev=0.044745, epochs=162, mean epochs=201, time=0:01:14.88\n",
      "#84: score=0.660492, mean score=0.648431, stdev=0.044497, epochs=193, mean epochs=201, time=0:01:23.00\n",
      "#85: score=0.651639, mean score=0.648469, stdev=0.044236, epochs=227, mean epochs=201, time=0:01:34.14\n",
      "#86: score=0.750126, mean score=0.649651, stdev=0.045308, epochs=148, mean epochs=201, time=0:00:59.56\n",
      "#87: score=0.638701, mean score=0.649525, stdev=0.045062, epochs=169, mean epochs=200, time=0:01:08.55\n",
      "#88: score=0.708435, mean score=0.650195, stdev=0.045239, epochs=201, mean epochs=200, time=0:01:18.31\n",
      "#89: score=0.730615, mean score=0.651098, stdev=0.045775, epochs=145, mean epochs=200, time=0:01:00.92\n",
      "#90: score=0.629939, mean score=0.650863, stdev=0.045574, epochs=306, mean epochs=201, time=0:02:26.12\n",
      "#91: score=0.622463, mean score=0.650551, stdev=0.045420, epochs=157, mean epochs=200, time=0:01:13.17\n",
      "#92: score=0.720805, mean score=0.651315, stdev=0.045756, epochs=251, mean epochs=201, time=0:01:53.27\n",
      "#93: score=0.668114, mean score=0.651495, stdev=0.045542, epochs=281, mean epochs=202, time=0:02:07.35\n",
      "#94: score=0.718123, mean score=0.652204, stdev=0.045812, epochs=151, mean epochs=201, time=0:01:07.41\n",
      "#95: score=0.675499, mean score=0.652449, stdev=0.045632, epochs=282, mean epochs=202, time=0:02:05.17\n",
      "#96: score=0.651784, mean score=0.652442, stdev=0.045394, epochs=200, mean epochs=202, time=0:01:29.75\n",
      "#97: score=0.646264, mean score=0.652379, stdev=0.045164, epochs=236, mean epochs=202, time=0:01:44.57\n",
      "#98: score=0.740089, mean score=0.653274, stdev=0.045789, epochs=201, mean epochs=202, time=0:01:35.27\n",
      "#99: score=0.587625, mean score=0.652611, stdev=0.046028, epochs=194, mean epochs=202, time=0:01:32.13\n",
      "#100: score=0.698883, mean score=0.653073, stdev=0.046028, epochs=148, mean epochs=202, time=0:01:09.97\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "import time\n",
    "import tensorflow.keras.initializers\n",
    "import statistics\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation, Dropout\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from tensorflow.keras.layers import LeakyReLU,PReLU\n",
    "\n",
    "SPLITS = 100\n",
    "\n",
    "# Bootstrap\n",
    "boot = StratifiedShuffleSplit(n_splits=SPLITS, test_size=0.1)\n",
    "\n",
    "# Track progress\n",
    "mean_benchmark = []\n",
    "epochs_needed = []\n",
    "num = 0\n",
    "\n",
    "# Loop through samples\n",
    "for train, test in boot.split(x,df['product']):\n",
    "    start_time = time.time()\n",
    "    num+=1\n",
    "\n",
    "    # Split train and test\n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "\n",
    "    # Construct neural network\n",
    "    # kernel_initializer = tensorflow.keras.initializers.he_uniform(seed=None)\n",
    "    model = Sequential()\n",
    "    model.add(Dense(100, input_dim=x.shape[1], activation=PReLU(), kernel_regularizer=regularizers.l2(1e-4)\n",
    "    )) # Hidden 1\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(100, activation=PReLU(), activity_regularizer=regularizers.l2(1e-4)\n",
    "    )) # Hidden 2\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(100, activation=PReLU(), activity_regularizer=regularizers.l2(1e-4)\n",
    "    )) # Hidden 3\n",
    "#    model.add(Dropout(0.5)) - Usually better performance without dropout on final layer\n",
    "    model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "    monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "        patience=100, verbose=0, mode='auto', restore_best_weights=True)\n",
    "\n",
    "    # Train on the bootstrap sample\n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=0,epochs=1000)\n",
    "    epochs = monitor.stopped_epoch\n",
    "    epochs_needed.append(epochs)\n",
    "    \n",
    "    # Predict on the out of boot (validation)\n",
    "    pred = model.predict(x_test)\n",
    "  \n",
    "    # Measure this bootstrap's log loss\n",
    "    y_compare = np.argmax(y_test,axis=1) # For log loss calculation\n",
    "    score = metrics.log_loss(y_compare, pred)\n",
    "    mean_benchmark.append(score)\n",
    "    m1 = statistics.mean(mean_benchmark)\n",
    "    m2 = statistics.mean(epochs_needed)\n",
    "    mdev = statistics.pstdev(mean_benchmark)\n",
    "    \n",
    "    # Record this iteration\n",
    "    time_took = time.time() - start_time\n",
    "    print(f\"#{num}: score={score:.6f}, mean score={m1:.6f},\"\\\n",
    "          stdev={mdev:.6f}, epochs={epochs},\"\\\n",
    "          mean epochs={int(m2)}, time={hms_string(time_took)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}