t81_558_class_04_1_feature_encode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_1_feature_encode.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* **Part 4.1: Encoding a Feature Vector for Keras Deep Learning** [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_1_feature_encode.ipynb)\n",
    "* Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_2_multi_class.ipynb)\n",
    "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_3_regression.ipynb)\n",
    "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.1: Encoding a Feature Vector for Keras Deep Learning\n",
    "\n",
    "Neural networks can accept many types of data.  We will begin with tabular data, where there are well defined rows and columns.  This is the sort of data you would typically see in Microsoft Excel.  An example of tabular data is shown below.\n",
    "\n",
    "Neural networks require numeric input.  This numeric form is called a feature vector.  Each row of training data typically becomes one vector.  The individual input neurons each receive one feature (or column) from this vector.  In this section, we will see how to encode the following tabular data into a feature vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>aspect</th>\n",
       "      <th>...</th>\n",
       "      <th>age</th>\n",
       "      <th>pop_dense</th>\n",
       "      <th>retail_dense</th>\n",
       "      <th>crime</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>0.885827</td>\n",
       "      <td>0.492126</td>\n",
       "      <td>0.071100</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>18.625000</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>0.874016</td>\n",
       "      <td>0.342520</td>\n",
       "      <td>0.400809</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>c</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>...</td>\n",
       "      <td>46</td>\n",
       "      <td>0.909449</td>\n",
       "      <td>0.598425</td>\n",
       "      <td>0.117803</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>16.891667</td>\n",
       "      <td>...</td>\n",
       "      <td>48</td>\n",
       "      <td>0.925197</td>\n",
       "      <td>0.539370</td>\n",
       "      <td>0.451973</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job area   income     aspect  ...  age  pop_dense  retail_dense  \\\n",
       "0        1  vv    c  50876.0  13.100000  ...   49   0.885827      0.492126   \n",
       "1        2  kd    c  60369.0  18.625000  ...   51   0.874016      0.342520   \n",
       "...    ...  ..  ...      ...        ...  ...  ...        ...           ...   \n",
       "1998  1999  qp    c  67949.0   5.733333  ...   46   0.909449      0.598425   \n",
       "1999  2000  pe    c  61467.0  16.891667  ...   48   0.925197      0.539370   \n",
       "\n",
       "         crime  product  \n",
       "0     0.071100        b  \n",
       "1     0.400809        c  \n",
       "...        ...      ...  \n",
       "1998  0.117803        c  \n",
       "1999  0.451973        c  \n",
       "\n",
       "[2000 rows x 14 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "pd.set_option('display.max_columns', 9)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following observations can be made from the above data:\n",
    "* The target column is the column that you seek to predict.  There are several candidates here.  However, we will initially use product.  This field specifies what product someone bought.\n",
    "* There is an ID column.  This column should not be fed into the neural network as it contains no information useful for prediction.\n",
    "* Many of these fields are numeric and might not require any further processing.\n",
    "* The income column does have some missing values.\n",
    "* There are categorical values: job, area, and product.\n",
    "\n",
    "To begin with, we will convert the job code into dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2000, 33)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>job_11</th>\n",
       "      <th>job_al</th>\n",
       "      <th>job_am</th>\n",
       "      <th>job_ax</th>\n",
       "      <th>job_bf</th>\n",
       "      <th>...</th>\n",
       "      <th>job_qw</th>\n",
       "      <th>job_rn</th>\n",
       "      <th>job_sa</th>\n",
       "      <th>job_vv</th>\n",
       "      <th>job_zz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      job_11  job_al  job_am  job_ax  job_bf  ...  job_qw  job_rn  job_sa  \\\n",
       "0          0       0       0       0       0  ...       0       0       0   \n",
       "1          0       0       0       0       0  ...       0       0       0   \n",
       "2          0       0       0       0       0  ...       0       0       0   \n",
       "3          1       0       0       0       0  ...       0       0       0   \n",
       "4          0       0       0       0       0  ...       0       0       0   \n",
       "...      ...     ...     ...     ...     ...  ...     ...     ...     ...   \n",
       "1995       0       0       0       0       0  ...       0       0       0   \n",
       "1996       0       0       0       0       0  ...       0       0       0   \n",
       "1997       0       0       0       0       0  ...       0       0       0   \n",
       "1998       0       0       0       0       0  ...       0       0       0   \n",
       "1999       0       0       0       0       0  ...       0       0       0   \n",
       "\n",
       "      job_vv  job_zz  \n",
       "0          1       0  \n",
       "1          0       0  \n",
       "2          0       0  \n",
       "3          0       0  \n",
       "4          0       0  \n",
       "...      ...     ...  \n",
       "1995       1       0  \n",
       "1996       0       0  \n",
       "1997       0       0  \n",
       "1998       0       0  \n",
       "1999       0       0  \n",
       "\n",
       "[2000 rows x 33 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dummies = pd.get_dummies(df['job'],prefix=\"job\")\n",
    "print(dummies.shape)\n",
    "\n",
    "pd.set_option('display.max_columns', 9)\n",
    "pd.set_option('display.max_rows', 10)\n",
    "\n",
    "display(dummies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because there are 33 different job codes, there are 33 dummy variables.  We also specified a prefix, because the job codes (such as \"ax\") are not that meaningful by themselves.  Something such as \"job_ax\" also tells us the origin of this field.\n",
    "\n",
    "Next, we must merge these dummies back into the main data frame.  We also drop the original \"job\" field, as it is now represented by the dummies. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>aspect</th>\n",
       "      <th>subscriptions</th>\n",
       "      <th>...</th>\n",
       "      <th>job_qw</th>\n",
       "      <th>job_rn</th>\n",
       "      <th>job_sa</th>\n",
       "      <th>job_vv</th>\n",
       "      <th>job_zz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>18.625000</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>34.766667</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>c</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>15.808333</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>d</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>40.941667</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>c</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>38.233333</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>d</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>33.358333</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>d</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>39.425000</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>c</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>c</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>16.891667</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 46 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id area   income     aspect  subscriptions  ...  job_qw  job_rn  \\\n",
       "0        1    c  50876.0  13.100000              1  ...       0       0   \n",
       "1        2    c  60369.0  18.625000              2  ...       0       0   \n",
       "2        3    c  55126.0  34.766667              1  ...       0       0   \n",
       "3        4    c  51690.0  15.808333              1  ...       0       0   \n",
       "4        5    d  28347.0  40.941667              3  ...       0       0   \n",
       "...    ...  ...      ...        ...            ...  ...     ...     ...   \n",
       "1995  1996    c  51017.0  38.233333              1  ...       0       0   \n",
       "1996  1997    d  26576.0  33.358333              2  ...       0       0   \n",
       "1997  1998    d  28595.0  39.425000              3  ...       0       0   \n",
       "1998  1999    c  67949.0   5.733333              0  ...       0       0   \n",
       "1999  2000    c  61467.0  16.891667              0  ...       0       0   \n",
       "\n",
       "      job_sa  job_vv  job_zz  \n",
       "0          0       1       0  \n",
       "1          0       0       0  \n",
       "2          0       0       0  \n",
       "3          0       0       0  \n",
       "4          0       0       0  \n",
       "...      ...     ...     ...  \n",
       "1995       0       1       0  \n",
       "1996       0       0       0  \n",
       "1997       0       0       0  \n",
       "1998       0       0       0  \n",
       "1999       0       0       0  \n",
       "\n",
       "[2000 rows x 46 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.concat([df,dummies],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "pd.set_option('display.max_columns', 9)\n",
    "pd.set_option('display.max_rows', 10)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also introduce dummy variables for the area column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>income</th>\n",
       "      <th>aspect</th>\n",
       "      <th>subscriptions</th>\n",
       "      <th>dist_healthy</th>\n",
       "      <th>...</th>\n",
       "      <th>job_zz</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>1</td>\n",
       "      <td>9.017895</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>18.625000</td>\n",
       "      <td>2</td>\n",
       "      <td>7.766643</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>34.766667</td>\n",
       "      <td>1</td>\n",
       "      <td>3.632069</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>15.808333</td>\n",
       "      <td>1</td>\n",
       "      <td>5.372942</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>40.941667</td>\n",
       "      <td>3</td>\n",
       "      <td>3.822477</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>38.233333</td>\n",
       "      <td>1</td>\n",
       "      <td>5.454545</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>33.358333</td>\n",
       "      <td>2</td>\n",
       "      <td>3.632069</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>39.425000</td>\n",
       "      <td>3</td>\n",
       "      <td>7.168218</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0</td>\n",
       "      <td>8.936292</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>16.891667</td>\n",
       "      <td>0</td>\n",
       "      <td>4.312097</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id   income     aspect  subscriptions  dist_healthy  ...  job_zz  \\\n",
       "0        1  50876.0  13.100000              1      9.017895  ...       0   \n",
       "1        2  60369.0  18.625000              2      7.766643  ...       0   \n",
       "2        3  55126.0  34.766667              1      3.632069  ...       0   \n",
       "3        4  51690.0  15.808333              1      5.372942  ...       0   \n",
       "4        5  28347.0  40.941667              3      3.822477  ...       0   \n",
       "...    ...      ...        ...            ...           ...  ...     ...   \n",
       "1995  1996  51017.0  38.233333              1      5.454545  ...       0   \n",
       "1996  1997  26576.0  33.358333              2      3.632069  ...       0   \n",
       "1997  1998  28595.0  39.425000              3      7.168218  ...       0   \n",
       "1998  1999  67949.0   5.733333              0      8.936292  ...       0   \n",
       "1999  2000  61467.0  16.891667              0      4.312097  ...       0   \n",
       "\n",
       "      area_a  area_b  area_c  area_d  \n",
       "0          0       0       1       0  \n",
       "1          0       0       1       0  \n",
       "2          0       0       1       0  \n",
       "3          0       0       1       0  \n",
       "4          0       0       0       1  \n",
       "...      ...     ...     ...     ...  \n",
       "1995       0       0       1       0  \n",
       "1996       0       0       0       1  \n",
       "1997       0       0       0       1  \n",
       "1998       0       0       1       0  \n",
       "1999       0       0       1       0  \n",
       "\n",
       "[2000 rows x 49 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "pd.set_option('display.max_columns', 9)\n",
    "pd.set_option('display.max_rows', 10)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last remaining transformation is to fill in missing income values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are more advanced ways of filling in missing values, but they require more analysis.  The idea would be to see if another field might give a hint as to what the income were.  For example, it might be beneficial to calculate a median income for each of the areas or job categories.  This is something to keep in mind for the class Kaggle competition.\n",
    "\n",
    "At this point, the Pandas dataframe is ready to be converted to Numpy for neural network training. We need to know a list of the columns that will make up *x* (the predictors or inputs) and *y* (the target). \n",
    "\n",
    "The complete list of columns is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'income', 'aspect', 'subscriptions', 'dist_healthy', 'save_rate', 'dist_unhealthy', 'age', 'pop_dense', 'retail_dense', 'crime', 'product', 'job_11', 'job_al', 'job_am', 'job_ax', 'job_bf', 'job_by', 'job_cv', 'job_de', 'job_dz', 'job_e2', 'job_f8', 'job_gj', 'job_gv', 'job_kd', 'job_ke', 'job_kl', 'job_kp', 'job_ks', 'job_kw', 'job_mm', 'job_nb', 'job_nn', 'job_ob', 'job_pe', 'job_po', 'job_pq', 'job_pz', 'job_qp', 'job_qw', 'job_rn', 'job_sa', 'job_vv', 'job_zz', 'area_a', 'area_b', 'area_c', 'area_d']\n"
     ]
    }
   ],
   "source": [
    "print(list(df.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This includes both the target and predictors.  We need a list with the target removed.  We also remove **id** because it is not useful for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['income', 'aspect', 'subscriptions', 'dist_healthy', 'save_rate', 'dist_unhealthy', 'age', 'pop_dense', 'retail_dense', 'crime', 'job_11', 'job_al', 'job_am', 'job_ax', 'job_bf', 'job_by', 'job_cv', 'job_de', 'job_dz', 'job_e2', 'job_f8', 'job_gj', 'job_gv', 'job_kd', 'job_ke', 'job_kl', 'job_kp', 'job_ks', 'job_kw', 'job_mm', 'job_nb', 'job_nn', 'job_ob', 'job_pe', 'job_po', 'job_pq', 'job_pz', 'job_qp', 'job_qw', 'job_rn', 'job_sa', 'job_vv', 'job_zz', 'area_a', 'area_b', 'area_c', 'area_d']\n"
     ]
    }
   ],
   "source": [
    "x_columns = df.columns.drop('product').drop('id')\n",
    "print(list(x_columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate X and Y for a Classification Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now generate *x* and *y*.  Note, this is how we generate y for a classification problem.  Regression would not use dummies and would simply encode the numeric value of the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can display the *x* and *y* matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5.08760000e+04 1.31000000e+01 1.00000000e+00 ... 0.00000000e+00\n",
      "  1.00000000e+00 0.00000000e+00]\n",
      " [6.03690000e+04 1.86250000e+01 2.00000000e+00 ... 0.00000000e+00\n",
      "  1.00000000e+00 0.00000000e+00]\n",
      " [5.51260000e+04 3.47666667e+01 1.00000000e+00 ... 0.00000000e+00\n",
      "  1.00000000e+00 0.00000000e+00]\n",
      " ...\n",
      " [2.85950000e+04 3.94250000e+01 3.00000000e+00 ... 0.00000000e+00\n",
      "  0.00000000e+00 1.00000000e+00]\n",
      " [6.79490000e+04 5.73333333e+00 0.00000000e+00 ... 0.00000000e+00\n",
      "  1.00000000e+00 0.00000000e+00]\n",
      " [6.14670000e+04 1.68916667e+01 0.00000000e+00 ... 0.00000000e+00\n",
      "  1.00000000e+00 0.00000000e+00]]\n",
      "[[0 1 0 ... 0 0 0]\n",
      " [0 0 1 ... 0 0 0]\n",
      " [0 1 0 ... 0 0 0]\n",
      " ...\n",
      " [0 0 0 ... 0 1 0]\n",
      " [0 0 1 ... 0 0 0]\n",
      " [0 0 1 ... 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The x and y values are now ready for a neural network.  Make sure that you construct the neural network for a classification problem.  Specifically,\n",
    "\n",
    "* Classification neural networks have an output neuron count equal to the number of classes.\n",
    "* Classification neural networks should use **categorical_crossentropy** and a **softmax** activation function on the output layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate X and Y for a Regression Neural Network\n",
    "\n",
    "For a regression neural network, the *x* values are generated the same.  However, *y* does not use dummies.  Make sure to replace **income** with your actual target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['income'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 4](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class1.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}