t81_558_class_02_2_pandas_cat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_02_2_pandas_cat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 2: Python for Machine Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 2 Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* Part 2.1: Introduction to Pandas [[Video]](https://www.youtube.com/watch?v=bN4UuCBdpZc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_1_python_pandas.ipynb)\n",
    "* **Part 2.2: Categorical Values** [[Video]](https://www.youtube.com/watch?v=4a1odDpG0Ho&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_2_pandas_cat.ipynb)\n",
    "* Part 2.3: Grouping, Sorting, and Shuffling in Python Pandas [[Video]](https://www.youtube.com/watch?v=YS4wm5gD8DM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_3_pandas_grouping.ipynb)\n",
    "* Part 2.4: Using Apply and Map in Pandas for Keras [[Video]](https://www.youtube.com/watch?v=XNCEZ4WaPBY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_4_pandas_functional.ipynb)\n",
    "* Part 2.5: Feature Engineering in Pandas for Deep Learning in Keras [[Video]](https://www.youtube.com/watch?v=BWPTj4_Mi9E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_5_pandas_features.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2.2: Categorical and Continuous Values\n",
    "\n",
    "Neural networks require their input to be a fixed number of columns.  This input format is very similar to spreadsheet data.  This input must be entirely numeric.  \n",
    "\n",
    "It is essential to represent the data in a way that the neural network can train from it.  In class 6, we will see even more ways to preprocess data.  For now, we will look at several of the most basic ways to transform data for a neural network.\n",
    "\n",
    "Before we look at specific ways to preprocess data, it is important to consider four basic types of data, as defined by [[Cite:stevens1946theory]](http://psychology.okstate.edu/faculty/jgrice/psyc3214/Stevens_FourScales_1946.pdf).  Statisticians commonly refer to as the [levels of measure](https://en.wikipedia.org/wiki/Level_of_measurement):\n",
    "\n",
    "* Character Data (strings)\n",
    "    * **Nominal** - Individual discrete items, no order. For example, color, zip code, shape.\n",
    "    * **Ordinal** - Individual distinct items have an implied order.  For example grade level, job title, Starbucks(tm) coffee size (tall, vente, grande) \n",
    "* Numeric Data\n",
    "    * **Interval** - Numeric values, no defined start.  For example, temperature.  You would never say, \"yesterday was twice as hot as today.\"\n",
    "    * **Ratio** - Numeric values, clearly defined start.  For example, speed.  You would say that \"The first car is going twice as fast as the second.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Continuous Values\n",
    "\n",
    "One common transformation is to normalize the inputs.  It is sometimes valuable to normalization numeric inputs to be put in a standard form so that the program can easily compare these two values.  Consider if a friend told you that he received a 10 dollar discount.  Is this a good deal?  Maybe.  But the cost is not normalized.  If your friend purchased a car, then the discount is not that good.  If your friend bought dinner, this is an excellent discount!\n",
    "\n",
    "Percentages are a prevalent form of normalization.  If your friend tells you they got 10% off, we know that this is a better discount than 5%.  It does not matter how much the purchase price was.  One widespread machine learning normalization is the Z-Score:\n",
    "\n",
    "$z = \\frac{x - \\mu}{\\sigma} $\n",
    "\n",
    "To calculate the Z-Score you need to also calculate the mean($\\mu$) and the standard deviation ($\\sigma$).  The mean is calculated as follows:\n",
    "\n",
    "$\\mu = \\bar{x} = \\frac{x_1+x_2+\\cdots +x_n}{n}$\n",
    "\n",
    "The standard deviation is calculated as follows:\n",
    "\n",
    "$\\sigma = \\sqrt{\\frac{1}{N} \\sum_{i=1}^N (x_i - \\mu)^2}, {\\rm \\ \\ where\\ \\ } \\mu = \\frac{1}{N} \\sum_{i=1}^N x_i$\n",
    "\n",
    "The following Python code replaces the mpg with a z-score.  Cars with average MPG will be near zero, above zero is above average, and below zero is below average.  Z-Scores above/below -3/3 are very rare, these are outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>...</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>...</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>...</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>0.574601</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>...</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>0.958913</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>...</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          mpg  cylinders  displacement  ...  year  origin  \\\n",
       "0   -0.706439          8         307.0  ...    70       1   \n",
       "1   -1.090751          8         350.0  ...    70       1   \n",
       "..        ...        ...           ...  ...   ...     ...   \n",
       "396  0.574601          4         120.0  ...    82       1   \n",
       "397  0.958913          4         119.0  ...    82       1   \n",
       "\n",
       "                          name  \n",
       "0    chevrolet chevelle malibu  \n",
       "1            buick skylark 320  \n",
       "..                         ...  \n",
       "396                ford ranger  \n",
       "397                 chevy s-10  \n",
       "\n",
       "[398 rows x 9 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "pd.set_option('display.max_columns', 7)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "df['mpg'] = zscore(df['mpg'])\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Categorical Values as Dummies\n",
    "The traditional means of encoding categorical values is to make them dummy variables.  This technique is also called one-hot-encoding.  Consider the following data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>...</th>\n",
       "      <th>retail_dense</th>\n",
       "      <th>crime</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.492126</td>\n",
       "      <td>0.071100</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.342520</td>\n",
       "      <td>0.400809</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.598425</td>\n",
       "      <td>0.117803</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.539370</td>\n",
       "      <td>0.451973</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job area  ...  retail_dense     crime  product\n",
       "0        1  vv    c  ...      0.492126  0.071100        b\n",
       "1        2  kd    c  ...      0.342520  0.400809        c\n",
       "...    ...  ..  ...  ...           ...       ...      ...\n",
       "1998  1999  qp    c  ...      0.598425  0.117803        c\n",
       "1999  2000  pe    c  ...      0.539370  0.451973        c\n",
       "\n",
       "[2000 rows x 14 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "pd.set_option('display.max_columns', 7)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of areas: 4\n",
      "Areas: ['c', 'd', 'a', 'b']\n"
     ]
    }
   ],
   "source": [
    "areas = list(df['area'].unique())\n",
    "print(f'Number of areas: {len(areas)}')\n",
    "print(f'Areas: {areas}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four unique values in the areas column.  To encode these to dummy variables, we would use four columns, each of which would represent one of the areas.  For each row, one column would have a value of one, the rest zeros.  For this reason, this type of encoding is sometimes called one-hot encoding.  The following code shows how you might encode the values \"a\" through \"d.\"  The value A becomes [1,0,0,0] and the value B becomes [0,1,0,0]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   area_a  area_b  area_c  area_d\n",
      "0       1       0       0       0\n",
      "1       0       1       0       0\n",
      "2       0       0       1       0\n",
      "3       0       0       0       1\n"
     ]
    }
   ],
   "source": [
    "dummies = pd.get_dummies(['a','b','c','d'],prefix='area')\n",
    "print(dummies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    area_a  area_b  area_c  area_d\n",
      "0        0       0       1       0\n",
      "1        0       0       1       0\n",
      "..     ...     ...     ...     ...\n",
      "8        0       0       1       0\n",
      "9        1       0       0       0\n",
      "\n",
      "[10 rows x 4 columns]\n"
     ]
    }
   ],
   "source": [
    "dummies = pd.get_dummies(df['area'],prefix='area')\n",
    "print(dummies[0:10]) # Just show the first 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([df,dummies],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To encode the \"area\" column, we use the following. Note that it is necessary to merge these dummies back into the data frame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>c</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>c</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job area   income  area_a  area_b  area_c  area_d\n",
       "0        1  vv    c  50876.0       0       0       1       0\n",
       "1        2  kd    c  60369.0       0       0       1       0\n",
       "2        3  pe    c  55126.0       0       0       1       0\n",
       "3        4  11    c  51690.0       0       0       1       0\n",
       "4        5  kl    d  28347.0       0       0       0       1\n",
       "...    ...  ..  ...      ...     ...     ...     ...     ...\n",
       "1995  1996  vv    c  51017.0       0       0       1       0\n",
       "1996  1997  kl    d  26576.0       0       0       0       1\n",
       "1997  1998  kl    d  28595.0       0       0       0       1\n",
       "1998  1999  qp    c  67949.0       0       0       1       0\n",
       "1999  2000  pe    c  61467.0       0       0       1       0\n",
       "\n",
       "[2000 rows x 8 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 10)\n",
    "\n",
    "display(df[['id','job','area','income','area_a',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, you will remove the original column ('area'), because it is the goal to get the data frame to be entirely numeric for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>income</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job   income  area_a  area_b  area_c  area_d\n",
       "0        1  vv  50876.0       0       0       1       0\n",
       "1        2  kd  60369.0       0       0       1       0\n",
       "...    ...  ..      ...     ...     ...     ...     ...\n",
       "1998  1999  qp  67949.0       0       0       1       0\n",
       "1999  2000  pe  61467.0       0       0       1       0\n",
       "\n",
       "[2000 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "display(df[['id','job','income','area_a',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target Encoding for Categoricals\n",
    "\n",
    "Target encoding can sometimes increase the predictive power of a machine learning model.  However, it also dramatically increases the risk of overfitting.  Because of this risk, you must take care if you are using this method.  Target encoding is a popular technique for Kaggle competitions.  \n",
    "\n",
    "Generally, target encoding can only be used on a categorical feature when the output of the machine learning model is numeric (regression).\n",
    "\n",
    "The concept of target encoding is straightforward.  For each category, we calculate the average target value for that category.  Then to encode, we substitute the percent that corresponds to the category that the categorical value has.  Unlike dummy variables, where you have a column for each category, with target encoding, the program only needs a single column.  In this way, target coding is more efficient than dummy variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cont_9</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.505457</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60.906654</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.339096</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.058962</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32.713906</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>85.913749</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>66.609021</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>54.116221</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.901382</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73.374830</td>\n",
       "      <td>cat</td>\n",
       "      <td>tiger</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cont_9 cat_0  cat_1  y\n",
       "0  11.505457   dog   wolf  1\n",
       "1  60.906654   dog   wolf  0\n",
       "2  13.339096   dog   wolf  1\n",
       "3  24.058962   dog   wolf  1\n",
       "4  32.713906   dog   wolf  1\n",
       "5  85.913749   cat   wolf  1\n",
       "6  66.609021   cat   wolf  0\n",
       "7  54.116221   cat   wolf  0\n",
       "8   2.901382   cat   wolf  0\n",
       "9  73.374830   cat  tiger  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a small sample dataset\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(43)\n",
    "df = pd.DataFrame({\n",
    "    'cont_9': np.random.rand(10)*100,\n",
    "    'cat_0': ['dog'] * 5 + ['cat'] * 5,\n",
    "    'cat_1': ['wolf'] * 9 + ['tiger'] * 1,\n",
    "    'y': [1, 0, 1, 1, 1, 1, 0, 0, 0, 0]\n",
    "})\n",
    "\n",
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 0)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than creating dummy variables for \"dog\" and \"cat,\" we would like to change it to a number. We could use 0 for cat, 1 for dog.  However, we can encode more information than just that.  The simple 0 or 1 would also only work for one animal.  Consider what the mean target value is for cat and dog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'cat': 0.2, 'dog': 0.8}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "means0 = df.groupby('cat_0')['y'].mean().to_dict()\n",
    "means0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The danger is that we are now using the target value for training.  This technique will potentially lead to overfitting.  The possibility of overfitting is even greater if there are a small number of a particular category.  To prevent this from happening, we use a weighting factor.  The stronger the weight, the more than categories with a small number of values will tend towards the overall average of y. You can perform this calculation as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['y'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can implement target encoding as follows.  For more information on Target Encoding, refer to the article [\"Target Encoding Done the Right Way\"](https://maxhalford.github.io/blog/target-encoding-done-the-right-way/), that I based this code upon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_smooth_mean(df1, df2, cat_name, target, weight):\n",
    "    # Compute the global mean\n",
    "    mean = df[target].mean()\n",
    "\n",
    "    # Compute the number of values and the mean of each group\n",
    "    agg = df.groupby(cat_name)[target].agg(['count', 'mean'])\n",
    "    counts = agg['count']\n",
    "    means = agg['mean']\n",
    "\n",
    "    # Compute the \"smoothed\" means\n",
    "    smooth = (counts * means + weight * mean) / (counts + weight)\n",
    "\n",
    "    # Replace each value by the according smoothed mean\n",
    "    if df2 is None:\n",
    "        return df1[cat_name].map(smooth)\n",
    "    else:\n",
    "        return df1[cat_name].map(smooth),df2[cat_name].map(smooth.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code encodes these two categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cont_9</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>y</th>\n",
       "      <th>cat_0_enc</th>\n",
       "      <th>cat_1_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.505457</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60.906654</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.339096</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.058962</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32.713906</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>85.913749</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>66.609021</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>54.116221</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.901382</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73.374830</td>\n",
       "      <td>cat</td>\n",
       "      <td>tiger</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cont_9 cat_0  cat_1  y  cat_0_enc  cat_1_enc\n",
       "0  11.505457   dog   wolf  1       0.65   0.535714\n",
       "1  60.906654   dog   wolf  0       0.65   0.535714\n",
       "2  13.339096   dog   wolf  1       0.65   0.535714\n",
       "3  24.058962   dog   wolf  1       0.65   0.535714\n",
       "4  32.713906   dog   wolf  1       0.65   0.535714\n",
       "5  85.913749   cat   wolf  1       0.35   0.535714\n",
       "6  66.609021   cat   wolf  0       0.35   0.535714\n",
       "7  54.116221   cat   wolf  0       0.35   0.535714\n",
       "8   2.901382   cat   wolf  0       0.35   0.535714\n",
       "9  73.374830   cat  tiger  0       0.35   0.416667"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "WEIGHT = 5\n",
    "df['cat_0_enc'] = calc_smooth_mean(df1=df, df2=None, \n",
    "    cat_name='cat_0', target='y', weight=WEIGHT)\n",
    "df['cat_1_enc'] = calc_smooth_mean(df1=df, df2=None, \n",
    "    cat_name='cat_1', target='y', weight=WEIGHT)\n",
    "\n",
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 0)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Categorical Values as Ordinal\n",
    "\n",
    "Typically categoricals will be encoded as dummy variables.  However, there might be other techniques to convert categoricals to numeric. Any time there is an order to the categoricals, a number should be used.  Consider if you had a categorical that described the current education level of an individual.   \n",
    "\n",
    "* Kindergarten (0)\n",
    "* First Grade (1)\n",
    "* Second Grade (2)\n",
    "* Third Grade (3)\n",
    "* Fourth Grade (4)\n",
    "* Fifth Grade (5)\n",
    "* Sixth Grade (6)\n",
    "* Seventh Grade (7)\n",
    "* Eighth Grade (8)\n",
    "* High School Freshman (9)\n",
    "* High School Sophomore (10)\n",
    "* High School Junior (11)\n",
    "* High School Senior (12)\n",
    "* College Freshman (13)\n",
    "* College Sophomore (14)\n",
    "* College Junior (15)\n",
    "* College Senior (16)\n",
    "* Graduate Student (17)\n",
    "* PhD Candidate (18)\n",
    "* Doctorate (19)\n",
    "* Post Doctorate (20)\n",
    "\n",
    "The above list has 21 levels.  This would take 21 dummy variables. However, simply encoding this to dummies would lose the order information.  Perhaps the easiest approach would be to assign simply number them and assign the category a single number that is equal to the value in parenthesis above.  However, we might be able to do even better.  Graduate student is likely more than a year, so you might increase more than just one value.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}