finding_donors_Feature_Engineering_00.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Donors for *CharityML*\n",
    "## Feature Engineering 00\n",
    "### Kebei Jiang 06/04/2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goal  \n",
    " * benchmark ft engineering: standard normalization and scaling, no discarding or regrouping  \n",
    " * EDA inspired ft engineering: with discarding and regrouping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries necessary for this project\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from time import time\n",
    "from IPython.display import display # Allows the use of display() for DataFrames\n",
    "\n",
    "# Import libraries for visulization\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "#plt.style.use('ggplot')\n",
    "%matplotlib inline\n",
    "\n",
    "sns.set(color_codes=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the Census dataset\n",
    "data = pd.read_csv(\"census.csv\")\n",
    "ft_num = data.select_dtypes(include=['int64','float64']).columns.values\n",
    "ft_cat = data.select_dtypes(exclude=['int64','float64']).columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1st round EDA decisions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| feature-numerical | 1st round decision        |\n",
    "|-------------------|---------------------------|\n",
    "| age               | as-is                     |\n",
    "| education-num     | as-is                     |\n",
    "| capital-gain      | divide into zero/non-zero |\n",
    "| capital-loss      | divide into zero/non-zero |\n",
    "| hours-per-week    | as-is                     |\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| feature-categorical | 1st round decision                                                                                         | reasoning                                                         |\n",
    "|---------------------|------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|\n",
    "| workclass           | {without-pay}, {*-gov, private, self-emp-not-inc}, {self-emp-inc}                                          | '-gov', 'private' and 'self-emp-not-inc' are all paid employees   |\n",
    "| occupation          | {Exec-managerial, prof-specialty}, {farming, handlers, machine-op, other-service, priv-house-serv}, {rest} | based on income classes ratio                                     |\n",
    "| marital-status      | {Married-AF-spouse, Married-civ-spouse}, {rest}                                                            |                                                                   |\n",
    "| relationship        | {Husband, Wife}, {rest}                                                                                    | correlation with marital-status, should include both or just one? |\n",
    "| race                | {Asian-Pac-Islander, white}, {rest}                                                                        | based on income classes ratio                                     |\n",
    "| sex                 | as-is                                                                                                      |                                                                   |\n",
    "| native-country      | drop                                                                                                       | just assume everyone is from US                                   |\n",
    "| education-level      | drop                                                                                                       | duplicated                                   |\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " [ref 1](http://scg.sdsu.edu/dataset-adult_r/)\n",
    " * Capital-gain/loss into low/high groups\n",
    " * combine government works; self-employed...  \n",
    " * 'occupation' to 'blue collar' and 'white collar'  \n",
    " * 'native-cournty' into continents  \n",
    " * scaling/normalizing features  \n",
    " * put all feature engineering into a function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[ref 2](https://faculty.biu.ac.il/~yahavi1/Projects/CP2010T1_rep.pdf)  \n",
    " * visualize DT  \n",
    " * average hours-per-week w.r.t. Gender (married or not)  \n",
    " * check predictive error in different classes  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[ref3](http://rstudio-pubs-static.s3.amazonaws.com/265200_a8d21a65d3d34b979c5aafb0de10c221.html)  \n",
    "Capital gain:\n",
    "\n",
    "We mark all values of “capital_gain” which are less than the first quartile of the nonzero capital gain (which is equal to 3464) as “Low”; all values that are between the first and third quartile (between 3464 and 14080) - as “Medium”; and all values greater than or equal to the third quartile are marked as “High”.\n",
    "\n",
    "\n",
    "Asia_East <- c(\" Cambodia\", \" China\", \" Hong\", \" Laos\", \" Thailand\",\n",
    "               \" Japan\", \" Taiwan\", \" Vietnam\")\n",
    "\n",
    "Asia_Central <- c(\" India\", \" Iran\")\n",
    "\n",
    "Central_America <- c(\" Cuba\", \" Guatemala\", \" Jamaica\", \" Nicaragua\", \n",
    "                     \" Puerto-Rico\",  \" Dominican-Republic\", \" El-Salvador\", \n",
    "                     \" Haiti\", \" Honduras\", \" Mexico\", \" Trinadad&Tobago\")\n",
    "\n",
    "South_America <- c(\" Ecuador\", \" Peru\", \" Columbia\")\n",
    "\n",
    "\n",
    "Europe_West <- c(\" England\", \" Germany\", \" Holand-Netherlands\", \" Ireland\", \n",
    "                 \" France\", \" Greece\", \" Italy\", \" Portugal\", \" Scotland\")\n",
    "\n",
    "Europe_East <- c(\" Poland\", \" Yugoslavia\", \" Hungary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "def ft_num_engineer(data, ft_num):\n",
    "\n",
    "    # logrithmic transform on 'capital-gain' and 'capital-loss'\n",
    "    data['capital-gain']=np.log(data['capital-gain'] + 1)\n",
    "    data['capital-loss']=np.log(data['capital-loss'] + 1)\n",
    "    \n",
    "    # scaling the features\n",
    "    # scaling works on multiple featurs simultaneously\n",
    "    scaler = MinMaxScaler()\n",
    "    data[ft_num] = scaler.fit_transform(data[ft_num])\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to apply EAD inspired feature engineering on CATEGORICAL features\n",
    "# When 'True' the EDA was applied\n",
    "\n",
    "def ft_cat_eda(data, workclass=False, occupation=False, marital=False, relationship=False, race=False):\n",
    "\n",
    "    # workclass\n",
    "    workclass_dict = {' Without-pay':'without-pay',' State-gov':'employee', ' Federal-gov':'employee', ' Local-gov':'employee', \\\n",
    "                      ' Private':'employee', ' Self-emp-not-inc':'employee', ' Self-emp-inc':'owner'}\n",
    "    # occupation\n",
    "    occupation_income = pd.Series([0, 1, 1, 2, 0, 0, 0, 0, 0, 2, 1, 1, 1, 1]).map({0:'low', 1:'mid', 2:'high'})\n",
    "    occupation_dict = dict(zip(sorted(data['occupation'].unique()), occupation_income))\n",
    "    # marital-status\n",
    "    marital_group = pd.Series([1 if x in [' Married-civ-spouse', ' Married-AF-spouse'] else 0 for x in data['marital-status'].unique()]).map({0:'single', 1:'couple'})\n",
    "    marital_dict = dict(zip(data['marital-status'].unique(), marital_group))\n",
    "    # relationship\n",
    "    relationship_group = pd.Series([1 if x in [' Husband', ' Wife'] else 0 for x in data['relationship'].unique()]).map({0:'single', 1:'couple'})\n",
    "    relationship_dict = dict(zip(data['relationship'].unique(), relationship_group))\n",
    "    # race\n",
    "    race_dict = {' White': 'high', ' Asian-Pac-Islander': 'high', ' Black':'low', ' Amer-Indian-Eskimo':'low', ' Other':'low'}\n",
    "    \n",
    "    # replacement\n",
    "    filter = np.array([workclass, occupation, marital, relationship, race])\n",
    "    fts = np.array(['workclass', 'occupation', 'marital-status', 'relationship', 'race'])\n",
    "    dicts = np.array([workclass_dict, occupation_dict, marital_dict, relationship_dict, race_dict])\n",
    "    \n",
    "    replace_dict = dict(zip(fts[filter], dicts[filter]))\n",
    "\n",
    "    data = data.replace(replace_dict)\n",
    "    \n",
    "    return data\n",
    "\n",
    "def ft_engineer(data, capital=False, workclass=False, occupation=False, marital=False, \\\n",
    "                    relationship=False, race=False, drop_native_country=False):\n",
    "\n",
    "    # numerical engineering\n",
    "    ft_num = data.select_dtypes(include=['int64','float64']).columns.values    \n",
    "    data = ft_num_engineer(data, ft_num)\n",
    "      \n",
    "    # should we binarize 'capital'\n",
    "    if capital:\n",
    "        data['capital-gain']= data['capital-gain'].apply(lambda x: 'no' if x==0 else 'yes')\n",
    "        data['capital-loss']= data['capital-loss'].apply(lambda x: 'no' if x==0 else 'yes')\n",
    "    \n",
    "    # EDA suggested update\n",
    "    data = ft_cat_eda(data, workclass, occupation, marital, relationship, race)\n",
    "    \n",
    "    # should we drop 'native-country'\n",
    "    if drop_native_country:\n",
    "        data.drop('native-country', axis=1, inplace=True)\n",
    "\n",
    "    # target and get_dummies\n",
    "    target = np.array(data['income'] != '<=50K').astype(int)\n",
    "    \n",
    "    #data.drop(['education_level', 'income'], axis=1, inplace=True)\n",
    "    data.drop(['income'], axis=1, inplace=True)\n",
    "    data = pd.get_dummies(data)\n",
    "    \n",
    "    return data, target\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminary Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import train_test_split\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split the 'features' and 'income' data into training and testing sets\n",
    "def xy_split(data, target, random_state):\n",
    "    X_train, X_test, y_train, y_test = train_test_split(data, \n",
    "                                                    target, \n",
    "                                                    test_size = 0.2, \n",
    "                                                    random_state = random_state)\n",
    "    # Show the results of the split\n",
    "    print(\"Training set has {} samples.\".format(X_train.shape[0]))\n",
    "    print(\"Testing set has {} samples.\".format(X_test.shape[0]))\n",
    "\n",
    "    return X_train, X_test, y_train, y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import fbeta_score\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import precision_recall_fscore_support, classification_report, confusion_matrix\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier\n",
    "\n",
    "def train_predict(data, target, clf, beta):\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = xy_split(data, target, 0)\n",
    "    \n",
    "    clf = clf\n",
    "    clf_name = clf.__class__.__name__\n",
    "    \n",
    "    clf.fit(X_train, y_train)\n",
    "    y_pred = clf.predict(X_test)\n",
    "    \n",
    "    cm = confusion_matrix(y_test, y_pred)\n",
    "    print('\\nThe confusion matrix looks like: ')\n",
    "    print(cm)\n",
    "    #sns.heatmap(cm, annot=True, fmt='.2g')\n",
    "    \n",
    "    print('\\nThe classification report looks like: ')\n",
    "    print(classification_report(y_test, y_pred))\n",
    "\n",
    "    print(\"Final F-score with beta={} on the testing data: {:.4f}\".format(beta, fbeta_score(y_test, y_pred, beta=beta))) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Kebei\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\preprocessing\\data.py:323: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by MinMaxScaler.\n",
      "  return self.partial_fit(X, y)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(45222, 103)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# when all 'False', no categorical EDA was applied\n",
    "# this serves as the benchmark\n",
    "\n",
    "tmp, target = ft_engineer(data.copy(), capital=False, workclass=False, occupation=False, \n",
    "                          marital=False, relationship=False, race=False, drop_native_country=False)\n",
    "\n",
    "tmp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set has 36177 samples.\n",
      "Testing set has 9045 samples.\n",
      "\n",
      "The confusion matrix looks like: \n",
      "[[6301  539]\n",
      " [ 891 1314]]\n",
      "\n",
      "The classification report looks like: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.92      0.90      6840\n",
      "           1       0.71      0.60      0.65      2205\n",
      "\n",
      "   micro avg       0.84      0.84      0.84      9045\n",
      "   macro avg       0.79      0.76      0.77      9045\n",
      "weighted avg       0.84      0.84      0.84      9045\n",
      "\n",
      "Final F-score with beta=0.5 on the testing data: 0.6832\n"
     ]
    }
   ],
   "source": [
    "train_predict(tmp, target, LogisticRegression(solver='liblinear'), 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**  \n",
    " * the recall on '>50K' is low. Does it make sense to set $\\beta=0.5$?  \n",
    " * how to incorporate the unbalance between two classes?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## metric selection\n",
    "\n",
    "The tutorial used **F-beta score** where   \n",
    " * precision, what proportion of people classified as '>50K' actually makes more than 50K, rather than  \n",
    " * recall, what proportion of people that makes more than 50K actually get classified as '>50K'.  \n",
    "\n",
    "Wth $\\beta = 0.5$, which places more emphasis on precision, we are not asking the right question. It's OK to wrongly classify someone as '>50K' -- all you gonna lose is a phone call or a pamphlet; on the other hand, we do want to identify as many people makes more than 50K as possible, for they are the ones likely to donate. Therefore I am **changing the $\\beta$ value to 2.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set has 36177 samples.\n",
      "Testing set has 9045 samples.\n",
      "\n",
      "The confusion matrix looks like: \n",
      "[[6301  539]\n",
      " [ 891 1314]]\n",
      "\n",
      "The classification report looks like: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.92      0.90      6840\n",
      "           1       0.71      0.60      0.65      2205\n",
      "\n",
      "   micro avg       0.84      0.84      0.84      9045\n",
      "   macro avg       0.79      0.76      0.77      9045\n",
      "weighted avg       0.84      0.84      0.84      9045\n",
      "\n",
      "Final F-score with beta=2 on the testing data: 0.6156\n"
     ]
    }
   ],
   "source": [
    "train_predict(tmp, target, LogisticRegression(solver='liblinear'), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion   \n",
    " * More emphasis should be placed on Recall. \n",
    " * **the negative class has a very high precision and recall.** Could this be related to the unbalanced population between these classes?  \n",
    " * Further testing is required."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}