4_TelecomChurnCaseStudy_Modelling_LASSO_DT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color='blue'>Telecom Churn Case Study</font>\n",
    "* Institution: IIIT, Bangalore and UpGrad\n",
    "* Course: PG Diploma in Machine Lerning and AI March 2018\n",
    "* Date: 14-Aug-2018\n",
    "* Submitted by:\n",
    "    1. Pandinath Siddineni (ID- APFE187000194)\n",
    "    2. AKNR Chandra Sekhar (ID- APFE187000315)\n",
    "    3. Brajesh Kumar       (ID- APFE187000149)\n",
    "    4. Shweta Tiwari\n",
    "-----------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color='blue'>PART 3: LASSO & DECISSION TREE</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os.path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import datetime as dt\n",
    "\n",
    "pd.options.display.float_format = '{:.2f}'.format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mobile_number</th>\n",
       "      <th>arpu_6</th>\n",
       "      <th>arpu_7</th>\n",
       "      <th>arpu_8</th>\n",
       "      <th>onnet_mou_6</th>\n",
       "      <th>onnet_mou_7</th>\n",
       "      <th>onnet_mou_8</th>\n",
       "      <th>offnet_mou_6</th>\n",
       "      <th>offnet_mou_7</th>\n",
       "      <th>offnet_mou_8</th>\n",
       "      <th>...</th>\n",
       "      <th>fb7_1.0</th>\n",
       "      <th>fb8_0.0</th>\n",
       "      <th>fb8_1.0</th>\n",
       "      <th>total_rech_data_amt_6</th>\n",
       "      <th>total_rech_data_amt_7</th>\n",
       "      <th>total_rech_data_amt_8</th>\n",
       "      <th>churn</th>\n",
       "      <th>rech_days_left_6</th>\n",
       "      <th>rech_days_left_7</th>\n",
       "      <th>rech_days_left_8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7000701601</td>\n",
       "      <td>1069.18</td>\n",
       "      <td>1349.85</td>\n",
       "      <td>3171.48</td>\n",
       "      <td>57.84</td>\n",
       "      <td>54.68</td>\n",
       "      <td>52.29</td>\n",
       "      <td>453.43</td>\n",
       "      <td>567.16</td>\n",
       "      <td>325.91</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>3.00</td>\n",
       "      <td>6.00</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7001524846</td>\n",
       "      <td>378.72</td>\n",
       "      <td>492.22</td>\n",
       "      <td>137.36</td>\n",
       "      <td>413.69</td>\n",
       "      <td>351.03</td>\n",
       "      <td>35.08</td>\n",
       "      <td>94.66</td>\n",
       "      <td>80.63</td>\n",
       "      <td>136.48</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>354.00</td>\n",
       "      <td>207.00</td>\n",
       "      <td>0</td>\n",
       "      <td>5.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7002124215</td>\n",
       "      <td>514.45</td>\n",
       "      <td>597.75</td>\n",
       "      <td>637.76</td>\n",
       "      <td>102.41</td>\n",
       "      <td>132.11</td>\n",
       "      <td>85.14</td>\n",
       "      <td>757.93</td>\n",
       "      <td>896.68</td>\n",
       "      <td>983.39</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7000887461</td>\n",
       "      <td>74.35</td>\n",
       "      <td>193.90</td>\n",
       "      <td>366.97</td>\n",
       "      <td>48.96</td>\n",
       "      <td>50.66</td>\n",
       "      <td>33.58</td>\n",
       "      <td>85.41</td>\n",
       "      <td>89.36</td>\n",
       "      <td>205.89</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>712.00</td>\n",
       "      <td>540.00</td>\n",
       "      <td>0</td>\n",
       "      <td>12.00</td>\n",
       "      <td>24.00</td>\n",
       "      <td>7.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7000149764</td>\n",
       "      <td>977.02</td>\n",
       "      <td>2362.83</td>\n",
       "      <td>409.23</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5285.00</td>\n",
       "      <td>20424.00</td>\n",
       "      <td>455.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 144 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mobile_number  arpu_6  arpu_7  arpu_8  onnet_mou_6  onnet_mou_7  \\\n",
       "0     7000701601 1069.18 1349.85 3171.48        57.84        54.68   \n",
       "1     7001524846  378.72  492.22  137.36       413.69       351.03   \n",
       "2     7002124215  514.45  597.75  637.76       102.41       132.11   \n",
       "3     7000887461   74.35  193.90  366.97        48.96        50.66   \n",
       "4     7000149764  977.02 2362.83  409.23         0.00         0.00   \n",
       "\n",
       "   onnet_mou_8  offnet_mou_6  offnet_mou_7  offnet_mou_8        ...         \\\n",
       "0        52.29        453.43        567.16        325.91        ...          \n",
       "1        35.08         94.66         80.63        136.48        ...          \n",
       "2        85.14        757.93        896.68        983.39        ...          \n",
       "3        33.58         85.41         89.36        205.89        ...          \n",
       "4         0.00          0.00          0.00          0.00        ...          \n",
       "\n",
       "   fb7_1.0  fb8_0.0  fb8_1.0  total_rech_data_amt_6  total_rech_data_amt_7  \\\n",
       "0        0        0        0                   0.00                   0.00   \n",
       "1        1        0        1                   0.00                 354.00   \n",
       "2        0        0        0                   0.00                   0.00   \n",
       "3        1        0        1                   0.00                 712.00   \n",
       "4        1        0        1                5285.00               20424.00   \n",
       "\n",
       "   total_rech_data_amt_8  churn  rech_days_left_6  rech_days_left_7  \\\n",
       "0                   0.00      1              3.00              6.00   \n",
       "1                 207.00      0              5.00              0.00   \n",
       "2                   0.00      0              0.00              0.00   \n",
       "3                 540.00      0             12.00             24.00   \n",
       "4                 455.00      0              0.00              1.00   \n",
       "\n",
       "   rech_days_left_8  \n",
       "0              5.00  \n",
       "1              1.00  \n",
       "2              0.00  \n",
       "3              7.00  \n",
       "4              5.00  \n",
       "\n",
       "[5 rows x 144 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load clean telecom data file\n",
    "master_df = pd.read_csv('telecom_churn_data_clean.csv', low_memory=False)\n",
    "master_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframe Shape:  (28504, 144)\n",
      "Dataframe Info: \n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 28504 entries, 0 to 28503\n",
      "Columns: 144 entries, mobile_number to rech_days_left_8\n",
      "dtypes: float64(105), int64(39)\n",
      "memory usage: 31.3 MB\n"
     ]
    }
   ],
   "source": [
    "print('Dataframe Shape: ', master_df.shape)\n",
    "print(\"Dataframe Info: \\n\"); master_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop MemberID/Phone-number\n",
    "telecom = master_df.drop(['mobile_number'], axis=1)\n",
    "\n",
    "# Create X (independent variable) & y (dependent variable) \n",
    "df_telecom = telecom.drop(['churn'], axis=1)\n",
    "X = telecom.drop(['churn'], axis=1)\n",
    "y = telecom['churn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28504, 144)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#list(master_df)\n",
    "master_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Standardization/Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "\n",
    "scaler = preprocessing.StandardScaler().fit(X)\n",
    "X = scaler.transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Data into Train & Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Split in train & Test\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, train_size=0.7, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train Dataframe Shape (19952, 142)\n",
      "X_test Dataframe Shape (8552, 142)\n",
      "Imbalance in Train Data: 0.05941698083151914\n",
      "Imbalance in Test Data: 0.059071207430340555\n"
     ]
    }
   ],
   "source": [
    "print(\"X_train Dataframe Shape {}\".format(X_train.shape))\n",
    "print(\"X_test Dataframe Shape {}\".format(X_test.shape))\n",
    "\n",
    "y_train_imb = (y_train != 0).sum()/(y_train == 0).sum()\n",
    "y_test_imb = (y_test != 0).sum()/(y_test == 0).sum()\n",
    "print(\"Imbalance in Train Data: {}\".format(y_train_imb))\n",
    "print(\"Imbalance in Test Data: {}\".format(y_test_imb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Balance data set by oversampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# (Training) Balance Data-Set --- SMOT\n",
    "from imblearn.over_sampling import SMOTE\n",
    "\n",
    "sm = SMOTE(kind = \"regular\")\n",
    "X_tr,y_tr = sm.fit_sample(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_tr Dataframe Shape (37666, 142)\n",
      "y_tr Dataframe Shape (37666,)\n",
      "Imbalance in Train Data: 1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"X_tr Dataframe Shape {}\".format(X_tr.shape))\n",
    "print(\"y_tr Dataframe Shape {}\".format(y_tr.shape))\n",
    "\n",
    "data_imbalance = (y_tr != 0).sum()/(y_tr == 0).sum()\n",
    "print(\"Imbalance in Train Data: {}\".format(data_imbalance))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature reduction using LASSO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(37666, 47)\n",
      "[  0   1   3  10  13  14  20  23  29  34  39  40  47  49  52  53  58  65\n",
      "  78  79  80  83  91  92  95  98 101 102 104 107 108 109 110 111 112 113\n",
      " 116 118 120 121 125 128 132 135 139 140 141]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    " \n",
    "lsvc = LinearSVC(C=0.001, penalty=\"l1\", dual=False).fit(X_tr, y_tr)\n",
    "model = SelectFromModel(lsvc, prefit=True)\n",
    "X_lasso = model.transform(X_tr)\n",
    "pos = model.get_support(indices=True)\n",
    " ### Feature reduction using RFE\n",
    "print(X_lasso.shape)\n",
    "print(pos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features identified by LASSO for model buidling:  ['arpu_6', 'arpu_7', 'onnet_mou_6', 'roam_ic_mou_7', 'roam_og_mou_7', 'roam_og_mou_8', 'loc_og_t2m_mou_8', 'loc_og_t2f_mou_8', 'loc_og_mou_8', 'std_og_t2m_mou_7', 'std_og_mou_6', 'std_og_mou_7', 'spl_og_mou_8', 'og_others_7', 'total_og_mou_7', 'total_og_mou_8', 'loc_ic_t2m_mou_7', 'loc_ic_mou_8', 'total_ic_mou_6', 'total_ic_mou_7', 'total_ic_mou_8', 'spl_ic_mou_8', 'total_rech_num_7', 'total_rech_num_8', 'total_rech_amt_8', 'max_rech_amt_8', 'last_day_rch_amt_8', 'vol_2g_mb_6', 'vol_2g_mb_8', 'vol_3g_mb_8', 'monthly_2g_6', 'monthly_2g_7', 'monthly_2g_8', 'sachet_2g_6', 'sachet_2g_7', 'sachet_2g_8', 'monthly_3g_8', 'sachet_3g_7', 'aon', 'aug_vbc_3g', 'night6_1.0', 'night8_0.0', 'fb7_0.0', 'fb8_1.0', 'rech_days_left_6', 'rech_days_left_7', 'rech_days_left_8']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#feature vector for decision tree#feature \n",
    "lasso_features = list(df_telecom.columns[pos])\n",
    "print(\"Features identified by LASSO for model buidling: \", lasso_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = X_lasso\n",
    "y_train = y_tr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature space holds 37666 observations and 47 features\n",
      "Unique target labels: [0 1]\n"
     ]
    }
   ],
   "source": [
    "print (\"Feature space holds %d observations and %d features\" % X_train.shape)\n",
    "print (\"Unique target labels:\", np.unique(y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Tree with default hyperparameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=5,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Importing decision tree classifier from sklearn library\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn import tree\n",
    "\n",
    "# Fitting the decision tree with default hyperparameters, apart from\n",
    "# max_depth which is 5 so that we can plot and read the tree.\n",
    "dt_default = DecisionTreeClassifier(max_depth=5)\n",
    "dt_default.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.98      0.87      0.92      8075\n",
      "          1       0.26      0.75      0.38       477\n",
      "\n",
      "avg / total       0.94      0.87      0.89      8552\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's check the evaluation metrics of our default model\n",
    "from sklearn.metrics import classification_report, confusion_matrix, accuracy_score\n",
    "\n",
    "# Making predictions\n",
    "X_test = pd.DataFrame(data=X_test).iloc[:, pos]\n",
    "y_pred_default = dt_default.predict(X_test)\n",
    "\n",
    "# Printing classification report\n",
    "print(classification_report(y_test, y_pred_default))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[7043 1032]\n",
      " [ 120  357]]\n",
      "0.865294667914\n"
     ]
    }
   ],
   "source": [
    "# Printing confusion matrix and accuracy\n",
    "print(confusion_matrix(y_test,y_pred_default))\n",
    "print(accuracy_score(y_test,y_pred_default))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning\n",
    "\n",
    "NOTE: \n",
    "1. Hyperparameter Tunning is commented as it takes heavy computing power and time. It can be run by uncommenting it.\n",
    "2. Getting 86% accuracy that looks to be pretty good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # GridSearchCV to find optimal max_depth\n",
    "# from sklearn.model_selection import KFold\n",
    "# from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "# # specify number of folds for k-fold CV\n",
    "# n_folds = 5\n",
    "\n",
    "# # parameters to build the model on\n",
    "# parameters = {'max_depth': range(1, 40)}\n",
    "\n",
    "# # instantiate the model\n",
    "# dtree = DecisionTreeClassifier(criterion = \"gini\", \n",
    "#                                random_state = 100)\n",
    "\n",
    "# # fit tree on training data\n",
    "# tree = GridSearchCV(dtree, parameters, \n",
    "#                     cv=n_folds, \n",
    "#                    scoring=\"accuracy\")\n",
    "# tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # scores of GridSearch CV\n",
    "# scores = tree.cv_results_\n",
    "# pd.DataFrame(scores).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # plotting accuracies with max_depth\n",
    "# plt.figure()\n",
    "# plt.plot(scores[\"param_max_depth\"], \n",
    "#          scores[\"mean_train_score\"], \n",
    "#          label=\"training accuracy\")\n",
    "# plt.plot(scores[\"param_max_depth\"], \n",
    "#          scores[\"mean_test_score\"], \n",
    "#          label=\"test accuracy\")\n",
    "# plt.xlabel(\"max_depth\")\n",
    "# plt.ylabel(\"Accuracy\")\n",
    "# plt.legend()\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "conclusion for max depth:\n",
    "You can see that as we increase the value of max_depth, both training and test score increase till about max-depth = 10, after which the test score is constant. Note that the scores are average accuracies across the 5-folds.\n",
    "\n",
    "we can consider max_depth=10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning min_samples_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # GridSearchCV to find optimal max_depth\n",
    "# from sklearn.model_selection import KFold\n",
    "# from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "# # specify number of folds for k-fold CV\n",
    "# n_folds = 5\n",
    "\n",
    "# # parameters to build the model on\n",
    "# parameters = {'min_samples_leaf': range(5, 200, 20)}\n",
    "\n",
    "# # instantiate the model\n",
    "# dtree = DecisionTreeClassifier(criterion = \"gini\", \n",
    "#                                random_state = 100)\n",
    "\n",
    "# # fit tree on training data\n",
    "# tree = GridSearchCV(dtree, parameters, \n",
    "#                     cv=n_folds, \n",
    "#                    scoring=\"accuracy\")\n",
    "# tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # scores of GridSearch CV\n",
    "# scores = tree.cv_results_\n",
    "# pd.DataFrame(scores).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # plotting accuracies with min_samples_leaf\n",
    "# plt.figure()\n",
    "# plt.plot(scores[\"param_min_samples_leaf\"], \n",
    "#          scores[\"mean_train_score\"], \n",
    "#          label=\"training accuracy\")\n",
    "# plt.plot(scores[\"param_min_samples_leaf\"], \n",
    "#          scores[\"mean_test_score\"], \n",
    "#          label=\"test accuracy\")\n",
    "# plt.xlabel(\"min_samples_leaf\")\n",
    "# plt.ylabel(\"Accuracy\")\n",
    "# plt.legend()\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "conclusion for min_samples_leaf:\n",
    "at low values of min_samples_leaf seems overfitted. At values 125,the model becomes more stable and the training and test accuracy start to converge.\n",
    "min_samples_leaf=125"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning min_samples_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # GridSearchCV to find optimal min_samples_split\n",
    "# from sklearn.model_selection import KFold\n",
    "# from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "# # specify number of folds for k-fold CV\n",
    "# n_folds = 5\n",
    "\n",
    "# # parameters to build the model on\n",
    "# parameters = {'min_samples_split': range(5, 200, 20)}\n",
    "\n",
    "# # instantiate the model\n",
    "# dtree = DecisionTreeClassifier(criterion = \"gini\", \n",
    "#                                random_state = 100)\n",
    "\n",
    "# # fit tree on training data\n",
    "# tree = GridSearchCV(dtree, parameters, \n",
    "#                     cv=n_folds, \n",
    "#                    scoring=\"accuracy\")\n",
    "# tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # scores of GridSearch CV\n",
    "# scores = tree.cv_results_\n",
    "# pd.DataFrame(scores).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # plotting accuracies with min_samples_leaf\n",
    "# plt.figure()\n",
    "# plt.plot(scores[\"param_min_samples_split\"], \n",
    "#          scores[\"mean_train_score\"], \n",
    "#          label=\"training accuracy\")\n",
    "# plt.plot(scores[\"param_min_samples_split\"], \n",
    "#          scores[\"mean_test_score\"], \n",
    "#          label=\"test accuracy\")\n",
    "# plt.xlabel(\"min_samples_split\")\n",
    "# plt.ylabel(\"Accuracy\")\n",
    "# plt.legend()\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as increase min_samples_split, the tree overfits lesser since the model is less complex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Create the parameter grid \n",
    "# param_grid = {\n",
    "#     'max_depth': range(5, 15, 5),\n",
    "#     'min_samples_leaf': range(50, 150, 50),\n",
    "#     'min_samples_split': range(50, 150, 50),\n",
    "#     'criterion': [\"entropy\", \"gini\"]\n",
    "# }\n",
    "\n",
    "# n_folds = 5\n",
    "\n",
    "# # Instantiate the grid search model\n",
    "# dtree = DecisionTreeClassifier()\n",
    "# grid_search = GridSearchCV(estimator = dtree, param_grid = param_grid, \n",
    "#                           cv = n_folds, verbose = 1)\n",
    "\n",
    "# # Fit the grid search to the data\n",
    "# grid_search.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # cv results\n",
    "# cv_results = pd.DataFrame(grid_search.cv_results_)\n",
    "# cv_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # printing the optimal accuracy score and hyperparameters\n",
    "# print(\"best accuracy\", grid_search.best_score_)\n",
    "# print(grid_search.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # model with optimal hyperparameters\n",
    "# clf_gini = DecisionTreeClassifier(criterion = \"gini\", \n",
    "#                                   random_state = 100,\n",
    "#                                   max_depth=10, \n",
    "#                                   min_samples_leaf=50,\n",
    "#                                   min_samples_split=50)\n",
    "# clf_gini.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # accuracy score\n",
    "# clf_gini.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# <font color='blue'>SUMMARY PART 3: LASSO & DECISSION TREE</font>\n",
    "OBSERVATIONS\n",
    "1. 47 features identified by LASSO for model buiding.\n",
    "2. Getting around 86% accuracy \n",
    "3. Confusion matix shows lot of false positives still exist.\n",
    "\n",
    "NEXT STEPS:\n",
    "1. Try Random Forrest."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}