08_logistic/08_03_roc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ROC Curves\n",
    "\n",
    "The ROC (Receiver Operating Characteristic) curve or ROC curve is a plot with the True Positive Rate (Sensitivity) on the y-axis, and the False Positive Rate (1 - Specificity) on the x-axis.\n",
    "\n",
    "\n",
    "    True Positive Rate: TP / (TP + FN) [aka. Recall]\n",
    "    False Positive Rate: 1 - TN/(TN + FP) = FP/(FP + TN)\n",
    "\n",
    "---\n",
    "per Wiki:\n",
    "A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.\n",
    "\n",
    "-----\n",
    "\n",
    "The 'ideal' decision boundary provided by logistic regression is the boundary at which the probability of binary classes are even (i.e. P(A) == P(A') )\n",
    "\n",
    "We can modulate the threshold for acceptance if we want a higher or lower True Positive Rate, traded off for a higher or lower False Positive Rate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import linear_model\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Heart Disease\n",
    "\n",
    "\n",
    "The following dataset is a set of data from the UCI data storage. Patients who visited Cleveland hospitals had 76 attributes about them collected, as well as the number of heart diseases they had.\n",
    "\n",
    "Typically, only 14 of the attributes have been used for learning, and rather than predicting how many heart diseases a patient has, we will look at the presence or absence of heart disease.\n",
    "\n",
    "\n",
    "\n",
    "[UCI Heart Disease Link](https://archive.ics.uci.edu/ml/datasets/Heart+Disease)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features = [\"age\",\"sex\",\"cp\",\"trestbps\",\"chol\",\"fbs\",\n",
    "           \"restecg\",\"thatlach\",\"exang\",\"oldpeak\",\"slope\",\n",
    "           \"ca\",\"thal\",\"num\"]\n",
    "data = pd.read_csv(\"https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.cleveland.data\",names=features  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thatlach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trestbps  chol  fbs  restecg  thatlach  exang  oldpeak  \\\n",
       "0   63    1   1       145   233    1        2       150      0      2.3   \n",
       "1   67    1   4       160   286    0        2       108      1      1.5   \n",
       "2   67    1   4       120   229    0        2       129      1      2.6   \n",
       "3   37    1   3       130   250    0        0       187      0      3.5   \n",
       "4   41    0   2       130   204    0        2       172      0      1.4   \n",
       "\n",
       "   slope  num  \n",
       "0      3    0  \n",
       "1      2    2  \n",
       "2      2    1  \n",
       "3      3    0  \n",
       "4      1    0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thatlach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>54.438944</td>\n",
       "      <td>0.679868</td>\n",
       "      <td>3.158416</td>\n",
       "      <td>131.689769</td>\n",
       "      <td>246.693069</td>\n",
       "      <td>0.148515</td>\n",
       "      <td>0.990099</td>\n",
       "      <td>149.607261</td>\n",
       "      <td>0.326733</td>\n",
       "      <td>1.039604</td>\n",
       "      <td>1.600660</td>\n",
       "      <td>0.937294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.038662</td>\n",
       "      <td>0.467299</td>\n",
       "      <td>0.960126</td>\n",
       "      <td>17.599748</td>\n",
       "      <td>51.776918</td>\n",
       "      <td>0.356198</td>\n",
       "      <td>0.994971</td>\n",
       "      <td>22.875003</td>\n",
       "      <td>0.469794</td>\n",
       "      <td>1.161075</td>\n",
       "      <td>0.616226</td>\n",
       "      <td>1.228536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>48.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>211.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>133.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>56.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>241.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>153.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>61.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>275.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>166.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>564.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age         sex          cp    trestbps        chol         fbs  \\\n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000   \n",
       "mean    54.438944    0.679868    3.158416  131.689769  246.693069    0.148515   \n",
       "std      9.038662    0.467299    0.960126   17.599748   51.776918    0.356198   \n",
       "min     29.000000    0.000000    1.000000   94.000000  126.000000    0.000000   \n",
       "25%     48.000000    0.000000    3.000000  120.000000  211.000000    0.000000   \n",
       "50%     56.000000    1.000000    3.000000  130.000000  241.000000    0.000000   \n",
       "75%     61.000000    1.000000    4.000000  140.000000  275.000000    0.000000   \n",
       "max     77.000000    1.000000    4.000000  200.000000  564.000000    1.000000   \n",
       "\n",
       "          restecg    thatlach       exang     oldpeak       slope         num  \n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000  \n",
       "mean     0.990099  149.607261    0.326733    1.039604    1.600660    0.937294  \n",
       "std      0.994971   22.875003    0.469794    1.161075    0.616226    1.228536  \n",
       "min      0.000000   71.000000    0.000000    0.000000    1.000000    0.000000  \n",
       "25%      0.000000  133.500000    0.000000    0.000000    1.000000    0.000000  \n",
       "50%      1.000000  153.000000    0.000000    0.800000    2.000000    0.000000  \n",
       "75%      2.000000  166.000000    1.000000    1.600000    2.000000    2.000000  \n",
       "max      2.000000  202.000000    1.000000    6.200000    3.000000    4.000000  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#looks like issues with ca and thal columns. let's remove them for now\n",
    "data = data.drop(['ca','thal'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Any heart disease amount greater > 0, we'll set to 1.\n",
    "# This way we have a binary class for \"disease\" and \"no disease\"\n",
    "data.ix[ data.num > 0, 'num'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create our data\n",
    "X = data.ix[:,:-1]\n",
    "y = data.num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = linear_model.LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.77899517982944011"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's first use cross validation to see how well it does on average\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "cv_scores = cross_val_score(model,X,y,cv=10)\n",
    "cv_scores.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we can not only output our predictions, but we can also predict the probabilities for those predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 1 0 0 0 1 0 1 1 1 0 1 0 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 1 0 1 0 0 0 0 1\n",
      " 1 1 1 1 0 0 0 0 1 0 1 0 1 0 0 1 0 1 1 0 0 0 1 1 0 1 0 1 1 1 1 1 1 0 0 1 0\n",
      " 0 0 1 0 0 1 1 0 0 1 0 0 0 0 0 0 1 1 1 0 0 1 1 1 0 0 0 0 0 0 1 0 1 0 1 1 1\n",
      " 1 0 1 1 0 0 0 1 1 1 1 1 1 0 0 1 1 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 1 1 0 1 0\n",
      " 0 0 0 0 0 1 1 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 1 0\n",
      " 0 0 1 0 1 0 1 1 1 0 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0\n",
      " 0 1 0 0 0 0 1 1 0 1 1 0 0 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 1 1 1 0 0 0 0 0 0\n",
      " 0 0 0 0 0 1 1 1 1 0 0 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 1 1 0 0 0 1 0 1 1 0 0\n",
      " 1 1 0 1 1 0 0]\n",
      "[[ 0.58681065  0.41318935]\n",
      " [ 0.02029305  0.97970695]\n",
      " [ 0.04515051  0.95484949]\n",
      " [ 0.53968083  0.46031917]\n",
      " [ 0.93896818  0.06103182]\n",
      " [ 0.90507489  0.09492511]\n",
      " [ 0.36556866  0.63443134]\n",
      " [ 0.67571995  0.32428005]\n",
      " [ 0.24657859  0.75342141]\n",
      " [ 0.07372865  0.92627135]\n",
      " [ 0.49291996  0.50708004]\n",
      " [ 0.83688749  0.16311251]\n",
      " [ 0.25807334  0.74192666]\n",
      " [ 0.91330213  0.08669787]\n",
      " [ 0.69640589  0.30359411]\n",
      " [ 0.72698452  0.27301548]\n",
      " [ 0.85137416  0.14862584]\n",
      " [ 0.47031899  0.52968101]\n",
      " [ 0.84784887  0.15215113]\n",
      " [ 0.86585808  0.13414192]\n",
      " [ 0.49770442  0.50229558]\n",
      " [ 0.94517839  0.05482161]\n",
      " [ 0.59955533  0.40044467]\n",
      " [ 0.41750961  0.58249039]\n",
      " [ 0.05292437  0.94707563]\n",
      " [ 0.87209933  0.12790067]\n",
      " [ 0.94928945  0.05071055]\n",
      " [ 0.6678002   0.3321998 ]\n",
      " [ 0.48774967  0.51225033]\n",
      " [ 0.04566385  0.95433615]\n",
      " [ 0.93603299  0.06396701]\n",
      " [ 0.330147    0.669853  ]\n",
      " [ 0.68134837  0.31865163]\n",
      " [ 0.57455765  0.42544235]\n",
      " [ 0.70401655  0.29598345]\n",
      " [ 0.7639296   0.2360704 ]\n",
      " [ 0.03850563  0.96149437]\n",
      " [ 0.04117186  0.95882814]\n",
      " [ 0.08330758  0.91669242]\n",
      " [ 0.23678924  0.76321076]\n",
      " [ 0.32522785  0.67477215]\n",
      " [ 0.82960684  0.17039316]\n",
      " [ 0.93956057  0.06043943]\n",
      " [ 0.57035896  0.42964104]\n",
      " [ 0.8445244   0.1554756 ]\n",
      " [ 0.4658598   0.5341402 ]\n",
      " [ 0.50787809  0.49212191]\n",
      " [ 0.06550918  0.93449082]\n",
      " [ 0.75739785  0.24260215]\n",
      " [ 0.49631331  0.50368669]\n",
      " [ 0.98275548  0.01724452]\n",
      " [ 0.51250488  0.48749512]\n",
      " [ 0.47741048  0.52258952]\n",
      " [ 0.92583417  0.07416583]\n",
      " [ 0.16927808  0.83072192]\n",
      " [ 0.02207193  0.97792807]\n",
      " [ 0.70645608  0.29354392]\n",
      " [ 0.63008372  0.36991628]\n",
      " [ 0.51418553  0.48581447]\n",
      " [ 0.34749225  0.65250775]\n",
      " [ 0.40887249  0.59112751]\n",
      " [ 0.65409159  0.34590841]\n",
      " [ 0.05533392  0.94466608]\n",
      " [ 0.94733049  0.05266951]\n",
      " [ 0.16676985  0.83323015]\n",
      " [ 0.0403809   0.9596191 ]\n",
      " [ 0.29015055  0.70984945]\n",
      " [ 0.4948717   0.5051283 ]\n",
      " [ 0.01692464  0.98307536]\n",
      " [ 0.19883199  0.80116801]\n",
      " [ 0.82346737  0.17653263]\n",
      " [ 0.6612986   0.3387014 ]\n",
      " [ 0.03076316  0.96923684]\n",
      " [ 0.50524029  0.49475971]\n",
      " [ 0.7583821   0.2416179 ]\n",
      " [ 0.6974657   0.3025343 ]\n",
      " [ 0.05205391  0.94794609]\n",
      " [ 0.62190599  0.37809401]\n",
      " [ 0.87796844  0.12203156]\n",
      " [ 0.03950848  0.96049152]\n",
      " [ 0.08363025  0.91636975]\n",
      " [ 0.66265897  0.33734103]\n",
      " [ 0.76366092  0.23633908]\n",
      " [ 0.13548037  0.86451963]\n",
      " [ 0.88228718  0.11771282]\n",
      " [ 0.80682672  0.19317328]\n",
      " [ 0.62401495  0.37598505]\n",
      " [ 0.68398082  0.31601918]\n",
      " [ 0.83214367  0.16785633]\n",
      " [ 0.82843477  0.17156523]\n",
      " [ 0.37680483  0.62319517]\n",
      " [ 0.09113245  0.90886755]\n",
      " [ 0.44612698  0.55387302]\n",
      " [ 0.96944033  0.03055967]\n",
      " [ 0.9355809   0.0644191 ]\n",
      " [ 0.43814813  0.56185187]\n",
      " [ 0.14206988  0.85793012]\n",
      " [ 0.46503344  0.53496656]\n",
      " [ 0.81601797  0.18398203]\n",
      " [ 0.77987233  0.22012767]\n",
      " [ 0.76053201  0.23946799]\n",
      " [ 0.94696818  0.05303182]\n",
      " [ 0.80758536  0.19241464]\n",
      " [ 0.81317142  0.18682858]\n",
      " [ 0.41202708  0.58797292]\n",
      " [ 0.84946434  0.15053566]\n",
      " [ 0.49141815  0.50858185]\n",
      " [ 0.56155943  0.43844057]\n",
      " [ 0.05317774  0.94682226]\n",
      " [ 0.34046677  0.65953323]\n",
      " [ 0.33570736  0.66429264]\n",
      " [ 0.1351156   0.8648844 ]\n",
      " [ 0.96885226  0.03114774]\n",
      " [ 0.11264802  0.88735198]\n",
      " [ 0.42056726  0.57943274]\n",
      " [ 0.69729062  0.30270938]\n",
      " [ 0.74907445  0.25092555]\n",
      " [ 0.90221319  0.09778681]\n",
      " [ 0.05413703  0.94586297]\n",
      " [ 0.06736659  0.93263341]\n",
      " [ 0.26107509  0.73892491]\n",
      " [ 0.17996396  0.82003604]\n",
      " [ 0.36535493  0.63464507]\n",
      " [ 0.00583336  0.99416664]\n",
      " [ 0.84221519  0.15778481]\n",
      " [ 0.95662838  0.04337162]\n",
      " [ 0.03670304  0.96329696]\n",
      " [ 0.05685405  0.94314595]\n",
      " [ 0.91696287  0.08303713]\n",
      " [ 0.91766113  0.08233887]\n",
      " [ 0.52717605  0.47282395]\n",
      " [ 0.64528231  0.35471769]\n",
      " [ 0.95430786  0.04569214]\n",
      " [ 0.51964809  0.48035191]\n",
      " [ 0.94718885  0.05281115]\n",
      " [ 0.88941375  0.11058625]\n",
      " [ 0.05213176  0.94786824]\n",
      " [ 0.19173668  0.80826332]\n",
      " [ 0.11358203  0.88641797]\n",
      " [ 0.45290405  0.54709595]\n",
      " [ 0.76016433  0.23983567]\n",
      " [ 0.7850682   0.2149318 ]\n",
      " [ 0.95182961  0.04817039]\n",
      " [ 0.13625463  0.86374537]\n",
      " [ 0.4105835   0.5894165 ]\n",
      " [ 0.7620437   0.2379563 ]\n",
      " [ 0.10045253  0.89954747]\n",
      " [ 0.88686248  0.11313752]\n",
      " [ 0.82773731  0.17226269]\n",
      " [ 0.94119049  0.05880951]\n",
      " [ 0.89114753  0.10885247]\n",
      " [ 0.51654104  0.48345896]\n",
      " [ 0.60315646  0.39684354]\n",
      " [ 0.09865134  0.90134866]\n",
      " [ 0.01493582  0.98506418]\n",
      " [ 0.03730773  0.96269227]\n",
      " [ 0.28005354  0.71994646]\n",
      " [ 0.60722838  0.39277162]\n",
      " [ 0.39329076  0.60670924]\n",
      " [ 0.60849918  0.39150082]\n",
      " [ 0.90579027  0.09420973]\n",
      " [ 0.33089674  0.66910326]\n",
      " [ 0.88683394  0.11316606]\n",
      " [ 0.49820477  0.50179523]\n",
      " [ 0.86599673  0.13400327]\n",
      " [ 0.53856602  0.46143398]\n",
      " [ 0.82487944  0.17512056]\n",
      " [ 0.87292883  0.12707117]\n",
      " [ 0.27025477  0.72974523]\n",
      " [ 0.95039566  0.04960434]\n",
      " [ 0.0346301   0.9653699 ]\n",
      " [ 0.07301117  0.92698883]\n",
      " [ 0.51366241  0.48633759]\n",
      " [ 0.55342899  0.44657101]\n",
      " [ 0.12091366  0.87908634]\n",
      " [ 0.01940811  0.98059189]\n",
      " [ 0.603824    0.396176  ]\n",
      " [ 0.02495336  0.97504664]\n",
      " [ 0.50462862  0.49537138]\n",
      " [ 0.79393252  0.20606748]\n",
      " [ 0.49742718  0.50257282]\n",
      " [ 0.21777816  0.78222184]\n",
      " [ 0.88955465  0.11044535]\n",
      " [ 0.19641283  0.80358717]\n",
      " [ 0.76621426  0.23378574]\n",
      " [ 0.98449654  0.01550346]\n",
      " [ 0.88787135  0.11212865]\n",
      " [ 0.30881187  0.69118813]\n",
      " [ 0.87130168  0.12869832]\n",
      " [ 0.28525959  0.71474041]\n",
      " [ 0.82048136  0.17951864]\n",
      " [ 0.01424314  0.98575686]\n",
      " [ 0.19361705  0.80638295]\n",
      " [ 0.2460374   0.7539626 ]\n",
      " [ 0.51635642  0.48364358]\n",
      " [ 0.0875412   0.9124588 ]\n",
      " [ 0.67742408  0.32257592]\n",
      " [ 0.55886136  0.44113864]\n",
      " [ 0.95016151  0.04983849]\n",
      " [ 0.59803702  0.40196298]\n",
      " [ 0.85607948  0.14392052]\n",
      " [ 0.55090002  0.44909998]\n",
      " [ 0.87074512  0.12925488]\n",
      " [ 0.78694917  0.21305083]\n",
      " [ 0.71825382  0.28174618]\n",
      " [ 0.16359947  0.83640053]\n",
      " [ 0.0307575   0.9692425 ]\n",
      " [ 0.07743919  0.92256081]\n",
      " [ 0.83637335  0.16362665]\n",
      " [ 0.50170607  0.49829393]\n",
      " [ 0.96236791  0.03763209]\n",
      " [ 0.61946143  0.38053857]\n",
      " [ 0.51072446  0.48927554]\n",
      " [ 0.61407134  0.38592866]\n",
      " [ 0.6981926   0.3018074 ]\n",
      " [ 0.79994303  0.20005697]\n",
      " [ 0.98498267  0.01501733]\n",
      " [ 0.57918337  0.42081663]\n",
      " [ 0.35294361  0.64705639]\n",
      " [ 0.69111493  0.30888507]\n",
      " [ 0.87338286  0.12661714]\n",
      " [ 0.93700759  0.06299241]\n",
      " [ 0.98043329  0.01956671]\n",
      " [ 0.02150408  0.97849592]\n",
      " [ 0.66990214  0.33009786]\n",
      " [ 0.98716656  0.01283344]\n",
      " [ 0.56089276  0.43910724]\n",
      " [ 0.94502368  0.05497632]\n",
      " [ 0.09084154  0.90915846]\n",
      " [ 0.19598047  0.80401953]\n",
      " [ 0.93066639  0.06933361]\n",
      " [ 0.03717933  0.96282067]\n",
      " [ 0.40334315  0.59665685]\n",
      " [ 0.65154899  0.34845101]\n",
      " [ 0.9348385   0.0651615 ]\n",
      " [ 0.01610456  0.98389544]\n",
      " [ 0.02164587  0.97835413]\n",
      " [ 0.31676825  0.68323175]\n",
      " [ 0.96082906  0.03917094]\n",
      " [ 0.8618571   0.1381429 ]\n",
      " [ 0.86122541  0.13877459]\n",
      " [ 0.96237533  0.03762467]\n",
      " [ 0.88941886  0.11058114]\n",
      " [ 0.63297348  0.36702652]\n",
      " [ 0.69708586  0.30291414]\n",
      " [ 0.04462563  0.95537437]\n",
      " [ 0.68461581  0.31538419]\n",
      " [ 0.06073931  0.93926069]\n",
      " [ 0.62963215  0.37036785]\n",
      " [ 0.72588384  0.27411616]\n",
      " [ 0.12154397  0.87845603]\n",
      " [ 0.06972721  0.93027279]\n",
      " [ 0.07897421  0.92102579]\n",
      " [ 0.85560781  0.14439219]\n",
      " [ 0.63934565  0.36065435]\n",
      " [ 0.96497368  0.03502632]\n",
      " [ 0.83860039  0.16139961]\n",
      " [ 0.59599271  0.40400729]\n",
      " [ 0.64498606  0.35501394]\n",
      " [ 0.73861708  0.26138292]\n",
      " [ 0.89954969  0.10045031]\n",
      " [ 0.91607753  0.08392247]\n",
      " [ 0.97687064  0.02312936]\n",
      " [ 0.84631106  0.15368894]\n",
      " [ 0.02396521  0.97603479]\n",
      " [ 0.05134826  0.94865174]\n",
      " [ 0.31713049  0.68286951]\n",
      " [ 0.33037559  0.66962441]\n",
      " [ 0.7611016   0.2388984 ]\n",
      " [ 0.74398665  0.25601335]\n",
      " [ 0.08031966  0.91968034]\n",
      " [ 0.11280259  0.88719741]\n",
      " [ 0.04254345  0.95745655]\n",
      " [ 0.62023311  0.37976689]\n",
      " [ 0.90765127  0.09234873]\n",
      " [ 0.76463573  0.23536427]\n",
      " [ 0.84168112  0.15831888]\n",
      " [ 0.90946822  0.09053178]\n",
      " [ 0.80022151  0.19977849]\n",
      " [ 0.69477435  0.30522565]\n",
      " [ 0.06700288  0.93299712]\n",
      " [ 0.86696688  0.13303312]\n",
      " [ 0.26080191  0.73919809]\n",
      " [ 0.93072462  0.06927538]\n",
      " [ 0.64927143  0.35072857]\n",
      " [ 0.05298976  0.94701024]\n",
      " [ 0.17774694  0.82225306]\n",
      " [ 0.76205785  0.23794215]\n",
      " [ 0.83838644  0.16161356]\n",
      " [ 0.8989777   0.1010223 ]\n",
      " [ 0.46861087  0.53138913]\n",
      " [ 0.92903869  0.07096131]\n",
      " [ 0.10788362  0.89211638]\n",
      " [ 0.03759143  0.96240857]\n",
      " [ 0.63189122  0.36810878]\n",
      " [ 0.95513426  0.04486574]\n",
      " [ 0.04731864  0.95268136]\n",
      " [ 0.40357736  0.59642264]\n",
      " [ 0.71144057  0.28855943]\n",
      " [ 0.14283466  0.85716534]\n",
      " [ 0.10140286  0.89859714]\n",
      " [ 0.96785777  0.03214223]\n",
      " [ 0.86098501  0.13901499]]\n"
     ]
    }
   ],
   "source": [
    "model.fit(X,y)\n",
    "y_pred  = model.predict(X)\n",
    "y_pred_prob = model.predict_proba(X)\n",
    "\n",
    "print y_pred\n",
    "print y_pred_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thought Exercise\n",
    "\n",
    "We are predicting the number of heart diseases based on the patients' other characteristics such as cholesterol, age, etc. \n",
    "\n",
    "-------\n",
    "### Cheap Drugs\n",
    "- What if we had very cheap and low side effect medication that we could give patients when we predict they might have a heart condition?\n",
    "\n",
    "- In that case, we would ideally want to predict \"Heart Disease\" when we get a **lower** probability of heart disease. Say, instead of 50% being the demarcation, we predict \"Heart Disease\" when they have a 40% chance of having it.\n",
    "\n",
    "- What might be the result? We might have both a **higher** True Positive Rate, as well as a **higher** False Positive Rate (think about why)\n",
    "--------\n",
    "### Very Expensive Drugs\n",
    "- On the other hand, imagine patients with heart disease could only be cured with an extremely expensive and invasive procedure.\n",
    "\n",
    "- In that case, we would ideally want to predict \"Heart Disease\" when we get a **higher** probability of heart disease. Say, instead of 50% being the demarcation, we predict \"Heart Disease\" when they have a 60% chance of having it.\n",
    "\n",
    "- What might be the result? We might have both a **lower** True Positive Rate, as well as a **lower** False Positive Rate (think about why)\n",
    "\n",
    "\n",
    "We changed our TPR and FPR by modulating the threshold of acceptance for predictions. \n",
    "\n",
    "We can plot out this curve through the `roc_curve` method within `sklearn.metrics`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x118beeb90>"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEcCAYAAAAydkhNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXHWZ9//3qeru9J7urGRfSLjZCcgqyCZqQASRTVwA\nYZRxhpnhp47jOI+D2zwz/NTRcXAcFFGUTVwBlU12kC1CWITcEpJAyEb2dJZeq54/zumm0nZ1V3e6\n6tTyeV1XrtSp+tapu086567vHqTTaURERAaSiDsAEREpXkoSIiKSlZKEiIhkpSQhIiJZKUmIiEhW\nShIiIpKVkoSIiGRVFXcAUl7MbAUwCegBtgN3AZe7+46MMm8HvgocDqSAh4F/cveXM8o0A18GzgLG\nAeuAO4CvuvvGAT43AP4O+DgwB9gMPA582d1fHO2fc0+Z2VXApdHhte7+uUHKngd8CZgGrAQ+7+63\nRa+NAf4LeD9QDTwG/LW7rx7kfI3AWuBhdz+t32spYJ67L8t47ovA3u7+0eh4WP82UtpUk5DRlgZO\nd/cmYAFwKPDPvS+a2THA3cCvgCmEN/TngMfMbE5Upga4D9gPeE90rmOADcCRWT73v4C/J0wUrcA+\nwK+B9w73BzCzvH55MrPLgDOBg6M/74ueG6jsNOAnwBXu3gz8I3CTmU2IivwDcDRwEDCVMDn+9xAh\nnA20A6eY2eQcQu6bcTvCfxspYapJSN64+zozu4cwWfT6/4Hr3b33RrYD+IKZvQ34InARcCEwAzjB\n3XdG51oP/NtAn2Nm84G/AY5290XR013ATRllHgR+4u4/iI4vBi5193dExyngcuAKoMrM7gJ2uPs/\nZpzjNuBBd/+mmU0lvBm/g7DG9M2Mn2koFwFf7/22b2ZfBz4BXDNA2enAFne/O7oOvzOzHcDehDfm\n2cDd0fXBzG4FvpHD5/8vcCrwkRzKBxmPh/VvI6VPNQnJhwDAzKYDC4FXouN6wm+dPxvgPbcC74oe\nnwLc2XsTysE7gZUZCWIgaTK+EWdxJuG34f2Am4Hze18ws9YovpvNLEHYvPIs4bf3dwJXmNm7o7LH\nmdnmQT5nf8LaU6/ngQOylH0aeNnMTjezpJm9n7AW8Hz0+g+AY81sSnR9Pwz8LtsHm9ks4ATgBuBG\nwpt+Lnqv3XD/baTEKUnIaAuAX5vZNuB1wvbqK6PXxhH+zq0Z4H1rgd4mlPFZymQzPnr/nvp3d9/i\n7h3Ao0DazN4RvXYO8Ad3XwscAUxw96+6e7e7LweuBT4I4O6PunvrIJ/TCGzNON4WPfcX3D1F2Nx0\nM2FyuBG4zN13RUWWAm8Aq6JzGvCVQT77o8Bz7r4EuAU4wMwWDFK+v3EM799GSpyShIy2NHBm1H5+\nIuG38onRa5sJO6qnDPC+KcD66PEGwm/oudqY5ZzDtbL3gbunCW+iF0RPfYjwBg0wC5hqZpt7/xD2\nu0zK8XO2A80Zx2Oj5/6CmZ0CXEXYvFNNWAv4gZkdEhX5DlBDePNuIOzruXOQz76QqBkuau56mLD5\nqVcPYQd4pmrC5jsIr/Vw/m2kxClJSN64+8PAj4CvR8c7CEccnTdA8fMIO0QBfg+8J2o+ycV9wPSo\nXyObHYQ30V57DVCmf3PUzcA5URPNkcAvoudfB5a7e2vGn2Z3Pz3HeP/E7v00hwDZRmAtIByF9AxA\n1KT2JGETV+97fxjVgDqBq4EjzWxc/xNFo8rmAf9sZmvMbE30c30oakLr/dnm9HvrHOC16PFw/22k\nxKnjWvLtW8AKMzvY3Z8HPgfcbWZLCBNIFfBp4CjCZhwIm1cuA35hZlcQ9mm0Rs896+67fVN291fM\n7H8I+ws+TpiIEoTDQme5+1XAYuADZnYt4VDSSxmiicrdF5vZBsKmpLvcfVv00lNAm5l9lrDzupOw\nxlQ7RL9Irx8DnzKz3xE2z32KcHTWQJ4CPmtmh7j7c2Z2KGFn+Xei158GLjKzh4BdhB34q9x90wDn\nugi4h937IeoJ+zdOA34D/BT4P2b2AmGz0snA6YRDlmGY/zZS+lSTkLxy9w2EN8UvRMePAe8BPgCs\nBlYQfhs+zt1fjcp0EnaQLgHuJWxrf5KwSeWJLJ/z94Tfor9D2Ky1lLAj+vaoyDcJb+brgB8Sdtxm\n1hyydWrfRHij7BspFfUTnE74LX8ZYTPZ94iakMzsHWbWNsg1uYaw4/sFwhv0He7+vd7XzexFM7sg\nKvsw4RyJn0f9PD8H/s3dfx8V/wxhX8UrwJuEAwXO6v+ZZlYLnAv8t7u/mfFnBeGNvzdxfBn4A2Gf\nzCbgP4APuftLUTzD/reR0hbkc9MhM7uOcJz6m+5+UJYy3yYcircTuNjdn81bQCIiMiz5rkn8kPCb\nzYDM7DTC2Z3zCceJfzfP8YiIyDDkNUm4+yOEVf9szgCuj8o+CbTkOANUREQKIO4+id61aHq9QTjD\nVEREikDcSQJ2n/IPQ8+KFRGRAol7COwqwnVgek2PnssqnU6ng6B/XhGRSnLdHX/i0cVvxB1GQazf\n0g5AS1PNsN/bPf4V0nWbqFp5JDd++b216XS6Y7jniDtJ3E64qNotZnY04UJm6wZ7QxAErF+fdXRh\nRZk4sUnXIqJr8ZZKuBYPPbOSzW0dNNX3nxy+u0QiIJUq7caJ5oZqDpg9jncdMWPowkBPuodkkASg\nO7U/L2xZzP4LjBu/PLLPz/eSyDcTLiMwwcxWEq7hUw3hWPFoRcvTzGwp4YzYj+UzHhEpTbfev5Sn\nlrz1/XFLlCCuOPeQQd4FrS0NbN6yY9Ay5SSdTnPT8h9x8l7vZlr9DKoSVRw67nA6O4ddgeiT1yTh\n7hfkUObyfMYgIiPT/8Ycp03bwptcc0NYc2isD79dy+6CIOCoCceyoWM90+pzq3kMJe7mJhEpUk8t\nWZdTk04hDLfJpZJs7tzEH9Y/zGlTzyQIAvZp3ndUz68kIVJmrrvjTzz0zMqhCw4h1yYdidfY6ha2\ndW3lzfa1TK4bjcWQd6ckIVJmHl38xqjUANSkU7xWbF9GIkgws2E2iSDB+bM+SiLIz4wGJQmREpWt\nz2BLW6dqAGUuTZq71/yWS/f+JIkgkbcEAcUxmU5ERqC3z6C7J7Xbn+ZG1QDK0codr5FKpwCY07g3\nF8y6MK/JoZdqEiIlbKAaQ6UN+6wE6XSaP6x/hNmNczlqwtsBaKxuKshnK0mIjEAxDA/d0tZBYxGM\nPJL82dm9g/qqBoIgYOHU0+lIjXy+w0gpSYgMIlsy6D9uPw7qWC5vmzs3cfOKH3Px3E9QX1XP2JqW\nWOJQkhAZRLa5Ahq3L/mSTqcJgoDWmnEcPeFYOlLt1BPfluJKEiID6K1BaK6AFNKzmxaxo3s7x006\nEYDDxh0x+BsKQKObRAbQW4NQk44U0vwmY+2uNX2jmIqBahJSkYbqeFYNQgohnU7zwLp7OWL80TRV\nN9NY3cQ5s4Zc8q6gVJOQipRtjkHvH9UgpBCCIGBMspanNz4ZdyhZqSYhFUs1BYlDR087y7YvZb+x\nBwJw9IRjY45ocEoSUhEG2o9AcwwkDinSPLjuPsZWtzK1flrfBkHFSklCykqu8xrUnCSF1N7TTleq\nk6bqZuqSdZw368O01LTGHVZOlCSkrGhegxSjl7a+iG97iQ/O+ihBEDB+zIS4Q8qZkoSUpP41hmQi\noCeV1qgkKRpdqS6qE+GXlQWth1EVJEmTJiCIObLh0egmKUn9Ryd1aVSSFJmfvXYTy9qWApAIEhzc\nemhBVm0dbapJSMnKrDFo5VMpNsdOOp6NHRvjDmOPKUlISclcLkOjk6SYbOvayn1r7ubMGeeQCBLM\napjDrIY5cYe1x0qv7iMVTctlSLFqqmomTZrVu1bFHcqoUk1CYjOSPRnUMS3F5PUdK+hMdTKvaR+C\nIOCsGecRBKXVMT0U1SQkNkMtjaHlMqTYVQXV/H7NXXSnugHKLkGAahISAy3DLaVsxfZlTK+fSVWi\niqn107hw7qVUJcr3Vlq+P5kUjf7NSpmzn1UrkFLz3OZneH3HCo6ffDIA9VUNMUeUX0oSknf9Z0Fr\n9rOUmu1dbTRWNwFwypSF7OiunOHWShIy6gZaTE/NSlKq2rq2cf2ya7lw7qU0V4+loaqRhqrGuMMq\nGCUJ2WODNSeBFtOT0tS713RTdTMnTn5nX+d0pVGSkD2m5iQpNy9sXsza9jW8a8qpABzYUrm1YCUJ\nyVm2eQ1qTpJyM795X15pc7pT3WU9cikXmichOcs2r0HNSVIO7l97D5uitZZqk7V8YOb5FZ8gQDUJ\nGSbVGKRcja1u4emNT/Ceqe+NO5SioiQhIhWpo6cD3/YSB7ceCsBh444gRSrmqIqPmptEpCIFQcAT\nGx5jxfZlfcfFvt90HFSTEJGK0d7Tzq6enbTWjKMmUcN5sz5MU3Vz3GEVtbwmCTNbCHwLSALXuvtV\n/V4fC9wAzIhi+bq7/yifMcnQBhvFpD0cpJS92vYKizY9wUfmXEIySNJS0xp3SEUvb81NZpYErgYW\nAvsDF5jZfv2K/S3worsvAE4EvmFmqt3ETKOYpJx0pjpJp9MA7D/2QI4Yf3TMEZWWfN6QjwSWuvsK\nADO7BTgTeDmjTArores1AxvdvTKnNRYZjWKScnHbyp9zQMvB7D/2QIIgYP+xB8UdUknJZ8f1NGBl\nxvEb0XOZrgb2N7PVwHPAP+QxHhGpQO+YdCK7unfGHUbJymdNIp1DmYXAM+5+kpntDdxrZoe4e9tg\nb5o4sWlUAiwHo3ktrrvjTzy6+A22tHXS3FhNa0tpLYFcavHmUyVfi+2d27nllVu4aN+LANhvynz2\nY37MUcWro2Pkt/p8JolVhB3SvWYQ1iYyXQz8O4C7v2pmywEDFg124vXrB80hFWPixKY9uhaDLcy3\n38xWNm8pneWQW1saSirefNK1CKiljhfXLOHQ6QdV+LUIdXZ2jPi9+UwSi4D5ZjYbWA2cD1zQr8zr\nwCnAY2Y2mTBBLMtjTJJBC/NJuVi54zXaurf19TecNvXMstxKNA55SxLu3m1mlwN3Ew6B/YG7v2xm\nl0WvXwN8BfiRmT0PBMBn3X1TvmKSv6QOaikHdVX1/GbVr5nXZNQkapQgRlHQOzSshKTV3BQaqLkp\n2xyHgfTOeyiHJKEmlrdUyrVYvv1VptZNZ0xyDADtPbuoTdbtVqZSrsVQOjs7OO7wvWvT6fSw2520\nLEeZyTbHYaA/mvcgpcy3vcxDb97Xd9w/Qcjo0MS1mAznG382yURAT2r3mqD2dpBytrVzC2NrWgA4\nafIpbOvaFnNE5U81iZgM5xt/tj9dqh1IBdnZvZOfLL+OjR0bABiTrGVi7aSYoyp/qknkWT53c1N7\nq1SCVDpFIkhQX1Xft52oFI6SRJ70JofMuQeZ9I1fZGgvb32RV9te4fTpZwFgzf2Xf5N8U5LIk97m\nJM09EBm5eU3GK21OZ08HNdEoJiksJYlR0r9ZSR3IIiPzwNp72X/sgUyum0J1opozpp8dd0gVTUli\nmLL1MfRvVlJzksjITKqdzKJNT/HeaWfGHYqgJDFs/Zey6KVmJZGR6Ux18sLmxRw27oi+pbz3G3tg\n3GFJREkiR701CDUjiYyuZJDkhS2LaapuZp/mfQmCgAAtq1EslCRylFmDUDOSyJ5p79nF9q7tTKid\nSDJIcvbMD1JfVbnLmxczJYkhqAYhMvre2LmSB9bdy8VzP0F1opqm6uah3ySxUJIYgmoQIqOjo6ed\nmsQYgiBgXtM+fZPkpLgpSeRANQiRPXfn6juY2TCHw8YdDsA+zfvGHJHkQmlcRPImcyuC4yedTE+6\nO8ZoZCRUk8gw0ByI3j0XRGR42nva+fnrN3PuzAsYk6xl3JjxjBszPu6wZJhUk8gw0MqsmhQnMjK1\nyVqm1E3ljZ0r4w5F9oBqEv2o/0Fk5FbtfIP1HetY0Po2AN6513tijkj2lGoSIjJqGqsaeXz9o7T3\n7Io7FBklQ9YkzGwccBUwDzgX+BrwKXffnOfYRKQELGtbyl51U6ivamBsTQuX7v3XWrG1jORSk/g+\nsAgYD7QBq4Eb8hmUiJSO13eu4P619/QdK0GUl1ySxBx3vwbocfcOd/8XQKvYiVSwzZ2b+h4fO/EE\njppwbIzRSD7lkiS6zGxs74GZzQd68hdS4d16/1I+8z+PsaWtI+5QRIpeR08HNy2/nrW7VgNQnajW\nXtNlLJfRTVcCDwIzzew24BjgknwGVWhaekNkaD3pHpJBkjHJMZw27QyqAs0fqgRDJgl3v8vM/ggc\nBSSBy4Cy67TW0FeR7F5pc57b/Axnz/ggQRAwp3HvuEOSAslldNPj7n4M8JvoOAksBg7Kc2wiUiTm\nNs7jlW1Oe6qdumRd3OFIAWVNEmb2AHBC9DiV8VIPcFue4xKRmD207n7mNu7NjIZZJIMkp007I+6Q\nJAZZk4S7nwRgZt92978vXEgiUgym1U/nmU1PM6NhVtyhSIxy6bj+rJmdBTQCAWG/xBx3/9e8RlYA\nmRsKaRE/qXRdqS7+uOkpjhr/9r49H+Y2zos7LIlZLknil0AdMB94GDgeeDyfQRWKRjWJvCUZJFnW\n9gqNVU0c2HIwgDYFkpyShBEuyfFt4DrgM8Av8hnUaBtoCXBAW5JKxWvvaWdr52Ym100hESQ4c8a5\n1CZr4w5LikguXxPWuXsaWAIc7O6rgZKadz/QEuBaBlwE1rev45crb+1bkK+hqoFkkIw5KikmudQk\n/mRm/w18F7jRzKbm+L6iohqDSKi9p53qRDXJIMmMhlksnHo61YmauMOSIpVLTeKTwK3u/hLh7Ou9\ngA/lNSoRyZv7197NUxv+0Hc8p3Fv1R4kq0GThJkZMMndHwFw99uB/wtoSKxICcnca/q4SSdSldBo\nPsnNYJPpvkjYSU00BPaB6PjzwBO5nNzMFgLfIhw2e627XzVAmROBbwLVwAZ3P3E4P4CIDK4r1cXN\nK37M2TPPp6GqkebqsRwx/ui4w5ISMVhN4iLCYa8nAP8fcBfwYeBcd3/3UCeOlu+4GlgI7A9cYGb7\n9SvTAnwHeJ+7HwicM5IfQkSyq05UM7dxnvaalhEZLElsc/c17v5H4AjgeWCBu9+d47mPBJa6+wp3\n7wJuAc7sV+ZDwC/c/Q0Ad98wvPBFZCBrdq3mqQ1vTWc6btIJWPN+g7xDZGCDJYnM9Zo2AJ929+Hs\nIzENyPzq8kb0XKb5wDgze8DMFpnZR4dxfhHJorl6LM9ufpod3dvjDkVKXK7TKdujuRLDkUv5auAw\n4DTgPcAXok2NRGSYlrUtZVvXViCc73DJ3p+koaox5qik1A023+EAM1sePZ6a8Rgg7e5zhzj3Knbf\n5nQGYW0i00rCzupdwC4zexg4BHhlsBNPnNg0xEfvLpkISKUDWlsahvW+UlCOP9NIVfq1aNu+iRfX\nP8tFEy+u+GuRSdcCOjpGPrVtsHfuM+KzhhYB881sNrAaOB+4oF+Z24Cro07uMYQbG/3nUCdev75t\nWIH0pNKkUmk2b9kxrPcVu9aWhrL7mUaqUq/Fhvb1TKidCMBBDYczvTr87laJ12Iglfp70V9n58i3\nZh5sqfAVIz5r+P5uM7scuJtwCOwP3P1lM7ssev0ad19iZncRdoqngO9Hk/ZEZAhdqS5+/vrNnDr1\nfcxqnEMiSDB+zIS4w5Iyk9flNdz9TuDOfs9d0+/468DX8xmHSDnpSnVRnaimOlHN6dPezxgtyCd5\npHWARUrIiu3LuPW1G0mlw8GH0xtmMrF2UsxRSTnLqSZhZscBBwI/Ao5094fzGZSIDGxWwxyWtv2Z\nXT07NXJJCmLImoSZXQF8FfgU0AB8z8z+Md+BiUjo0Tcf5JU2ByAIAk6ZslAJQgoml+amiwnnMOxw\n943A4cAl+QxKRN4yu2EuL2xeHHcYUqFySRI97p45fqod6M5TPCIVrzvVzaNvPkhPOlzgYHrDTM6a\ncV7MUUmlyiVJPGRm3wAazez9wO3A/fkNS6RyJYMk69rX8tzmZ/ueC4IgxoikkuWSJD5DOAP6OeBC\n4HfAp/MZlEilae9p71ulNQgC3jvtTA5pPTTmqERyG930TeAn7v6/+Q5GpFJt7dzMbSt/xkVzP05j\ndRO1ybq4QxIBcksSrwDfMrPxwI3ADXs6G1tEwtpDMkhSnahmct0Uzph+NnVV9XGHJbKbIZub3P1q\ndz+OcPOgduA2M3s075GJlLlH3nyAR998sO94RsMs7TUtRSenGddmNhY4BXg34TpMuW48JCIZdttr\neuIJNFYPb0VjkUIbsrnJzO4g3PPhl8AX3P3JvEclUoZ60j3csPw6zpx+Di01rdRV1WuvaSl6ufRJ\nfA+40901N0JkDySDJAeMPZg1u1bRUtMadzgiOcmaJMzsS+5+JfAB4CwzyxyonXZ3zboWGcK69rX8\nedvLvGPSSQAcPv6omCMSGZ7BahKLor8fBPrP5BnuVqYiFamlupUl217i4JZDGVvTEnc4IsM22KZD\nd0QPp7n7/818zcz+Pa9RjZJb71/KU0vWsaWtg8b66rjDkQqxfPurNFePZfyYCYxJjuFjcy+jKpHX\nrVtE8maw5qb/ACYDZ5jZPN6qTVQBRwP/nP/w9sxTS9axua2DpvpqDpg9Lu5wpEJs6dzCExse5YOz\nLiQIAiUIKWmD/fb+EtgfeCfwEG8liW7gK3mOa9Q01VdzxbmHxB2GlLl1u9YwuW4KAAtaD2NWw2yt\ntyRlYbDmpqeAp8zsV+6+tYAx7TE1M0kh9aR7uGPVrzh+0sns07wvQRAwbsz4uMMSGRWDNTc96+6H\nApvNrP/LaXcv2qmhamaSQuhMdVKTqCEZJDl92vtJBmpWkvIzWE3i0OjvktwHW81Mkk+rdq7knjV3\ncuHcS0kGSfaqmxp3SCJ5kcuM63nAUcDNwP8ChwKfcvdH8hybSNGaWjeduY3z2Nm9g6bq5rjDEcmb\nXGoJPwS6gDOAfQj3kvh6PoMSKUaPr3+UF7c8B4R7Ppww+WQlCCl7uSSJWne/FTgduMndHya35TxE\nysrcpnks2frSbov0iZS7XJJEt5mdQ5gkfhNtYdqT37BE4pdKp3ho3X10pjoBmFy7F2fP/KCGtkpF\nySVJXAacBvytu68GPgj8VV6jEikCiSBBW3cbz25a1PecEoRUmlw2HXqecAvTqWZ2BXBV9JxI2Wnv\naWfF9mV9x+/e61QtyicVbcgkYWYfBX4NzAFmA780s0vzHJdILHZ27+A3q37Fls7NANQkx2i3OKlo\nuXRAfwY40t03ApjZVwmX6fhBPgMTKZT2nnYCYEyylnFjxvOBGedr1JJIJJc+iURvggBw9w2o41rK\nyJMb/sD96+7tO55aP121B5FILjWJ583sW4Q1hwC4FHgur1GJ5FkqnSIRhN+RjplwLM9vWUw6nVbH\ntEg/udQkPg50AtcRTqzrBP4mn0GJ5FMqneIny3/A+vY3gbDf4fDxRylBiAxg0JqEmU0AZgFfcvfP\nFiYkkfxKBAkOG3cE69rXMLF2UtzhiBS1rDUJMzsXWAH8FlhuZicWKCaRUbehYz2/X3NX3/FBLQs4\nsEULQIoMZbDmpi8AR7j7XsBHgS8WJCKRPGipbuWNnSvZ2LEh7lBESspgSSLl7i8DuPvdgHZRkZKy\nfPurrN21GoCqRBUfnXsJ48dMiDkqkdIyWJLov4pZ93BPbmYLzWyJmb1iZv80SLkjzKzbzD4w3M8Q\nyaa9p5271/y2b0E+DWsVGb7BOq4bzez46HGQcRwQ7kz38GAnNrMkcDVwCrAKeNrMbu+tnfQrdxVw\nF2/toy0yIqt2vsHUumkEQcC+zfv3PRaRkRksSawCvjTI8UlDnPtIYKm7rwAws1uAM4GX+5X7O+Dn\nwBE5xCuSVTqd5r61d7Og9TAObj2UIAgYW9MSd1giJW2w7UtP3MNzTwNWZhy/QbjDXR8zm0aYOE4m\nTBJ7tFD/rfcv5akl69jS1kFjffWenEpKSHtPO9BAEAScNu0MUulU3CGJlI187l+dyw3/W8Dn3D1N\n2NS0R+0CTy1Zx+YoQRwwe9yenEpKxJvt6/jxsmvp6OkAYMKYiUyqnRxzVCLlI587zK0CZmQczyCs\nTWR6G3CLmQFMAE41sy53v32wE0+c2DTg88lEwNjGGq689JgRB11qWlsa4g4hFr1LaLQylyO7j2Rn\n105aW1rjDqtoVOrvxUB0LaCjY+S3+nwmiUXAfDObDawGzgcuyCzg7nN7H5vZD4E7hkoQAOvXt+12\n3L+ZafOWHaMQfvFrbWmomJ8109MbnyAg6NvnYUHjkbTWVua1GEil/l4MRNci1NnZMeL35rKfxDgz\n+76ZPWBmE8zsh2Y25Fc2d+8GLgfuBl4CfuruL5vZZWZ22YgjHoCamSrLvKZ9WL79Ve01LVIAudQk\nvg/cQ9jp3EZYK7gBeO9Qb3T3O4E7+z13TZayH8shlqya6qu54lwts1CO0uk0D6z7PcdMPI66ZB2t\nNeM4d9aH4g5LpCLk0nE9J7qx97h7h7v/C7v3NYjkVRAEpEnxx41Pxh2KSMXJpSbRZWZjew/MbD7a\ndEjyrL2nnTd2vs68pn0AOH7SyX37P4hI4eTyv+5K4EFgppndBjxGuPifSN50pTq5a/Vv2NC+HoDq\nRLWW1RCJwZA1CXe/y8z+SDiDOgl8wt3X5T0yqTjtPe2k0j3UVzXQVN3MubM+ROsYDUQQidOQScLM\nriScGNc70W2BmeHuX85rZFJxFm9axJr21bx/+rkEQcDk2r3iDkmk4uXS3JQ5E7qGcBkNTWmVUdGT\nfqt76/DxRzOrYe4gpUWk0HJpbvpi5rGZfRm4N18BSeVIp9PctPxHnLzXe5hWP52qRBWHjTs87rBE\nJMNIhos0oSGwMgqCIOCoCceyoePNuEMRkSxy6ZNYnnEYAK3A1/IWkZS1zZ2b+MP6Rzht6hkEQcA+\nzfvGHZKIDCKXeRLnAeujx2lgi7tvzV9IUs7GVrewtXMzb7avZXLdlLjDEZEh5JIkfuLuRfl1T/tH\nlIYV25eRCBLMbJhNIkjwwdkXamKcSInIJUksNrMLgSeBXb1PuvvreYsqR70L+zVpYb+ilibNPWt+\nxyV7/zVx2wbQAAAQMklEQVSJIKEEIVJCckkSR9NvR7nInFGOZUS0sF9xWrnjNabVzyARJJjTuDfn\nz/qIkoNICcqaJMzsIne/3t1nFzCenKiZqbil02n+sP5h5jTO48gJ4QZQTdXNMUclIiMx2Fe7KwoW\nxTBp/4jitLN7JxAObV049X3MbtTEOJFSV7L1/95mpncdoSkbxWBz5yZ+tOx77IoSxdiaFu01LVIG\nBuuT2L/fHIlM6cytRwvpkq/crWamItK313TNOI6ecCztqXbqqI87LBEZJYMliaXAaby1blNR2Li1\nXc1MReKZTYvY2b2d4yadCMBh446INyARGXWDJYlOd3+tYJHkqLmxhr8/++C4wxBgnybjrtW/IZVO\naeSSSJka7H/2YwWLQkpCOp3m/rX3sL2rDYDG6ibOmXWBEoRIGcv6v9vdLy9kIFL8giCgJjGGpzc+\nEXcoIlIguUymkwrW3tPO8u2vst/YAwA4ZuJxMUckIoWkdgIZVDqd4oF197Jm1yoAkkFSe02LVBDV\nJOQvtPe005XqpKm6mbqqes6f9RFaalrjDktEYqCahPyFl7a+wG9X3UY6nQZg/JgJqj2IVCjVJASA\nrlQX1YlwguKC1reRDJKkSRMU1zQZESkw1SSEdDrNz167keXbXwUgESQ4pPUwDW0VESUJCYe2vn3i\n8Wzp3Bx3KCJSZJQkKtS2rq38auXPSKVTAMxunMuh4w6POSoRKTZKEhWqqaqZVLqnb2iriMhA1HFd\nQV7fsYKuVBd7N80nCAI+MON8gkAd0yKSnWoSFSQZVPH7tXfSk+4BUIIQkSEpSZS517YvpzvVDcC0\n+ul8ZM6lmvMgIjlTkihzz25exOMbHuk7bqhqiDEaESk16pMoQ9u72misbgLglCmnsrN7R8wRiUip\nUk2izLR1beP6Zd+nrWsbAI1VjdprWkRGLO81CTNbCHwLSALXuvtV/V7/MPBZwm1S24BPuvvz+Y6r\n3PTuNd1U3czxk0+mK9UVd0giUgbyWpMwsyRwNbAQ2B+4wMz261dsGXC8ux8MfAX4Xj5jKkfPb36W\n+9be3Xd8UMsCxo0ZH2NEIlIu8l2TOBJY6u4rAMzsFuBM4OXeAu7+eEb5J4HpeY6p7OzTvC9L2/5M\nd6qbqoS6mURk9OS7T2IasDLj+I3ouWwuBX6X14jKxP1r72Hjrg0A1Cbr+MDM85UgRGTU5fuuks61\noJmdBFwCHDtU2dYWDePca+dEHln9CO/f+6y4Qyka+r14i67FW3QtoKNj5Lf6fCeJVcCMjOMZhLWJ\n3ZjZwcD3gYXuPuRSpJu3VN6Qzo6eDnzbSxzceigA+9YdwtF71VfktRhIa0uDrkVE1+Ituhahzs6O\nEb8330liETDfzGYDq4HzgQsyC5jZTOCXwEfcfWme4ylZQRDwxIbHGFvTwqyGOSSCBMmEZk6LSH7l\ntU/C3buBy4G7gZeAn7r7y2Z2mZldFhX7V6AV+K6ZPWtmT+UzplLS3tPet8dDTaKG82Z9mOn1M2OO\nSkQqSd57Ot39TuDOfs9dk/H4r4C/ynccpWhp2595ZtNTfHjOx0gGSVpqWuMOSUQqjIbDFJnOVCfV\nQTVBEHDA2IO0x7SIxErLchSZX6/8GUu2vQSE/RAHtBykVVtFJDZKEkXmHZNO0oJ8IlI0lCRitqN7\nOz997Ya+PR+m1E3lbeOPjDkqEZGQkkTMGqoaaUg2sGrnyqELi4gUmDquY7Byx2ts725jv7EHAvDe\nae/XVqIiUpRUk4hBbbKOB9fdR2eqE9Be0yJSvJQkCmT59lfp7Amnxk+sncTH9v4ENYmamKMSERmc\nkkSBLNn2Eg+9eX/fcW2yLsZoRERyoz6JPNrWtZXm6rEAnDT5XX1bioqIlArVJPJkZ/cOfrzsB2zq\n2AhAbbKWibWTYo5KRGR4lCRGWSqdAqC+qoF3TVlIOvctNUREio6SxCh6aeuL/G717X3H1rw/48dM\niDEiEZE9oyQxiuY17UNPqqdvaKuISKlTkthDD6y9l3Xta4Fwz4czZ5ytoa0iUjaUJPbQxNpJLNr4\nZNxhiIjkhZLEMHX2dPDHjU+RTocd0geMPZhTp74v5qhERPJDSWKYkokqnt+ymKVtfwbCJTUSgS6j\niJQnTabLQXtPO9u725gwZiLJIMk5Mz9IfVVD3GGJiOSdvgLnYOXO1/j1yp/RleoCoKm6WbvFiUhF\nUE0ii46edmoSYwiCgPlNRiqdUrOSiFQc3fWyuHP1HSze/EzfsTXvp9qDiFQcJYkMvSOWINxrujvd\nFWM0IiLxU5KItPe0c9OK6+noaQdg/JgJHDH+6JijEhGJl5JEpDZZy+TavXhDe02LiPSp6I7rVTtX\nsr7jTRa0vg2AU6YsjDkiEZHiUtE1icaqJh5f/yjtUROTiIjsruKSxLK2pezs3gnA2JoWLt37r6lN\n1sYclYhIcaq4JPHajhXcv+6evuOa5JgYoxERKW4V0SexuXMTrTXjADhu0gls6dwcc0QiIqWh7GsS\nHT0d3LT8etbuWg1AdaJae02LiOSobJNET7oHgDHJMZw69X1UBdUxRyQiUnrKMkm8ss359cqf9c2g\nnts0jwm1E2OOSkSk9JRlkpjbNI+6ZB3tKQ1tFRHZE2WTJB5adx8rd7wGQDJIctq0M6lL1sUclYhI\nacvr6CYzWwh8C0gC17r7VQOU+TZwKrATuNjdnx3JZ02rn8Gzmxcxo2HWnoQsIiIZ8laTMLMkcDWw\nENgfuMDM9utX5jRgnrvPBz4BfDfX83elunhyw2N9/Q7zmvbh9GlnjVb4IiJCfpubjgSWuvsKd+8C\nbgHO7FfmDOB6AHd/Emgxs8m5nDwZJFna9govbX2x7zltCiQiMrry2dw0DchcUvUN4KgcykwH1mU7\n6d57hzWHRJDg/TPOoVb9DiIieZPPJJEeuggAwXDet7LhQdp2zWZMspZqqulJddND98giLHEdHVV0\ndnbEHUZR0LV4i67FW3QtQp2dnSN+bz6TxCpgRsbxDMKawmBlpkfPZXXd2V+rvY6vjUqAIiKVIp1O\njyhb5jNJLALmm9lsYDVwPnBBvzK3A5cDt5jZ0cAWd8/a1AQj/0FFRGT48tbT6+7dhAngbuAl4Kfu\n/rKZXWZml0VlfgcsM7OlwDXA3+QrHhERGb6gdwipiIhIfxozKiIiWSlJiIhIVkoSIiKSVdHuTFfI\ndZ+K3VDXwsw+DHyWcM5JG/BJd3++4IEWQC6/F1G5I4DHgfPc/ZcFDLEgcvz/cSLwTaAa2ODuJxYy\nxkLJ4f/HWOAGwuH2VcDX3f1HhY6zEMzsOuC9wJvuflCWMsO6bxZlTSLf6z6VklyuBbAMON7dDwa+\nAnyvsFEWRo7XorfcVcBd/OVkzZKX4/+PFuA7wPvc/UDgnIIHWgA5/k78LfCiuy8ATgS+YWZF+wV5\nD/2Q8FoMaCT3zaJMEuR53acSM+S1cPfH3X1rdPgk4aTEcpTL7wXA3wE/B9YXMrgCyuU6fAj4hbu/\nAeDuGwocY6Hkci1SQHP0uBnYGA3RLzvu/giweZAiw75vFmuSGGhNp2k5lCnHm2Mu1yLTpcDv8hpR\nfIa8FmY2jfAm0fsNqRzHeOfyOzEfGGdmD5jZIjP7aMGiK6xcrsXVwP5mthp4DviHAsVWjIZ93yzW\nJJGXdZ9KVM4/k5mdBFwC/FP+wolVLtfiW8Dn3D1N+PtRds1N5HYdqoHDgNOA9wBfMLP5eY0qHrlc\ni4XAM+4+FVgAfMfMmvIbVlEb1n2zWJNEXtZ9KlG5XAvM7GDg+8AZ7j5YdbOU5XIt3ka4zMty4Gzg\nf8zsjALFVyi5XIeVwD3uvsvdNwIPA4cUKL5CyuVaXAz8EsDdXwWWA1aI4IrQsO+bxdp5k5d1n0rU\nkNfCzGYS/if4iLsvLXiEhTPktXD3ub2PzeyHwB3ufnshgyyAXP5/3AZcHXXsjiFcpv8/CxlkgeRy\nLV4HTgEei9rfjXCwRyUa9n2zKGsSWvfpLblcC+BfgVbgu2b2rJk9FVO4eZXjtSh7Of7/WEI4uut5\nwsEM33f3l+KKOV9y/J34CvB2M3se+D3wWXffFE/E+WVmNwN/CB/aSjO7ZE/vm1q7SUREsirKmoSI\niBQHJQkREclKSUJERLJSkhARkayUJEREJCslCRERyapYJ9NJhYomRf0Z+FO/l0539wFnhprZF4G0\nu39pDz73YsLJZq9FT9UBDwF/4+49wzzXl4Cn3f03ZvaAu58UPf+sux860hijczxIuP7O9uipZsKJ\nYR929zcHed8ngG3ufsuefL5UHiUJKUarhnkzHY3JPmng1+5+CYCZJYAHCZeZ/vZwTuTuV2YcnpDx\n/B4liEgauNTdHwYws4BwxdtPAZ8b5H1vBx4Yhc+XCqMkISXDzA4kvGE3ApOAb7j7f2e8XkW4nv4B\n0VP/4+7XRksx/C/hmjUp4J/d/b4BPqJv4TN3T5nZ44SrqWJmHyO8EaeBPxLO8u0Erhvg835EeEM+\nLHrv4+5+jJmlCBfeWwkscPc3zWwc8AIwE3gX8KWozHLg41lmBmcu0NYITACeiD7r3CjOuujPXwE1\nwPuAk6KVUJ8nnG07fYjrIaI+CSlKU6PlRXr/fDp6/lLgK+5+JHAy8G/93vd2oNXdDyNcq+ft0fP/\nBVzn7ocTLiN+jZk1DhaAmY0nXD30MTM7CPg8b23stAO4Ejgmy+elCZu//gHA3Y/pPW/UdHUrcG70\n1NnArwiXVfl34N3R+e4h3DipvwC41swWRzf8x6Oy34xqP5cB74022LkK+McoAdwOfMHd742uxw+G\ncz2kcqkmIcVodZammU8Dp5rZ5whXNG2Ing8Ib8wvEq5Zcxfhnhq9zS+nRM9/OTquAuYSfqMm4xxn\nmNmz0eME4aY9t5jZ5cDtGavrfo+wxvIfWT5vKD8hXNL8O4SL0X0eOJqwNvGgmUG4FefGAd7b19xk\nZscAvwDu7N1Ex8zOin4OI2zqGmhznVyuhwigJCGl5WeEN847CHcgOz/zRXffZGYHEDbbnAY8Ex0n\ngJPcfQuAmU0B+q98mQZu6+2T6Kf/vhQJoGqQzxuUu//RzMZF+3BPc/cnzOxM4FF3PzOKcQyQbc+D\nIDrP49F+xT+OloqvJ1wV9XrC/pTnCJvF+svleogAam6S0nIKcKW730G4V3FvBzPR4/cBN7j7bwl3\nH9tO2A9xP2EHNNFN/HnC9vpMg21Q9CDht/PW6PjjwP2DfF6mnmi57v5uJOwXuDk6fhI4JmNjoH8F\nvpYlnsyO+v8krFF9krD/pIew2epBwsTV+9ndhH0dkNv1EAGUJKQ4ZRut9EXgUTP7I/Buws7dOVH5\nNHAnsMvM/kR40/2Fu79IuOf10Wb2HOFN+SPuvmOAzxzwc939BcIb70Nm9jLhsNP/M8jnZboNWBzV\nDDLPfyNwMHBD9BlrCXcVvDVa0noBYQf0oNy9E/gXwqTyKrAYWELYud5G2IQF4RLZnzezD+R4PUQA\nLRUuIiKDUE1CRESyUpIQEZGslCRERCQrJQkREclKSUJERLJSkhARkayUJEREJCslCRERyer/Ac/p\nrqBi1DKGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x118b9cf10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We are going to pass in the true y, as well as the predicted *PROBABILITIES*\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y, y_pred_prob[:,1])\n",
    "roc_auc = metrics.roc_auc_score(y,y_pred_prob[:,1]) \n",
    "plt.plot(fpr,tpr) \n",
    "plt.plot(np.linspace(0,1,1000),np.linspace(0,1,1000),\":\")\n",
    "plt.fill_between(fpr,tpr,0,alpha=.2)\n",
    "plt.xlabel(\"False Positive Rate\"), plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"ROC Curve: %.2f AUC\" % roc_auc )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from the top right, we have a threshold of 0. All of our predictions are \"YES HEART DISEASE\". Therefore, we are never wrong when they do have it (100% True Positive Rate), but we're mostly wrong when they don't have it (100% False Positive Rate)\n",
    "\n",
    "At the bottom left, we have our threshold of 1.0. All of our predictions are \"nope, go home you're fine.\" Therefore, we are always wrong when they do have heart issues (0% True Positive Rate), but we're never wrong when they don't have it (0% False Positive Rate)\n",
    "\n",
    "The curve traces out this path. Ideally, with a perfect model, we would have a point on the top left corner with 100% TPR and 0% FPR. Very good binary classification models have steep slopes in the beginning of the curve.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "The dotted line represents when TPR == FPR. Models with curves above this line imply models better than random, while lines below the line imply models worse than random choice. \n",
    "\n",
    "\n",
    "The AUC (Area Under Curve) is the integral of the ROC curve. It can be used as a rough comparison measure when comparing multiple classifiers. An AUC of 1 implies a perfect model (i.e. the integral == 1 across the entire curve). In addition, AUC is a better metric than Accuracy for imbalanced classes. \n",
    "\n",
    "-----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Metrics / Reports\n",
    "\n",
    "It turns out the ROC curve was originally developed by electrical engineers during World War 2 for detecting enemy objects on radar screens (Makes sense why they care about TPR and FPR).\n",
    "\n",
    "A few more Metrics to help you diagnose issues:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[143,  21],\n",
       "       [ 35, 104]])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confusion Matrix\n",
    "metrics.confusion_matrix(y,y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "      No HD       0.80      0.87      0.84       164\n",
      "         HD       0.83      0.75      0.79       139\n",
      "\n",
      "avg / total       0.82      0.82      0.81       303\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification Stats\n",
    "print metrics.classification_report(y,y_pred,target_names=[\"No HD\",\"HD\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}