modeling_staff_picks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# standard imports\n",
    "import datetime\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from subprocess import check_output\n",
    "\n",
    "# third party imports\n",
    "import matplotlib.pyplot as plt # MATLAB like plotting framework\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # Data ETL, csv file I/O\n",
    "import sklearn.metrics as skm\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# src directory imports\n",
    "module_path = os.path.abspath(os.path.join('src'))\n",
    "if module_path not in sys.path:\n",
    "    sys.path.append(module_path)\n",
    "\n",
    "from src.helper_functions import load_dataset, plot_roc, plot_confusion_matrix, CleanTextTransformer\n",
    "\n",
    "%matplotlib inline\n",
    "pd.options.mode.chained_assignment = None  # default='warn'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise I will begin to model the likelihood of being a staff_pick for a given campaign. I will start with a tradiitional logistic regression model and examine the feature space using the coefficients. Accuracy will be assessed using the scoring function provided by the sklearn module for each model. For binary classification problems such as what we have here, this will be accuracy. The other metrics I will use are Area under the curve (AUC) and the confusion matrix to examine true/false positive/negative rates.\n",
    "\n",
    "To begin we load the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = load_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    151234\n",
       "True      21599\n",
       "Name: staff_pick, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.staff_pick.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['success'] = (df['state'] == 'successful') # notice how I didn't convert to integer. This is not our target variable\n",
    "test_live = df[df.state == 'live']\n",
    "df_all = df[df['state'].isin(['successful', 'failed'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    82718\n",
       "True     73810\n",
       "Name: success, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.success.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>staff_pick</th>\n",
       "      <th>False</th>\n",
       "      <th>True</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>success</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>79279</td>\n",
       "      <td>3439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>56314</td>\n",
       "      <td>17496</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "staff_pick  False  True \n",
       "success                 \n",
       "False       79279   3439\n",
       "True        56314  17496"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(df_all.success, df_all.staff_pick)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    135593\n",
       "True      20935\n",
       "Name: staff_pick, dtype: int64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.staff_pick.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to convert staff pick to integer value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['staff_pick'] = df_all.staff_pick.astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classes here are heavily imbalanced. To properly perform analysis of a datset like this, we will want to use stratified sampling in our train_test_split step. Drop _id_ and other columns not useful in predicting the response variable at campaign creation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all.drop(['id', 'backers_count', 'pledged', 'usd_pledged'], axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From EDA we figured out we wanted to convert unix time series to a datetime series. I will ignore the state_changed_at column and backers count column since this would not be available during campaign creation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['launched_at'] = pd.to_datetime(df_all['launched_at'],unit='s')\n",
    "df_all['state_changed_at'] = pd.to_datetime(df_all['state_changed_at'],unit='s')\n",
    "df_all['created_at'] = pd.to_datetime(df_all['created_at'],unit='s')\n",
    "df_all['deadline'] = pd.to_datetime(df_all['deadline'],unit='s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we aren't interested in a longitudinal model at this time and are working with panel data, we will not be using these dateime features in the models. Instead we will use the difference in these datetime columns as continuous numeric values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['launched-created'] = (df_all.launched_at - df_all.created_at).dt.components.days\n",
    "df_all['deadline-launched'] = (df_all.deadline - df_all.launched_at).dt.components.days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "datetime_columns = ['launched_at', 'created_at', 'deadline', 'state_changed_at']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['goal', 'static_usd_rate', 'launched-created', 'deadline-launched']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_features = df_all.describe().columns\n",
    "baseline_features = baseline_features.tolist()\n",
    "baseline_features = baseline_features[:]\n",
    "# del baseline_features[baseline_features.index('staff_pick')]\n",
    "baseline_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df_all.loc[:, baseline_features]\n",
    "y = df_all.staff_pick.astype('float')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=67, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First Model: Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n",
       "          fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "          multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression(class_weight='balanced')\n",
    "lr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75257139206541879"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "v_probs = lr.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FFUXwOHfoUtvkd47irQAAlIEC9iwiwUrIgpi/ey9\ndwEFAUGxgwUFBUGKUqX33kuoCb0GkpzvjzvEGJPNJmR3s8l5nycP2Z3ZmTNJ2LNzy7miqhhjjDEA\nuUIdgDHGmKzDkoIxxphElhSMMcYksqRgjDEmkSUFY4wxiSwpGGOMSWRJwWRpInKniMwI0rnqiMhi\nETksIn2Ccc7MJiIrRKR9EM5TVURURPIE+lwmuCwpmEQisllEjovIERHZJSLDRaRwsn1aicgU743z\noIj8KiL1k+1TVET6ishW71gbvMelAxz/XyLS/QwO8QTwp6oWUdX+mRVXMKnqOar6V6jjSEpE2otI\nVKjjMP6xpGCSu1JVCwONgMbA06c3iEhL4A9gNFAeqAYsAWaKSHVvn3zAZOAcoBNQFGgJxADNg3cZ\nGVIFWBHqIIwJKVW1L/tCVQE2AxclefwOMDbJ4+nAwBRe9zvwpfd9d2A3UDgd51WgD7ARlzzeBXJ5\n2+4EZiTZtxUwDzjo/dvKe/51IB44ARwBPk7lXFfh3vgPAH8B9bznpyR7fe0UXlsS+BzYAewHfvGe\nLwH8BkR7z/8GVEzyur+A14BZ3rF/BUoB3wCHvOuo6ufPo4YX615v2zdA8ZR+h8BZwBdeTKtwd0JR\nyfZ9HFjq/TxHAgVS+bnlBt7zzrkR6OXFmcfbfpd3jsPe9vu85wsBx4EE79qP4D5QNAf+9n4PO4GP\ngXyh/j9gX2pJwb7++Ur2hlIRWAb08x4X9N40L0zhdXcBO73vRwBfpPO8CvzpvelWBtYC3b1td+Il\nBW/7fqAbkAe42Xtcytv+1+nXpXKe2sBR4GIgr/cmuf70m5Efrx/rvXGW8F7fznu+FHCd9zMqAvyA\nlzCSHHe994ZeDFjpXeNF3nV8CXzu58+jphd/fiACmAb0TeV3+BYw1Yu3Iu7NP3lSmOu9SZf03tR7\npnLtPYHVQCVv3z/5d1K43Ls+AdoBx4Am3rb2Sc/rPdcUON+7/qreuR8O9f8B+1JrPjL/8YuIHAa2\nAXuAF73nS+KaG3em8JqdwOn+glKp7JOWt1V1n6puBfri3vCTuxxYp6pfqWqcqn6He6O60s9z3IS7\n85moqqdwn3zPwt19+CQi5YDOuDfN/ap6SlWnAqjqXlX9SVWPqeph3F1Lu2SH+FxVN6jqQdyd1QZV\nnaSqcbgk0jjZ/in+PFR1vRd/rKpGAx+kcK7TbgTe8OKNAlLqJ+mvqjtUdR/uDqaRj2P1VdVt3r5v\nJt2oqmO961Pv5/IH0CaVY6GqC1R1tvd73AwM9nEdJogsKZjkrlbVIrhPd3X5581+P64JoFwKrymH\na1YA16yR0j5p2Zbk+y24T6/Jlfe2kWzfCn6e41+vV9UE77z+vL4SsE9V9yffICIFRWSwiGwRkUO4\nT+/FRSR3kt12J/n+eAqP/9WhTyo/DxEpIyIjRGS7d66v+ed3lFz5ZMfZlsI+u5J8fyyFOFI71r9+\nDyLSWURmi8g+ETkAXOYjLkSktoj85g1oOAS84Wt/EzyWFEyKvE97w3GfplHVo7g24BtS2P1GXOcy\nwCTgUhEplM5TVkryfWVcu31yO3CdwSTbd/vpsNM4x79eLyLinXd7qq/4xzagpIgUT2HbY0AdoIWq\nFgXanj6FH8dNTWo/jzdw19nAO9dtPs6zE9dslNIx02tnCjEBICL5gZ9wfytlVLU4MC5JXCn9Xj7B\n3eXV8q7jGc7s52UyiSUF40tf4GIRaeg9fgq4Q0T6iEgRESkhIq/hRhe97O3zFe4N9CcRqSsiuUSk\nlIg8IyKX+TjX/7zjVQIewrXdJzcOqC0it4hIHhG5CaiP69gF9+m7uo9zfA9cLiIdRSQv7s08FtcB\n7JOq7sQ1+wz04swrIqff/IvgPu0fEJGS/NPkdiZS+3kUwXXWHhSRCsD/fBzje+Bp7zgVgN5nEM/3\nQB8RqSgiJXB/C6flw/VxRANxItIZuCTJ9t1AKREpluS5IrhO9iMiUhe4/wxiM5nIkoJJlddm/SXw\ngvd4BnApcC3uk+MWXFv4Baq6ztsnFteBuhqYiPuPPxfXNDDHx+lGAwuAxbgO3WEpxLMXuAL3Zr4X\n11F8haqebrrqB1wvIvtF5D/t56q6BvfJ+iNcc9eVuCG4J/37idANOOVd2x7gYe/5vri+iRhgNjDe\nz+P5ktrP42WgCW600FhglI9jvAJEAZtwd3A/4pJgRnwKTMANQV6Y9LxeP0ofXOLYD9wCjEmyfTXw\nHbBRRA6ISHncqKdbcKOVPiXlDwEmBETVFtkxoSUiimtGWB/qWLKCQP08ROR+oKuqWoeuSZXdKRiT\nTYlIORFp7TXh1cHdYf0c6rhM1mZ1S4zJvvLhhnpWw00SGwEMDGlEJsuz5iNjjDGJrPnIGGNMorBr\nPipdurRWrVo11GEYY0xYWbBgQYyqRqS1X9glhapVqzJ//vxQh2GMMWFFRJJXA0iRNR8ZY4xJZEnB\nGGNMIksKxhhjEllSMMYYk8iSgjHGmEQBSwoi8pmI7BGR5alsFxHpLyLrRWSpiDQJVCzGGGP8E8g7\nheG4hdtT0xmo5X31wNVXN8YYE0IBSwqqOg3Y52OXLrjF3lVVZ+NWqsrIil3GGJOtzVu7m1ndH2fR\nT38E/Fyh7FOowL+X94silWURRaSHiMwXkfnR0dFBCc4YY0JtY/QRut4/kLPatqbVsPeJHflDwM8Z\nFjOaVXUIMAQgMjLSKvgZY7KVk3EJ7D0ay4rth5i8ejdx8cqmqBja/zCEr2f/yJEixTn6zQjOv+Wm\ngMcSyqSwnX+v+VoR/9bKNcaYsKWqrNl9mN2HYpm1PoYv/97C8VPx/9qnadRKPvjjI6pEbyP6hluI\nGPwxlCgRlPhCmRTGAL1FZATQAjjorYNrjDHZiqqycOt+pq+Loe+kdf/aVq9cUeqVK0KjSsWpmi+B\nBgPfpvi3nyKVK8OECURcckkqRw2MgCUFEfkOaA+UFpEo3GLmeQFUdRBuEfbLgPXAMeCuQMVijDHB\ntnXvMSau2s345TuZt3l/4vMVS5xFw0rFubpRBZpXLUmxgnndhgkToEcP2LYNHnwQXn8dChcOetwB\nSwqqenMa2xXoFajzG2NMsB08doq/N8bw4HeLOBX/T/dnZJUSXFj3bG5uXpmShfL9+0X79sGjj8IX\nX0DdujB9OrRuHeTI/5FmUhCRRkAboDxwHFgOTFbVgwGOzRhjwsKMdTE88v1iog/HJj53TvmiPN25\nHpFVS1Agb+6UX/jTT9CrF8TEwLPPwnPPQYECQYo6ZakmBRHpBjyE6/xdAGwBCgAXAc+LyELgRVWN\nCkagxhiTVZw4Fc+YJTsYNHUDhfLlYdl29xn5uiYVaVGtJK1qlqJiiYKpH2DnTujdG0aNgiZNYPx4\naNQoSNH75utOoSTQTlWPprRRRCKBerj5BcYYk+2t3HGIDyauZdKq3YnPlS6cj3suqMa9bapTtlga\nn/JVXTPRI4/A8ePw1lvw2GOQJ+vMDkg1ElXt5+uFqmrLnxljsr2EBOWLvzczauH2xDuCumWL0K1l\nFTqfW+6/fQSp2bzZdSRPnAht2sDQoVC7dsDizihfzUcf+Hqhqj6a+eEYY0zWcODYSd77Yw1fz94K\nQImCeenRtjq3tqhMlVKF/D9QfDwMHAhPPw0iMGAA9OwJubJmkWpf9ywrghaFMcZkAarKD/Oj+HFh\nFIu3HeBkXAK1yxTmuiYV6d6mOrlzSfoOuGoVdO8Os2ZBp04weDBUrhyY4DOJr+ajYcEMxBhjQmnG\nuhieGrWUqP3HAWhRrSTPXV6fBhWLpf9gp07BO+/AK6+4uQZffQW33uruFLI4X81HPwOp1hlS1WsD\nEpExxgRRfILyzoTVDJ66EYCe7Wrw6MW1yZcng807CxbA3XfD0qVw443w0Udw9tmZGHFg+Wo++jho\nURhjTJDFxsXz4cR1DJq6AYBmVUvwwY2NqFTSx1BSX44fh5dfhvfec0ng55/h6qszMeLg8NV8NDmY\ngRhjTKCpKoOmbuSXRdtZs/sw4DqQu7WsSp8ONcmTO4N3B9Omub6DdevgnntcYihePBMjDx5/ZjTX\nAF4H6uMmrwGgqllvLJUxxqRi18ETnP/mP59129WO4JrGFejSqDyS0bb+Q4fcqKKBA6FaNZg0CTp2\nzKSIQ8OfGRPDgdeA93BLaN6Fj74GY4zJahZs2c91n8wCoPO5ZXn3hoYUzn+GE8bGjXNDS6Oi3GS0\nV1+FQukYqppF+fNTKaiqE0TkPVXdADwnIvOB5wMcmzHGnJFdB0/w4pjlTFjhZiB/cGNDrm1S8cwO\nGhPjksDXX0P9+m646fnnZ0K0WYM/SSFWRHIBG0SkJ64WUpHAhmWMMRmnqtzzxXymrN4DQIMKxfjo\n5sZULX0Gn+RV4YcfXM2i/fvhhRfgmWcgf/5Mijpr8CcpPAIUAvrg+haKAncHMihjjMmIkfO2siTq\nIFNW7WHXoRMA9OvaiC6NUlz+3X87dsADD8Do0RAZ6foOzjsvEyLOetJMCqo6x/v2MNAtsOEYY0z6\nqSq3DZvDzPV7AWhetSS9OtTk5maVMj6iyB0Yhg2Dxx+H2Fh49114+OEsVcAus/kz+mg80FVVD3iP\nSwBfq+rlgQ7OGGPSkpCgXDVgBsu3H+KCmqXpf3Nj/4vU+bJxI9x7L0yZAu3auQJ2NWue+XGzOH/S\nXZnTCQFAVfeLSPkAxmSMMX6ZujaaOz6bC8D51Uvy1T3NMz689LT4eOjf3y16kyePq1fUvXuWLWCX\n2fxJCgkiUvH0YjoikrWrORljsr0Tp+K55dPZLNzqPq92v6Aaz15e78wTwooVbvLZnDlw+eUwaBBU\nPMPRSmHGn6TwAjBTRKYAArQH7g9kUMYYkxJV5du5W/lo8np2HTrBZQ3K8lSnelQulcHSFKedPOkW\nvHntNShWDL79Frp2DYsCdpnNn47msSLSHGjpPfWEqu4JbFjGGPMPVeWr2Vv46u8trNtzBIBXrz6X\nbudXOfODz5vnCtgtXw433wz9+kFExJkfN0z524V+IVBDVV8XkUoi0lRVFwQyMGOMUVXGLdvFz4ui\nmLRqD6UL5+ONaxrQpVF5Cp3pjORjx9xcgw8/hHLlYMwYuPLKzAk8jPkz+uhjIC/QFjdP4SgwCGgW\n2NCMMTnRkdg4lkYdYNHWA7w7YU3i89c0rsDr15xLwXyZMBz0r79c5/GGDW6JzHfecc1Gxq87hVaq\n2kREFgGo6j4RyYTxXsYY4+w8eJwxi3fwzZytRO0/RoJXXa1u2SJ0rHc23c6vStliBXwfxB8HD8IT\nT8CQIVCjhhtueuGFZ37cbMSfpHDKK3OhACJSCkgIaFTGmGxNVZm3eT9Low7w69KdLNnmRhFVLHEW\nd7SqSrvaETSsWJwSmTHf4LTffnMF7HbuhMcec6uiFTzDDupsyJ+kMAD4CYgQkZeBG4GXAxqVMSbb\nSEhQdh8+wda9x9gQfZQVOw4yc30Mm/ceA6BGRCGe6FSHS+qXoUZE4TMfVppcdDQ89BB89x2cey6M\nGgXNm2fuObIRf0YffSkiC4CLcENSb1DV5QGPzBgTdnYcOM6kVbuZsS6GbfuPs+vgcWLjEjh2Mj5x\nn8L589Csagl6tqvBxfXLULJQvsxPBOBKVHz3HfTp49Y9eOklt/ZBPmv99sWvHhtVXQGsABCRoiLy\npKq+HdDIjDFZXkKCMnVdNL8v28nmmGPM3bwPgKIF8lC3XFEK5y9M1VKFaFipOJVLFqRa6UJUKH4W\nuXIFePx/VBTcf79rMmre3NUvOvfcwJ4zm0g1KYhIBeAZoALwCzASeAm3yM4PwQjOGJM1HTsZx+/L\ndvH2+NXsORwLuETQ+8KanFuhGB3qnp3xhe/PREICfPop/O9/EBcHH3zg7hRy5w5+LGHK153Cl8Df\nwDjgUuBxYCXQWFW3ByE2Y0wWsXjbAcYu3cGy7QfZuvcYOw+dQBXqlyvK45fW4fIG5c583sCZWr/e\nFbD76y/o0OGfEUYmXXz9Fkur6nPe92NFZDtws6rG+3iNMSab2HHgOOOW7eT7+dtYu9vNIm5cuTjn\nVy9F5VIFaVqlBK1qlCZ3oJuC0hIXB337wvPPu/6CTz919YtyYImKzOAztYtIEVznMkAMUFC8HiFV\nPZTWwUWkE9APyA0MVdW3km0vBnwNVPZieU9VP0/vRRhjMsemmKN8PXsLf6zcxbZ9xwGoV64oj1xU\nm5uaVcqcuQKZadkylwDmzYOrroKBA6HCGS6ok8P5SgqlcJ3LSdPtSu9fxb2Rp0pEcuOGs14MRAHz\nRGSMqq5MslsvYKWqXikiEcAaEflGVU+m8zqMMRmUkKD8vXEvn83YxJQ1e8gtQtvaEXRtVpnLGpSj\n2pksYRkosbHwxhvuq0QJGDECbrzR7g4yQapJQVXPtF5sc2C9qm4EEJERQBf+SSzgkksR7+6jMLAP\niDvD8xpj0hAbF89j3y9h6tpoDp9w/+VKF85Hnw61uPX8ypxdJIvdESQ1Z467O1ixAm67zdUuKl06\n1FFlG4HsGaoAbEvyOApokWyfj4ExwA6gCHCTqv5ntrSI9AB6AFSubMs5GHMm1uw6zKV9pwHQ+dyy\n1ClbhKqlCtHp3LIUyJuFR+kcPer6Dfr2dU1EY8fCZZeFOqpsJ9QLjV4KLAY6ADWAiSIyPXl/haoO\nAYYAREZGatCjNCYb2LbvGM/8vIzp62IAuLBOBJ/c1jTEUflpyhQ3smjjRjf/4K23oGjRUEeVLQUy\nKWwHKiV5XNF7Lqm7gLdUVYH1IrIJqAvMDWBcxuQoC7bsY/DUjUxZvYfcuYQ7W1Xl1haVqVWmSKhD\nS9uBA27OwdChUKuWG27arl2oo8rWApkU5gG1RKQaLhl0BW5Jts9WoCMwXUTKAHWAjQGMyZgcYUP0\nEd4ct5qdB4+zYschCuTNxd0XVOOu1lUpV+ysUIfnn9Gj3V3B7t2usulLL8FZYRJ7GEt3UhCRZd63\nA1R1UGr7qWqciPQGJuCGpH6mqitEpKe3fRDwKjDcO6YAT6pqTHpjMsY4p+IT+GzGJt78fTUA1UoX\n4p4LqnFvm+pZbzhpavbscbOQR46E885zi99ERoY6qhwjI3cKDYEI4Py0dlTVcbgZ0UmfG5Tk+x3A\nJRmIwRiTzPzN+3h93CoWbT1Ao0rFGXhrE8oXD6NP1qrwzTeuoumRI2695CeegLx5Qx1ZjuLPymv3\nA9+q6kEAb3TQbmB0gGMzxvhhWdRBHv1+ceLaxY9eXJsHO9QMTOXRQNm61a118Pvv0LKlK2BXr16o\no8qR/LlTqAIsFJE5uCagSQGOyRiThgPHTvLsz8vZEH2E1bsOkze30P2CatzfvgalCucPdXj+S0iA\nwYPdHUFCAvTrB716WQG7EPJnPYWnROQZoDPQU0Q+Ab7DJYjNAY7PGJPMih0HuXHQ3xw9Gc95FYvx\nVOe6XHFeOSqWCLNVxNaudeskT58OF1/skkO1aqGOKsfzdz2FBBHZDGwGGgDlgNEiMk5Vnw5ceMaY\n0+ITlMHTNvDOeLeY/YBbmnD5eeVCHFUGxMXB++/Diy+60USffw533GElKrIIf/oUegF3AIeAYcCz\nqhrrrdu8HrCkYEwA7T0Sy08LoxgybSMxR07SvFpJ3rimATXPLhzq0NJvyRK4+25YuBCuuQYGDIBy\nYZjYsjF/7hTK40pmb0j6pHf3cFVgwjImZ1NVXhyzghnrYtgYcxSAhpWK82CHWtzRqmpog8uI2Fh4\n9VV4+20oVQp+/BGuuy7UUZkU+JMUKiRPCCIyXFXvtLWajcl88QnKC6OX882crZxTvij3tqnGZQ3K\n0bhyiVCHljGzZrkCdqtXw+23uwJ2JUuGOiqTCn+SwnlJH3jNRs0CE44xOZuqclm/6azZfZie7Wrw\nZKc64TW0NKkjR+DZZ+Gjj6BSJRg/Hi69NNRRmTT4WqP5SeApXGnrfaefxpW7HhaE2IzJMWLj4pm/\neT/vTFjDmt2HuapheZ7qXDfUYWXcxInQowds2eKGmL7xBhQJg1pLxuedwjvA+8CbuOQAgC3HaUzm\n+n7eNl4ft4qDx09RqlA+nr2sHt3bhOnQzP374bHH3IiiOnVg2jS44IJQR2XSwVdSqKmq60TkK+Cc\n00+evpVV1aUBjs2YbG33oRPcPGQ2G2OOUqRAHvrf3JiL65XhrHxhOnFr1Ch3VxAdDU8/DS+8AAXC\npN6SSeQrKTwF3INbUjM5BdoGJCJjcoC/N+yl59cLOHj8FNc3rchLV51D4fyhXt4kg3btgt694aef\noFEjGDcOGjcOdVQmg3wtx3mP92+b4IVjTPa26+AJ7vt6AUu2HaBIgTx8070FrWuG6VKSqvDll/DI\nI3DsmOs3ePxxK2AX5vyZvLYQV9bie1XdEviQjMmeVuw4yOX9ZwBQomBefunVmiqlCoU4qgzasgXu\nuw8mTIDWrd0iOHXDuGPcJPLnfvUG4CZgjIgcA0biEsSOgEZmTDYyZskO3hy3ijy5hKF3RNKmVgS5\nc4XhUNOEBBg4EJ7yxp589BE88ADkyhXauEym8acg3gbgDeANEamHK2vxnj+vNSYnO3Eqnl+X7GDw\ntI2s98paD7y1Ce3rnB3iyDJozRo3CW3mTDffYPBgqFIl1FGZTObXG7uIVARuxN0x5AGeDWRQxoS7\nXxZt59HvF5OgUD2iEI9eXJvubapRMF8YfpY6dQreew9efhkKFoQvvoBu3ayAXTblT5/CLKAw8ANw\nm6quC3hUxoQpVWX4rM28/OtK8uXORf+bG3PpOWXCd1byokWugN3ixXD99a65qGzZUEdlAsifjy33\nquqKgEdiTJjbc/gEj45cwoz1MTSpXJyPbwmz5TCTOnHC3Rm8+y5ERLjhptdeG+qoTBD4KnNxs6p+\nB3QUkY7Jt6tq/4BGZkwY+XFBFK/+tpLjp+J5oH0NHr24Nnlyh2nn64wZru9g7Vq46y639kGJMC3G\nZ9LN153C6b+CiBS2aQBiMSbsJCQo387dynO/LKdOmSIMuLVJeK5zAHD4sJuJPGAAVK0Kf/zhVkQz\nOYqvyWsDvW/HqurspNtE5PyARmVMGIiNi+emwbNZvO0AdcoU4fv7WlKsYJhO3Bo/3s072LYNHnoI\nXnsNCodpcjNnxJ8+hYFAk2TPDQCaZn44xoSHnxdF8cjIJQD0aFudxy6pTf48YVizaO9eePRRNzO5\nXj033LRly1BHZULIV59Cc6AlECEifZJsKgqE6cchY87c4m0HEhPCm9c24ObmlUMcUQaous7jXr1g\n3z547jn3lT9/qCMzIebrTqEQUNrbJ2m/wmHcLGdjcpx9R09y9YCZAHx/X0uaVwvDFcR27nTJ4Oef\noWlT13fQsGGoozJZhK8+hT+BP0Xkc1XdGMSYjMmSth84Tqe+0wD46p7m4ZcQVGH4cNdcdOIEvPOO\nK2aXJwwn1JmA8dV89L6qPga8LyL/GW2kqjZo2eQIuw6eoP+Udfy0IIq8uXMx6LamtKmV0qC8LGzT\nJrcS2qRJ0LYtfPop1K4d6qhMFuTrI8JI79+PgxGIMVlNfILy86LtPP6D6z+4KbISPdpVp0ZEGI3K\niY+Hjz+GZ55xResGDnSjjKyAnUmFr+ajud6/k08/JyLFgAqqujIIsRkTMvuOnuTu4fNYvO0AVUoV\n5IMbG9G0SphN4Fq5Erp3h7//hs6dXQG7SpVCHZXJ4vypfTQZuAbIDSwE9onIFFX9X6CDMybYDp84\nxaiF23n51xUkKHRtVonnrqgfXquinToFb78Nr74KRYrA11/DLbdYATvjF3/+0kuq6iERuQf4WlWf\nF5GlQJpJQUQ6Af1wCWWoqr6Vwj7tgb64Ya4xqtouPRdgTGYZOn0j70xYw8m4BCKK5GfQbU1oWiXM\nOpPnz3clKpYuhZtugv794ewwLdVtQsKfpJBHRCJww1Bf8PfAIpIbN8ntYiAKmCciY5I2PYlIcdzk\nuE6qulVE7K/XBF18gvLIyMWMWbKD8sUK8OJV53BRvTLhtQjO8ePw4ouuTlHZsvDLL9ClS6ijMmHI\nn6TwOjAVmKGqc0WkOrDJj9c1B9afHs4qIiOALkDS/ohbgFGquhVAVfekJ3hjMsPrY1cxZskOWlYv\nxSe3NaF4wXyhDil9pk51fQfr18O997qhpsWLhzoqE6bSHIKgqiNUtb6q9vAeb1RVfz6CVAC2JXkc\n5T2XVG2ghIj8JSILROT2lA4kIj1EZL6IzI+Ojvbj1MakLTYunld/W8lnMzfRtEoJvr23RXglhEOH\n4P77oX17t0zm5MkwZIglBHNG/OloLg3cDVRNuv/pJJEJ528KdATOAv4WkdmqujbpTqo6BBgCEBkZ\naRVazRn7aUEUj3lDTW9tUZnnr6gfXgvhjB0LPXvCjh1uMtorr0ChQqGOymQD/jQfjQZmAzOA+HQc\nezuQdPxbRe+5pKKAvap6FDgqItOAhsBajAmA4yfj6fXtQqas3kP5YgV4+rJ6XNmwfKjD8l9MDDz8\nMHzzDdSvDz/+CC1ahDoqk434kxQKeTOb02seUEtEquGSQVdcH0JSo4GPRSQPkA9oAXyYgXMZk6aE\nBKW3lxBuaFqR169pQL48YTKJSxVGjoQHH4QDB1yn8tNPWwE7k+n8SQq/i8glqvpHeg6sqnEi0huY\ngBuS+pmqrhCRnt72Qaq6SkTGA0uBBNyw1eXpvAZj0rTn0Ale+nUFk1fv4eWrzuGOVlVDHZL/tm+H\nBx6AMWMgMhI++wwaNAh1VCabElXfTfQish8oBhwDTgICqKqGZAB3ZGSkzp8/PxSnNmHoaGwcT41a\nxq9LdgBwQ9OKvHP9eeHRf6AKQ4fC44+7CWmvvuoWwLECdiYDRGSBqkamtZ8/f12lMyEeY4LqZFwC\noxdv5+3xa4g5EkuXRuW5q3U1GlYsFh4JYcMGN7z0zz/d6KJPP4WaNUMdlckB0kwKqhovIl2B6qr6\nhohUBMq12jQZAAAgAElEQVQACwIenTEZsHjbgcQ1D0oXzs+HNzXkmsYVQxyVn+LjoV8/t+BN3ryu\nXlH37lbAzgSNP0NSP8aVoGgLvIFrRhoENAtsaMak3y+LtvPwyMUAPHNZXe5sVS18OpOXL3clKubO\nhSuugE8+gYphksxMtuFP81ErVW0iIosAVHWfiITRDB+TE8xcH8Nbv69m2faDFMmfh5H3taR++aKh\nDss/J0/Cm2/C669DsWLw7bfQtasVsDMh4U9SOCUiuQAFEJFSuJFCxoTcqfgE7v96IZNW7SZ/nlzc\nFFmJhy6qRfniZ4U6NP/MnevuDpYvd5VM+/aFiDBbwMdkK/4khQHAT0CEiLwM3Ai8HNCojPHDwWOn\nuH7QLNbtOULHumfzwY2NKFYwb6jD8s+xY/DCC/Dhh1CunBtueuWVoY7KGL86mr8UkQXARd5TN9hc\nAhNqC7fu57pPZqEKt7eswitdzg11SP7780/Xebxxo1sF7e23XbORMVlAqj1wIlLAK3+Nqq4AxuKa\njaoHKTZjUrQs6iDXDnQJ4e3rGoRPQjh40CWBDh1cf8GUKTBokCUEk6X4ulOYANwLrBWRGsBc3LrN\n14lIC1V9NhgBGnPaqp2HeG3sSmau38vZRfIz8r6WVCsdJkXgfv3VVTTdudNNRnv5ZShYMNRRGfMf\nvpJCySTVSu8ARqjqAyKSH5gPWFIwQZF0EZzcuYQ7W1WlZ7salC1WINShpS062s1C/u47OPdcGDUK\nmjcPdVTGpMpXUkha/6ID8D6AqsaKiI0+MkGxYsdBLu8/A4CaZxfmk1ubUKtMkRBH5QdVlwj69HHr\nHrz8Mjz1FOSz0dwma/OVFFaIyFu4Cqe1gT8ARKQYrv6RMQE1edVu7vnC1bm6ulF5+nZtHOKI/BQV\n5ZqKfvvNlbUeNgzOOSfUURnjF19JoTvwCFAXt4byUe/5c4EPAh2YyblUlSd+XMoPC6KoVPIsht3R\njNrhcHeQkOBqFP3vfxAXBx984O4UcucOdWTG+C3VpOAlgddSeH4mMDOQQZmcbdTC7fywIIp2tSP4\n6JbGFC0QBnMP1q1zBeymTnWjiz79FKrbQD0TfnwNSf1FRDp7C+Ak31ZFRF4QkbsDG57JafYcduse\nlC6cn6F3RGb9hBAXB++9B+edB4sXu1LXkyZZQjBhy1fzUS/gMWCAiOwGooECuHkKW4EBqvpT4EM0\nOUXMkViavz4ZgP53NSZv7ixeyG7pUleiYv586NIFBg6E8mG0tKcxKfDVfLQdeBR4VERqAuWA48Aa\nVT0cpPhMDvHBxLX0n7yOfHly8VqXc7mwztmhDil1sbHwxhvuq0QJt0zmDTdYATuTLfi1hJOqrgfW\nBzgWk0Mt2LKP/pPXUbxgXr7p3oJzymfhGb6zZ7u7g5Ur4bbbXAG7UqVCHZUxmSaL35+b7C7mSCzX\nffI3AGN6XZB1E8LRo/Doo9CqFRw+DGPHwldfWUIw2Y4t9mpCZvq6aLoNmwu4GkaVS2XRsg+TJ7uR\nRZs2ufkHb70FRcNkrQZj0smvOwURyef1KxiTKf5cs4duw+ZStmgBBndryk3NKoc6pP86cMBVM73o\nIsiTxw03HTjQEoLJ1tJMCiJyObAMmOg9biQiPwc6MJM97T50gsd/WMJdn88DYPjdzbj0nLIhjioF\no0dD/fowfDg8+SQsWQJt24Y6KmMCzp/mo1eAFsCfAKq62O4aTEZE7T9Gp77TORIbx+XnleOpTnWp\nVDKLNRnt3u1mIX//PTRs6KqbNm0a6qiMCRq/luNU1QPy7+F2mtrOxqRk2tpoun8xn5PxCXxyaxM6\nNygX6pD+TRW++cZVND1yBF57DZ54AvJm8clzxmQyf5LCKhG5EcglItWAPsDswIZlspNxy3bywDcL\nKVkoH0O6NSWyaslQh/RvW7dCz57w++/QsqUrYFevXqijMiYk/Olo7g00xa26NgqIBR4KZFAme4g5\nEstdn8/lgW8WcnaR/Hx0c+OslRASEuCTT1wF06lToV8/mD7dEoLJ0fy5U7hUVZ8Enjz9hIhci0sQ\nxqTot6U7eP6X5ew/doprG1fg2cvrUapw/lCH9Y+1a93IounT4eKLYfBgqFYt1FEZE3L+JIXn+G8C\neDaF54xhybYDdBngiugWOysvY/tksQlpp0tav/giFCgAn38Od9xhJSqM8aSaFETkUqATUEFEkq6f\nUBTXlGRMoiOxcTz+/RLGr9gFQIe6Z/PmtQ0oUzQLLZm5ZAncfTcsXAjXXAMDBkC5LNbhbUyI+bpT\n2AMsB04AK5I8fxh4KpBBmfCy90gsnfpNJ/pwLO3rRPD8FfWpEVE41GH9IzbWjSZ66y1XluLHH+G6\n60IdlTFZkq8qqYuARSLyjaqeCGJMJowcPnGKXt8uJOZILG9f14AbIyshWakpZtYsV8Bu9Wq4/Xb4\n8EMomYU6u43JYvwZfVRBREaIyFIRWXv6y5+Di0gnEVkjIutFJNW7CxFpJiJxInK935GbkFuwZR8N\nXvqDOZv28f4NDbmpWeWskxCOHHFzDi64AI4dg/Hj4YsvLCEYkwZ/ksJw4HNAgM7A98DItF4kIrmB\nAd5r6gM3i0j9VPZ7G/jD76hNyH07Z2tiddNHLqrNtU0qhjiiJCZOhAYNoH9/6NULli+HSy8NdVTG\nhAV/kkJBVZ0AoKobVPU53Bt9WpoD61V1o6qeBEYAXVLY70HgJ1wfhsni4uIT6P3tQp75eRllixbg\n+/ta0qdjrVCH5ezf7zqSL7kE8ud3w00/+giKFAl1ZMaEDX+GpMaKSC5gg4j0BLYD/vwvqwBsS/I4\nCldDKZGIVACuAS4EmqV2IBHpAfQAqFw5C1bTzCFW7TxE537TAWhZvRRDbm9KkayyhvKoUe6uIDoa\nnn4aXnjBDTk1xqSLP3cKjwCFcOUtWgP3Andn0vn7Ak+qqs8hrqo6RFUjVTUyIiIik05t0mPK6t2J\nCeHeNtX49t4WWSMh7NoF11/vRhOVLQvz5rllMi0hGJMhad4pqOoc79vDQDdI/ISflu1ApSSPK3rP\nJRUJjPA6J0sDl4lInKr+4sfxTZBMWxvN3cPnU7RAHr7u3oLzKhYPdUiugN2XX8Ijj7iO5DfegMcf\ntwJ2xpwhn0lBRJrhmoFmqGqMiJyDK3fRAfcm78s8oJZXRG870BW4JekOqppYV0BEhgO/WULIOlSV\nP1bu5vEfliACo3tfQLXShUIdFmzeDPfdB3/8Aa1bw9ChULduqKMyJltItflIRN4EvgFuBcaLyEu4\nNRWWALXTOrCqxuGK6U0AVgHfq+oKEenp9U2YLOzQiVPc99UC7vtqAfEJyqj7W4U+ISQkuI7jc891\n8w8+/himTbOEYEwm8nWn0AVoqKrHRaQkrtO4gapu9PfgqjoOGJfsuUGp7Hunv8c1gZWQoDR9dSKn\n4pVHLqrNbedXDn0xu9WrXQG7mTPd8NLBg6FKldDGZEw25Kuj+YSqHgdQ1X3A2vQkBBOeYuPi6fjB\nVE7FK49fUpuHLqoV2oRw6pTrL2jYEFaudBPQfv/dEoIxAeLrTqG6iJyuhCpAtSSPUdVrAxqZCbr4\nBOXmIbPZFHOUrs0q0evCEK+6umiRm3eweDHccINrOipTJrQxGZPN+UoKySuGfRzIQExoHT5xiis/\nmsHmvcfo2a4GT3UOYTv98ePwyivw7rsQEeHmIFxzTejiMSYH8VUQb3IwAzGhs+/oSZq8OhGA/11a\nhwfa1whdMDNmuAJ2a9e6u4T33oMSJUIXjzE5jD+T10w2Nmnlbpq9PgmAey6oRq8La4amqN3hw9C7\nN7RpAydPuvpFw4ZZQjAmyPwpc2GyqR0HjtPz6wUUyJOLL+9pTtMqIaogOn68m3ewbZurbPraa1A4\nC63HYEwO4vedgohkoQV2zZn6bMYmWr01hbgE5cObGoUmIezd65bC7NwZChVyw0379rWEYEwIpZkU\nRKS5iCwD1nmPG4rIRwGPzATMi6OX88pvKwEY3K0pl5xTNrgBqMIPP0D9+vDtt/D8826kUcuWwY3D\nGPMf/jQf9QeuAH4BUNUlInJhQKMyAXEyLoHbhs1h7qZ9FC+Yl98evICKJQoGN4idO+GBB+CXX6Bp\nU1eqomHD4MZgjEmVP0khl6puSdb5GB+geEyAbIo5yoXv/QVAs6ol+OLu5hTMF8QuJVX4/HN49FG3\nZvI777hidnmsW8uYrMSf/5HbRKQ5oN4qaQ8Cfi3HabKGTTFHuW2oK3b7YIeaPHJRbXLlCuIIo40b\nXUfypEnQti18+inUTrN8ljEmBPxJCvfjmpAqA7uBSd5zJgz8uCCKx39YAkC/ro3o0sifqueZJD7e\nzUJ+9lnInRs++QR69IBcNhLamKzKn6QQp6pdAx6JyXTDZ27ipV9XUiBvLt65viFXNSwfvJOvXOkm\noc2e7UYXDR4MlSql/TpjTEj5kxTmicgaYCQwSlUPBzgmc4ZOnIrn6gEzWb3rMFVLFWTUA60pWShf\ncE5+8iS8/baba1CkCHz9NdxyC4RiQpwxJt38WXmthoi0wi2S87KILAZGqOqIgEdn0i0uPoG6z48H\noH2dCAbd1pQCeXMH5+Tz57u7g6VLoWtX6NcPzj47OOc2xmQKvxp3VXWWqvYBmgCHcIvvmCxmyLQN\n1Hz2dwBubl6J4Xc1D05COH4cnngCWrSAmBgYPRq++84SgjFhKM07BREpjFtwpytQDxgNtApwXCad\n+k1ax4eT1pI7l/Dc5fW4q3W1tF+UGaZOdYvfrF8P997rhpoWzwJrOBtjMsSfPoXlwK/AO6o6PcDx\nmAyYsno3H05aS8USZ/HzA62JKBKEiiSHDsGTT8KgQVC9OkyeDB06BP68xpiA8icpVFfVhIBHYjLk\nrzV7uHv4fMoWLRC8hDB2LPTsCTt2uMlor7ziahcZY8JeqklBRN5X1ceAn0REk2+3lddCb8i0Dbwx\nbjXlihVg+F3NA58QYmLg4Yfhm29c3aIff3T9CMaYbMPXncJI719bcS0Lmr95H2+MW03RAnn4oWfL\nwNYwUoWRI+HBB+HgQXjxRXj6achvhXONyW58rbw21/u2nqr+KzGISG/AVmYLkfV7DnP9oL8pXjAv\nEx5uS5miBQJ3su3bXQG7MWOgWTO38E2DBoE7nzEmpPwZknp3Cs/dk9mBGP8s2rqfawfOAuD1qxsE\nLiGouhpF9eu7VdDeew/+/tsSgjHZnK8+hZtww1CricioJJuKAAcCHZj5r2lro7n9M3cDNySQ6yBs\n2OCGl/75J7Rv75JDzZqBOZcxJkvx1acwF9gLVAQGJHn+MLAokEGZ/9q27xi3fzaXCsXP4uvuLahW\nOgCjfeLj3Szk556DvHlhyBA3Q9kK2BmTY/jqU9gEbMJVRTUhdPjEKdq88ycAw+9qFpiEsHy5SwBz\n58KVV7qKphWCWFHVGJMlpPoRUESmev/uF5F9Sb72i8i+4IWYsx2NjeOGQX8D0KdjLWqVKZK5Jzh5\nEl5+GZo0cesefPedK1NhCcGYHMlX89HpJTdLByMQ819HY+M458UJANx2fmUevTiTF6aZO9fdHSxf\n7iqZ9usHpe3XbUxOluqdQpJZzJWA3KoaD7QE7gNs+mqAxScol/V3VUXua1ed167OxFE/x47BY49B\ny5awfz/8+qubkGYJwZgcz58exF9wS3HWAD4HagHfBjQqQ58Ri9iy9xjXNqnA053rZd6B//zTDSv9\n4AO3CtrKlXDFFZl3fGNMWPMnKSSo6ingWuAjVX0E8KvBWUQ6icgaEVkvIk+lsP1WEVkqIstEZJaI\nNExf+NlPQoLy9vjVjF26k9y5hPdvyKQfycGDLgl06OBGE/31l+tMLlo0c45vjMkW/FqOU0RuALoB\nV3vP5U3rRSKSGzeU9WIgCreC2xhVXZlkt01AO1XdLyKdgSFAji2mo6p0/3I+U1bvoXrpQgy5vSmS\nGSuW/fqrK2C3axf873/w0ktQMIBlMYwxYcvfGc0X4kpnbxSRasB3fryuObBeVTeq6klgBG5dhkTe\n4j37vYezcXMicqyXf13JlNV7uKR+Gf54pC01zz7DkUbR0XDzzXDVVVCqFMyZ49Y7sIRgjElFmklB\nVZcDfYD5IlIX2Kaqr/tx7ArAtiSPo/Dd7HQP8HtKG0Skh4jMF5H50dHRfpw6/CzedoDhszZTPaIQ\nn9zWlDy5z2DCmKrrOK5XD376yZW2nj8fIiMzL2BjTLbkz8prbYCvgO2AAGVFpJuqzsysIETkQlxS\nuCCl7ao6BNe0RGRk5H/KeIe7dbsPc+fnrnzF61c3IHeuM2gy2rYN7r/frXlw/vmugF39+pkUqTEm\nu/OnT+FD4LLTfQEiUg+XJNL62LkdN5z1tIrec/8iIucBQ4HOqrrXn6CzkxU7DnLNwFmgMLLH+bSo\nXipjB0pIcGUpnnjClavo2xd694bcQVij2RiTbfiTFPIl7RxW1VUiks+P180Danl9ENtxxfVuSbqD\niFQGRgHdVHWt/2FnD+OX76Tn1wsB+OLu5hlPCOvWuQJ2U6dCx44uOVSvnomRGmNyCn+SwkIRGQR8\n7T2+FT8K4qlqnLfuwgQgN/CZqq4QkZ7e9kHAC0ApYKA3yiZOVXNEw/fG6COJCWFcnzbUL5+BoaFx\ncfDhh/DCC27Bm2HD4K67IDNGLBljciRR9d1ELyIFcB3Np9v7p+PmK5wIcGwpioyM1Pnz54fi1Jlm\nyurd3D3cXcOwOyLpWK9M+g+yZIkrUbFgAVx9NQwYAOXLZ3KkxpjsQkQW+POh2+edgog0AGoAP6vq\nO5kVXE42evF2HhqxGICBtzZJf0KIjYXXXoO33oKSJeH77+H66+3uwBiTKXwtsvMMbkTQQqCZiLyi\nqp8FLbJsaNaGGB4asZiC+XLzxyNt07+u8t9/u7uDVavg9ttdqYpSGeyHMMaYFPgaDH8rcJ6q3gA0\nA+4PTkjZ04It+7jzs3mUKpSPL+5unr6EcPQoPPIItG7tvv/9d/jiC0sIxphM56v5KFZVjwKoarSI\n2PJbGXTsZBw3DZ5NXIIyqFtTmlUt6f+LJ01yI4s2b4YHHnDNRkUyeU0FY4zx+EoK1ZOszSxAjaRr\nNavqtQGNLJs4FZ/ARe9PJS5BeahjLf8Twv798Pjj8NlnULs2TJsGbdoENlhjTI7nKylcl+zxx4EM\nJDs6fOIUl344jR0HT/DoxbXp07GWfy/8+Wd3VxAdDU89BS++CAUKBDZYY4zB9xrNk4MZSHZzMi6B\nBi/9AUCnc8ryYIeaab9o92548EH44Qdo1MiVqmjSJMCRGmPMP6yfIABUlUe+d8NOe7StzqBuaZTA\nVoUvv3QF7EaPhtdfd0tlWkIwxgSZPzOaTTrdNmwOM9fv5cI6ETzdua7vnbduhfvug/HjoVUrNyu5\nbhqvMcaYAPH7TkFE8gcykOxAVencb3piQhh2R7PU7xASEtws5HPOgenToX9/968lBGNMCKWZFESk\nuYgsA9Z5jxuKyEcBjywM3T18Hqt2HqJI/jwM7hZJrtRKYK9ZA+3auSqmLVvC8uWuLyGXteYZY0LL\nn3eh/sAVwF4AVV2CW4nNJPHMz8v4c000BfLmYuELF5MvTwo/2lOn3DyDhg1dIvj8c5gwAapWDXq8\nxhiTEn/6FHKp6pZkzSDxAYon7Kgq7d/7iy17jwEw55mLyJvSqmmLFrkSFYsWwXXXwccfQ9myQY7W\nGGN88+dOYZuINAdURHKLyMNAjlv7ICUHj5/iyo9nsGXvMeqXK8rKVy6l2Fl5/73TiRPw7LPQrBns\n2AE//ui+LCEYY7Igf+4U7sc1IVUGdgOTsDpIALR+awpHYuO4/Lxy9Lup0X/XVZ45E7p3h9Wr4c47\n4f33XWVTY4zJotJMCqq6B7dqmvGoKnd+Ps8lhAblGHBLsvkER47AM8+4JqLKlV2/wSWXhCZYY4xJ\nhzSTgoh8CvxnJR5V7RGQiLK4hATl9s/mMmN9DJFVSvDhTY3+vcMff0CPHm7+Qe/e8MYbULhwaII1\nxph08qf5aFKS7wsA1wDbAhNO1hYbF0/H96cStf84bWtHMPzOZv8MO923Dx57DIYPhzp13JyD1q1D\nGq8xxqSXP81HI5M+FpGvgBkBiyiL2hRzlAvf+wuAyxqU5aObm/yTEH76CXr1gpgY16n83HNWwM4Y\nE5YyUuaiGpCBRYXD1wd/rKH/lPUAPHZxbR48Xe10507XRDRqFDRu7EpVNGrk40jGGJO1+dOnsJ9/\n+hRyAfuApwIZVFaRkKDcNORv5m3eT42IQrx69bm0qlHaFbD74gu3Gtrx425C2mOPQR4rJWWMCW8+\n38XEzVhrCGz3nkpQ1f90OmdH8QnKPV/MY97m/TSsWIxv7z2fQvnzuBXQevSAiRPhggtg6FDXh2CM\nMdmAz6Sgqioi41T13GAFlBUcOxlHo1cmcjIugVpnF+bnB1qTC4WPPoKnnwYRV8yuZ0+rV2SMyVb8\nae9YLCKNVXVRwKPJIjr3m87JuAT6dKxFnw41ybVmtZuENmsWdOoEgwZBlSqhDtMYYzJdqklBRPKo\nahzQGJgnIhuAo7j1mlVVs+UKMO9OWM2Wvce4tnEFHm1fDd56E155xc01+PJLuO02d6dgjDHZkK87\nhblAE+CqIMUScs//spyvZm8B4Llyx129oiVL4MYb3XoHZXLUoCtjTA7kKykIgKpuCFIsIaOq/O/H\npfy4IIrW5Qvy1fbx5OrwPkREwM8/w9VXhzpEY4wJCl9JIUJEHk1to6p+EIB4gk5VuXrATJZEHaTZ\ntuV8/uNQcm1Y78pcv/sulCgR6hCNMSZofCWF3EBhvDuG7OjEqXiuHjCTqC27GbZkJB3//MkteDNx\nIlx0UajDM8aYoPOVFHaq6itBiyTIlkUd5MqPZ9B+w3y+mjyQ0gei4eGH4bXXoFChUIdnjDEhkWaf\nQnZ031fzmTN/HX2nDOXq5VPQ+vWRsb+49ZKNMSYH8zXzquOZHlxEOonIGhFZLyL/KY0hTn9v+1IR\nCfgw1/99v5g8P/7IpKH302X1NHj+eWThQksIxhiDjzsFVd13JgcWkdzAAOBiIAo312GMqq5Msltn\noJb31QL4xPs3ICZPXsRFzz3Apetmk9CkKfL5Z3DeeYE6nTHGhJ1A1mhoDqxX1Y2qehIYAXRJtk8X\n4Et1ZgPFRaRcIIJZMuRbml3ehvabFnL8tTfJNWe2JQRjjEkmkEmhAv9ejCfKey69+yAiPURkvojM\nj46OzlAweerWYUvt89gwaSZnPfuUVTQ1xpgUhMU7o6oOAYYAREZGZqhK6zltm8LSWZkalzHGZDeB\nvFPYDlRK8rgi/5TgTs8+xhhjgiSQSWEeUEtEqolIPqArMCbZPmOA271RSOcDB1V1ZwBjMsYY40PA\nmo9UNU5EegMTcLOjP1PVFSLS09s+CBgHXAasB44BdwUqHmOMMWkLaJ+Cqo7DvfEnfW5Qku8V6BXI\nGIwxxvjPlg0zxhiTyJKCMcaYRJYUjDHGJLKkYIwxJpG4vt7wISLRwJYMvrw0EJOJ4YQDu+acwa45\nZziTa66iqhFp7RR2SeFMiMh8VY0MdRzBZNecM9g15wzBuGZrPjLGGJPIkoIxxphEOS0pDAl1ACFg\n15wz2DXnDAG/5hzVp2CMMca3nHanYIwxxgdLCsYYYxJly6QgIp1EZI2IrBeRp1LYLiLS39u+VESa\nhCLOzOTHNd/qXesyEZklIg1DEWdmSuuak+zXTETiROT6YMYXCP5cs4i0F5HFIrJCRKYGO8bM5sff\ndjER+VVElnjXHNbVlkXkMxHZIyLLU9ke2PcvVc1WX7gy3RuA6kA+YAlQP9k+lwG/AwKcD8wJddxB\nuOZWQAnv+8454ZqT7DcFV633+lDHHYTfc3FgJVDZe3x2qOMOwjU/A7ztfR8B7APyhTr2M7jmtkAT\nYHkq2wP6/pUd7xSaA+tVdaOqngRGAF2S7dMF+FKd2UBxESkX7EAzUZrXrKqzVHW/93A2bpW7cObP\n7xngQeAnYE8wgwsQf675FmCUqm4FUNVwv25/rlmBIiIiQGFcUogLbpiZR1Wn4a4hNQF9/8qOSaEC\nsC3J4yjvufTuE07Sez334D5phLM0r1lEKgDXAJ8EMa5A8uf3XBsoISJ/icgCEbk9aNEFhj/X/DFQ\nD9gBLAMeUtWE4IQXEgF9/wroIjsm6xGRC3FJ4YJQxxIEfYEnVTXBfYjMEfIATYGOwFnA3yIyW1XX\nhjasgLoUWAx0AGoAE0VkuqoeCm1Y4Sk7JoXtQKUkjyt6z6V3n3Di1/WIyHnAUKCzqu4NUmyB4s81\nRwIjvIRQGrhMROJU9ZfghJjp/LnmKGCvqh4FjorINKAhEK5JwZ9rvgt4S12D+3oR2QTUBeYGJ8Sg\nC+j7V3ZsPpoH1BKRaiKSD+gKjEm2zxjgdq8X/3zgoKruDHagmSjNaxaRysAooFs2+dSY5jWrajVV\nraqqVYEfgQfCOCGAf3/bo4ELRCSPiBQEWgCrghxnZvLnmrfi7owQkTJAHWBjUKMMroC+f2W7OwVV\njROR3sAE3MiFz1R1hYj09LYPwo1EuQxYDxzDfdIIW35e8wtAKWCg98k5TsO4wqSf15yt+HPNqrpK\nRMYDS4EEYKiqpji0MRz4+Xt+FRguIstwI3KeVNWwLaktIt8B7YHSIhIFvAjkheC8f1mZC2OMMYmy\nY/ORMcaYDLKkYIwxJpElBWOMMYksKRhjjElkScEYY0wiSwo5lIjEe5U0T39V9bFv1dQqNqbznH95\n1S6XiMhMEamTgWP0PF26QUTuFJHySbYNFZH6mRznPBFp5MdrHvbmBaT3XH1FpK33fW+v8qWKSOkM\nHKuOF/tiEVklIpm6SpeIXHW6SqmIRIjIHBFZJCJtRGSciBT38dpUf28+XjNJREpk3hUYv4S6IqB9\nheYLOJKOfauSSsXGdJ7zLyDS+74HMCazjpfJP5ukcd4FTPTjNZuB0uk8TylgdpLHjb2fdbqP5b1+\nAvsx8lMAAAWMSURBVNAlyeMGAfz76YqbAxGw3xtwB/BsoK7BvlL+sjsFk8i7I5j+//bOLsSqKorj\nv3+iOVkZWqEhqaFRGVJkYQU9JEplUpQJ4kfjiz5koaVE9GUPRWQFlZhFfpaWSE4fZpaYJcqYjR8z\nmiUTKgha+WASMj5kq4e17szxdu7MHU0vk/sHl3v2Pufsvc463L323ufc/5a0LT635RwzSNKW6I02\nSBoY+eMz+e9I6tRGdRuAAXHusOhx7pRryZ8f+S9L2h31vBp5syTNkK+NMARYGnVWRS95SPRKZ2ds\nrpY05xTtrCUjNibpbUl1ct3+FyLvMeAKYL2k9ZE3QlJt+HGFpAtzyn4QWFNImNl2M9vfhj2t0RuX\nuSiUtzNsqZb0afinUdLzmevJ9Yd8DYNtMVpalylnToycXgHuy/h+f2F0I2li3LN6Se9HXqn7NlLS\nJxl7hkuqieRnwNjT8EfiVKh0VEqfynyAE7iI2A6gJvIuALrG9kCgLrb7ESMF4C1gXGx3wUXXrgU+\nBzpH/lxgYk6d39LSA58JLAe64oqPV0f+EmAa3oveQ8sfLC+J71nAjOLysmlcU/+XTP6XuADgqdg5\nDXgps69HfHeK4wZHej/Ru8d1ljYA3SL9JPBcTj2LgVE5+c1ltfOeTgKOxvVOz/isGjgUPq0CdoWf\ncv0R/jsA9C+65mpgTvF21mZgEK6zdGnRubn3Df8H8s/AZZFelvUJ0Aj0rPTv5Vz6/O9kLhJl02Rm\nxXPlnYFCT/AELsNcTC3wtKQ+uG5/o6RhuDLnD3IJjSpKr1+wVFIT3og8iuvU7LMWPabFwCO4HPJx\nYL6kVcCqci/MzA5L2ivXhWnExdE2RbntsbMLrs+f9dMYSZNxiZjewHW4pESWoZG/KerpgvutmN7A\n4XKvqy3MbKGkr4C7cM39KWpZYW+thQiipJV4kPyLfH8MBTaY2b4otzVt/2LuBFZYyEy0da6ZWYwm\nxktaCNyKB6YCv+OjsI4u4NhhSEEhkWU68Buuqnke3iifhJktk/Q9MBJYLWkK3ttbbGZPlVHHODOr\nKyQk9cg7yFzz5hZc6Gw0MBVvcMrlI2AM3guticanXXYCW4HZ+OjoAUn9gRnAzWZ2RNIifKRTjPBG\nuK2pj6YS55ckGs4bgYNmdk/xfjM7CCwAFshfDri+sKv4UErcN0mj2mPTf8BCfMRyHA8o2QVyuuJ+\nSpwl0jOFRJbuwCHzBUom4FMkJyHpKmCvmb2JK3IOBtYBoyVdHsf0kNS3zDr3AP0kDYj0BOC7mIPv\nbmar8WCVt6b0n8BFJcqtwXvLY/EAQXvtNJ+/eBYYKuka4GLgGHBUrsZ5dwlbNgO3F65JUjdJeaOu\nn4jnKuViZpPM7Ia8gBDPATrHdi98uqggqTw8rrcKuB8fOZXyx2bgjgiCJQN3Cb4BHpLUs5VzT7pv\nEcgOAs/gAaJwPQJ64aPKxFkiBYVElrnAw5Lq8SmXYznHjAF2SdqB90KXmNlu/Af9taQGYC0+NdIm\nZnYcnwtfIVe5/BuYhzcaq6K8jcDjOacvAuYVHnYWlXsEb3T7mtmWyGu3nWbWBLwGzDSzemA7PvpY\nhjesBd4F1khab2aH8Tn3D6OeWtyfxXyBq2EC/sBarorZB2iQ9F5rtuUwAr839fibSDPN7NfYtwVf\nlrQB+NjM6kr5I+yfDKyMspaXa4CZ/Qi8iAf2euD1nMMW8e/7thQ4YGZZme+b8LezOuzSmh2RpJKa\nSFQQSRuBe83sjzNYRzX+YHfqmarjdJG/HbbdzOZn8t7AX1teVznLzj3SSCGRqCxPAFdW2ohKImkr\nPg35QdGuXSkgnH3SSCGRSCQSzaSRQiKRSCSaSUEhkUgkEs2koJBIJBKJZlJQSCQSiUQzKSgkEolE\nopl/ABbVM++unSBkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1415e63c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc(v_probs, y_test, \"ROC plot of campaign data\", \n",
    "         \"False Positive Rate (1 - Specificity)\", \"True Positive Rate (Sensitivity, Recall)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.54616664306269036"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skm.roc_auc_score(y_test, lr.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>col_0</th>\n",
       "      <th>0.0</th>\n",
       "      <th>1.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>staff_pick</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>22453</td>\n",
       "      <td>4666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>3080</td>\n",
       "      <td>1107</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "col_0         0.0   1.0\n",
       "staff_pick             \n",
       "0.0         22453  4666\n",
       "1.0          3080  1107"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(y_test, lr.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[22453  4666]\n",
      " [ 3080  1107]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARkAAAD0CAYAAACvvVL/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGwRJREFUeJzt3Xu4XFWZ5/Hvj4Q7AYHQCBE6gKADPBII8iAONDY2Ny+o\nT4sgDWIzImjTKtrdoPQYGRmZdoBWUelgbPAGRJEeBqIRmWlpfLglGC6Ra7iMYAgkKMGAITnnnT/W\nKrI5nKqz65zaZ6f2+X149nOq1t6196oK9dZaa++9XkUEZmZV2aDuCphZsznImFmlHGTMrFIOMmZW\nKQcZM6uUg4yZVcpBxswq5SBTM0mbSvrfkp6T9MMx7OcEST/rZd3qIulgSQ/UXQ/rDflivHIkfRA4\nE3gj8DywCDgvIm4e435PBM4ADoqItWOu6HpOUgC7R8TDddfFxsfkuivQDySdCZwFnAbMB14CjgDe\nDYwpyAB/Cjw4EQJMGZImT6TP4oi3bR4rnh0ote3Cu1fPj4gjK65S70WElw4LsBXwB+D9HbbZGPhn\n4Ld5+Wdg47zuUOAJ4NPA08BS4MN53RdIAWtNPsYpwCzge4V9TwcCmJyfnww8QmpNPQqcUCi/ufC6\ng4A7gOfy34MK6/4d+G/AL/N+fgZMbfPeWvX/+0L93wMcDTwIPAt8trD9AcAtwO/zthcDG+V1N+X3\nsiq/3w8U9v8PwFPAd1tl+TW75WPsl5/vCDwDHFr3/xu9WPZ708axZulupRZgQd31Hc3iMZmRvQXY\nBLimwzafAw4EZgD7kL5o5xTWv5YUrKaRAsnXJW0dEZ8H/jtwVURsERFzOlVE0ubAV4GjImIKKZAs\nGma7bYDr87bbAhcC10vatrDZB4EPA38CbAR8psOhX0v6DKYB/xW4FPgrYCZwMPCPknbJ2w4AnwKm\nkj67w4CPAUTEIXmbffL7vaqw/21IrbpTiweOiCWkAPQ9SZsB/wpcHhH/3qG+fSQYiMFSS79ykBnZ\ntsDy6NyEPwE4NyKejohnSC2UEwvr1+T1ayJiHulX/A2jrM8gsLekTSNiaUQsHmabdwAPRcR3I2Jt\nRFwB3A+8q7DNv0bEgxHxIjCXFCDbWUMaf1oDXEkKIF+JiOfz8X9NCq5ExMKIuDUf9zHgX4A/K/Ge\nPh8Rq3N9XiEiLgUeBm4DdiAF9UYIYJAotfQrB5mRrQCmSuo0frUj8Hjh+eO57OV9DAlSLwBbdFuR\niFhF6mKcBiyVdL2kN5aoT6tO0wrPn+qiPisiojVw0AoCywrrX2y9XtIekq6T9JSklaSW2tQO+wZ4\nJiL+OMI2lwJ7A1+LiNUjbNtXBkv+168cZEZ2C7CaNA7Rzm9JTf2WnXPZaKwCNis8f21xZUTMj4i/\nIP2i30/68o1Un1adnhxlnbrxTVK9do+ILYHPAhrhNR1/piVtQRrnmgPMyt3BRgiCgSi39CsHmRFE\nxHOkcYivS3qPpM0kbSjpHEnPSnqY1Eo4R9J2kqbm7b83ykMuAg6RtLOkrYCzWyskbS/pmDw2s5rU\n7RruJ24esIekD0qaLOkDwJ7AdaOsUzemACuBP+RW1ulD1i8Ddi2xn60lPS3pXuArpEHP/0Iaa7qk\nlxWum7tLRkRcQLpG5hzSmY3fkL78p5K+vFNJZ3zuBu4B7gS+OMpj3QBclfe1kFcGhg1yPX5LOuPy\nZ7z6S0xErADeSTqjtYJ0ZuidEbF8NHXq0mdIg8rPk1pZVw1ZPwu4XNLvJR3bYT+rgCNJQetI1r3P\nM4H9JJ3Qy0rXJYABotTSr3wx3ihIegswKyKOyM/PBoiIL9VasYaRNB24LiL2rrkqldlnn41i/ryR\nhqySHV63dGFE7F9xlXrOLZnRmUZqzbQ8wSsHVc1KGyy59Ctf8WtWo+jzrlAZDjKj8ySwU+H56xif\nMzfWNAEDzY4x7i6N0h3A7pJ2kbQRcBxwbc11sj6ULsZrdnfJQWYU8oV1f0O6WfI+YG6bK29tlCRd\nQbpG6Q2SnpB0St11qoYYKLn0K3eXRinfHjCv7no0VUQcX3cdxkMAgw3vLjnImNUogJca3qFwkDGr\n2WD0b1eoDAcZsxqlK34dZMysIoEYaHh3qdnvrmKSTh15KxuLifAZD4ZKLf3KQWZsGv8FWA80+jNu\ndZd8CtvMKiIGotm/9etVkJm6zaSYvtOGdVejtJ2nTWb/fTbpq6scHlyy7cgbrUc22WgrttxiWl99\nxn9c/XteWrOqVNMjgDVM6slxJe0EfAfYPu96dkR8JU/ydRVpUvrHgGMj4nf5NWeT5p0eAP42Iubn\n8pnAZcCmpOvBPhERIWnjfIyZpGlEPpCnWW1rvQoy03fakNvn7zTyhjZqR7zvpLqr0Hi33V1+Tq2I\nnrZk1gKfjog7JU0BFkq6gZTJ4saIOF/SWaT0Pv8gaU/SLTF7kaZs/bmkPfJUq98EPkKaV3keaU6f\nn5AC0u8i4vWSjgP+B2lK2Laa3U4z6wODqNQykjyx/J358fOkW16mAccAl+fNLmfdVLLHAFfmCdwf\nJU3WfoCkHYAt84TwQWq5FF/T2tePgMMkdazcetWSMZto0sBv73/r84Rf+5JaIttHxNK86ilSdwpS\nALq18LLWvEhr8uOh5a3X/AbSPXySniNn9GhXFwcZs1p11V2aKmlB4fnsiJj9qj2midevBj4ZESuL\nDY08rjKuY1wOMmY1SlM9lA4yy0eaflPShqQA8/2I+HEuXiZph4hYmrtCT+fydvMiPZkfDy0vvuaJ\nnCZoK9IAcFsekzGrUSBeikmllpHksZE5wH0RcWFh1bXAh/LjDwH/q1B+nKSNcwbQ3YHbc9dqpaQD\n8z5PGvKa1r7+Evg/McJE4W7JmNVssHdnl95Kylx6j6RW+uLPAucDc/OcPI8DxwJExGJJc0kZQNcC\nHy8k8fsY605h/yQvkILYd3MqoGdJZ6c6cpAxq1EvB34j4mbaJ9I7rM1rzgPOG6Z8ASlj59DyPwLv\n76ZeDjJmNQrEQB/fl1SGg4xZzboY+O1LDjJmNYrA9y6ZWZXKXc3bzxxkzGoUwEvR7K9hs9+d2Xou\n6O8JqcpwkDGrWdOn33SQMatRyrvkIGNmlenvqTXLcJAxq5FbMmZWObdkzKwyEWLNYLO/hs1+d2br\nuTSfjFsyZlYZp0QxswqlgV+3ZMysQr4Yz8wqMxFuK2h2CDXrA4NsUGopQ9K3JT0t6d5C2VWSFuXl\nsdbUnJKmS3qxsO6SwmtmSrpH0sOSvtrKrZTnA74ql9+WU6905JaMWY0iYM1gT3/rLwMuJiVky8eI\nlzM8SroAeK6w/ZKImDHMfpxB0qwJUndpg1JLqf1F3ESa4PtVcmvkWOCKTvvodQZJBxmzmg3k+5dG\nWnrgYGBZRDxUKNsld5V+IengXDaNkhkkSa2ibTsd1N0lsxp1eQq7VAbJDo7nla2YpcDOEbFC0kzg\n3yTt1cX+SnGQMauVurlBcsQMkm2PkrI9vg+Y2SqLiNXA6vx4oaQlwB44g6RZswzmeX5HWsbo7cD9\nEfFyN0jSdpIm5ce7kjJIPuIMkmYNkrIV9O46GUlXAIeSulZPAJ+PiDmkTI9DB3wPAc6VtAYYBE6L\niNagsTNImjVBINYOjpznuvT+Io5vU37yMGVXA1e32d4ZJM2awndhm1llfIOkmVXO02+aWXWi+TdI\nOsiY1cgz45lZ5dySMbPKBLC2t3dhr3cqfXeSjpT0QJ574qwqj2XWj1qTVpVZ+lVlQSZfrvx14Chg\nT+B4SXtWdTyzfjVOtxXUpsqWzAHAwxHxSES8BFxJmovCzFoCt2TG4OV5J7LinBQvk3SqpAWSFjyz\nYqDC6pitf1oX4znIVCgiZkfE/hGx/3bb9u4eDrN+0fQgU+XZpda8Ey3FOSnMjDTwO+CzS6N2B7C7\npF0kbUS6JfzaCo9n1peaPvBbWUsmItZK+htgPjAJ+HZELK7qeGb9KMIX441JRMwjpVMwszbCQcbM\nqtPfg7plOMiY1azpLZlmD2ubred6fZ1MmzS1syQ9WUhHe3Rh3dn5tp8HJB1RKO9ZmloHGbM65YnE\nyywlXUZKKTvURRExIy/zAPJtPscBe+XXfKOVvYB1aWp3z0trny+nqQUuIqWp7chBxqxGQeoulVlK\n7a9DmtphHANcGRGrI+JR4GHgAKepNWuUru7Cntq6BScvp3ZxoDMk3Z27U1vnsna3/vQ0Ta2DjFnN\nIsot5AyShaVsitpvArsCM0ipaS+o5p0Mz0HGrGa97C4Nv/9YFhEDETEIXEqaIQHa3/pTJk1tK/Wt\n09Sarc9SK6XaIJPHWFreC7TOPF0LHJfPGO1CGuC93WlqzRqmlxfjDZemFjhU0gzSOPNjwEcBImKx\npLnAr4G1wMcjojXfitPUmjXF4GDvgkybNLVzOmx/HnDeMOVOU2vWBMHYukL9wEHGrGYdBzQawEHG\nrE7R/HuXHGTM6tbwpkzbICNpy04vjIiVva+O2cQzkVsyi0kxtvgJtJ4HsHOF9TKbMDpfZdL/2gaZ\niNip3Toz640ICE8kDpKOk/TZ/Ph1kmZWWy2ziaOLe5f60ohBRtLFwNuAE3PRC8AlVVbKbEKJkkuf\nKnN26aCI2E/SrwAi4tmc4sTMxswX4wGskbQBOZZK2hYYrLRWZhNJH7dSyigzJvN14GpgO0lfAG6m\nxJR7ZlbCONyFXbcRWzIR8R1JC4G356L3R8S9nV5jZl1oeEum7BW/k4A1pI+j2efbzMZbH7dSyihz\ndulzwBXAjqQZsn4g6eyqK2Y2YfjsEicB+0bECwCSzgN+BXypyoqZTQhB41syZYLM0iHbTc5lZtYD\n/XyhXRltu0uSLpJ0IWmKvcWSviXpUuAeYPl4VdCs8XrYXWqTQfLLku7PKVGukfSaXD5d0ouFzJKX\nFF4zLhkk7yXdJHk9MAu4BbgVOJd1832a2ViFyi3lXMarM0jeAOwdEW8CHgSKY6pLCpklTyuU9yyD\nZKcbJNvOC2pmPRKgHl7aGhE3DW1dRMTPCk9vJWUZaKuYQTI/b2WQ/Akpg+SsvOmPgIslqVPGgjJn\nl3aTdGVuaj3YWkZ6nZmVUbIVM/YMki1/zSt7IrvkrtIvJB2cy3qaQbLMwO9lwBeB/wkcBXyYvj6h\nZraeKf9tWh4R+4/2MPlylLXA93PRUmDniFiRZ1b4N0l7jXb/7ZS5sG6ziJgPEBFLIuIcUrAxs14Y\nh+tkJJ0MvBM4odW1iYjVEbEiP14ILAH2oIYMkqvzDZJLJJ0m6V3AlHJvzcxGVHGQkXQk8PfAu1vX\nu+Xy7SRNyo93JQ3wPlJHBslPAZsDf0tKArUVqV9nZmPV44vx2mSQPBvYGLghn4m+NZ9JOgQ4V9Ia\n0swKp0XEs3lX45dBMiJuyw+fZ93EVWbWI+rhCGc3GSQj4mrSDAvDras+g6Ska+jQSIuI93VzIDNr\no+GnUTq1ZC4et1pkD93/Gt5x0LvH+7ATy2N3112D5osXu9q8ly2Z9VGni/FuHM+KmE1YvkHSzCrT\n59M4lOEgY1Y3B5lE0sYRsbrKyphNRE0fkylz79IBku4BHsrP95H0tcprZjZRNHxmvDJX/H6VdDly\n6/Lju0jJ3sxsjJTvwi6z9Ksy3aUNIuLxfKVgy0BF9TGbeHx2id9IOgCIfJ/DGaSJb8ysF/q4K1RG\nmSBzOqnLtDOwDPh5LjOzHmj6wG+Ze5eepsRNUGY2ShM9yOTJw1/1MUTEaGblMrOicEsGUveoZRPg\nveTp98ysByZ6kImIq4rPJX0XuLmyGplNMP18erqM0eS13gXYvtcVMbNmKjMm8zvWNeg2IM2GdVaV\nlTKbUBreXerYksnze+4DbJeXrSNi14iYOx6VM2u8PPBbZimjTQbJbSTdIOmh/HfrwrqzczbIByQd\nUSgflwyS5AmC50XEQF4aHnPNatDbe5cu49UZJM8CboyI3YEb83Mk7Um6PGWv/JpvtCYWp4cZJMuM\nySyStG+J7cxsNHoYZCLiJtKQRtExwOX58eWkbJCt8itzapRHgYeBA4oZJHPD4jtDXtPa14+Aw1qt\nnHY6zfE7OWeI2xe4Q9ISYBWg9F5ivxHfsZl1JLq6TmaqpAWF57MjYnaJ122f05wAPMW6EzfTSGlr\nW1qZItdQMoOkpFYGyeXtDt5p4Pd2YD/Ak+6aVaW7XNhjyiAJqXUgje/lf52CjCBljRynuphNTNV/\n5ZdJ2iEiluau0NO5/OVskFkrU2SZDJJPlM0g2SnIbCfpzHYrI+LCTjs2s5KqDzKtrI/n57/FbJA/\nkHQhsCNpgPf2iBiQtFLSgcBtpAySXxuyr1voQQbJScAW5BaNmVWjl52XNhkkzwfmSjoFeBw4FiAi\nFkuaC/waWAt8PCJac0WNSwbJpRFxbjdv0MxGofoMkgCHtdn+PFL66aHl1WeQxC0Ys+r1+fy9ZXQK\nMsNGPjPrrabfINkpg+TQC3rMrAKeT8bMquUgY2aVmeBjMmZWMdH8MywOMmZ1c0vGzKrkgV8zq9ZE\nPYVtZuPAKVHMrHIOMmZWJbdkzKxaDjJmViW3ZMysOr7i18yqJCbwXdhmNk4a3pIZTS7sUobLZGdm\nr6aIUsuI+5HeIGlRYVkp6ZOSZkl6slB+dOE1XWWQHI3KggzDZ7Izs6Kyid1KtHYi4oGImBERM4CZ\nwAvANXn1Ra11ETEPRp1BsmuVBZk2mezMbIhe5sIuOAxYEhGPd9hmNBkku1ZlS6YUSadKWiBpwUsD\nL9RdHbPxV74lM7X1XcnLqR32ehxwReH5GZLuzsMYW+eyl7NBZq1MkdNon0Gya7UHmYiYHRH7R8T+\nG03arO7qmI27Lloyy1vflbwMm6JW0kakzK8/zEXfBHYFZgBLgQuqf1fr+OySWZ26S1Nb1lHAnRGx\nDKD1F0DSpcB1+eloMkh2rfaWjNmE16OB34LjKXSV8hhLy3uB1hnfa4HjJG0saRfWZZBcCqyUdGA+\nq3QS67JOdq2ylsxwmewiYk5VxzPrR6LnGSQ3B/4C+Gih+J8kzSCFqsda60aZQbJrlQWZDpnszKyo\nxDUw5XcVq4Bth5Sd2GH7rjJIjobHZMxq5hskzaw6vkHSzKrmGyTNrFIOMmZWnaCnA7/rIwcZs5p5\n4NfMquUgY2ZV6fXFeOsjBxmzOkV4TMbMquWzS2ZWKXeXzKw6AQw2O8o4yJjVrdkxxkHGrG7uLplZ\ntXx2ycyq5JaMmVVGAWr4wK/n+DWr22DJpQRJj+XMj4skLchl20i6QdJD+e/Whe37OoOkmZXQqzS1\nBW/LmSL3z8/PAm6MiN2BG/Pz/s8gaWYl9DBNbQfHAJfnx5ezLhvkxMggaTaxxbr7l0ZaSu+Qn0ta\nWMgwuX1OcwLwFLB9fjwuGSQ98GtWsy7OLk1tjbNks4fJIvmfI+JJSX8C3CDp/uLKiAhpfM9nOciY\n1a18K2V5YZylza7iyfz3aUnXAAcAyyTtEBFLc1fo6by5M0iaNV6ABqLUMhJJm0ua0noMHE7KFnkt\n8KG82YdYlw2yvzNImllJveu8bA9ck882TwZ+EBE/lXQHMFfSKcDjwLHQgAySZlZOl6en24qIR4B9\nhilfARzW5jXOIGnWeL53ycwqE5S+mrdfOciY1Uh0fTVv33GQMaubg4yZVSaAEqen+5mDjFnN3F0y\ns2o5yJhZdZzczcyqFDjImFnFfJ2MmVXJA79mVp0ABprdlHGQMauVB37H1cqXli3/6aMXPl53Pbow\nFVhedyUarh8/4z/tamsHmfETEdvVXYduSFow0kxlNjYT4jN2kDGzygTQ8ORuDjJmtQqIZg/8eo7f\nsRk6U7z1XrM/49bZpTLLCCTtJOn/Svq1pMWSPpHLZ0l6MmeVXCTp6MJrnEFyfTZMOopXkDSQ/1Hv\nlfRDSZuN9liSDpV0XX78bklnddj2NZI+NopjzJL0mbLlQ7a5TNJfdnGs6ZLuHWm7kT7jRuhd3qW1\nwKcjYk/gQODjOUskwEU5q+SMiJgHziDZFC/mf9S9gZeA04orlXT9bxAR10bE+R02eQ1pImjrBz0K\nMhGxNCLuzI+fB+6jc1I2Z5BsmP8AXp9/wR+Q9B1SuoqdJB0u6RZJd+YWzxYAko6UdL+kO4H3tXYk\n6WRJF+fH20u6RtJdeTkIOB/YLbeivpy3+ztJd0i6W9IXCvv6nKQHJd0MvGGkNyHpI3k/d0m6ekjr\n7O2SFuT9vTNvP0nSlwvH/uhYP8hm6XkGSSC1FIF9gdty0Rn58/+2pK1z2bhkkHSQGQeSJgNHAffk\not2Bb0TEXsAq4Bzg7RGxH7AAOFPSJsClwLuAmcBr2+z+q8AvImIfYD9gMSmh+pLcivo7SYfnYx4A\nzABmSjpE0kxSc3kGcDTw5hJv58cR8eZ8vPuAUwrrpudjvAO4JL+HU4DnIuLNef8fyTl+DPLZpcFy\nS84gWVhOHW6X+UfqauCTEbGS1PXZlfTvvBS4YHzeXOKzS9XaVNKi/Pg/gDnAjsDjEXFrLj8Q2BP4\nZR5b2wi4BXgj8GhEPAQg6XvAcP9T/Tkp+RY5Z85zhV+qlsPz8qv8fAtS0JkCXBMRL+RjXFviPe0t\n6YukLtkWwPzCurkRMQg8JOmR/B4OB95UGK/ZKh/7wRLHmhh6mEFS0oakAPP9iPhx2n0sK6y/FLgu\nPx2XDJIOMtV6MSJmFAtyIFlVLAJuiIjjh2z3iteNkYAvRcS/DDnGJ0exr8uA90TEXZJOBg4trBv6\nbYl87DMiohiMWs15g55djJfPAM0B7ouICwvlO+SskADvJXXTIWWQ/IGkC0k/fq0MkgOSVko6kNTd\nOgn42mjr5e5S/W4F3irp9fByqtE9gPuB6ZJ2y9sd3+b1NwKn59dOkrQV8DypldIyH/jrwljPNKWE\n7DcB75G0qVJ603eVqO8UYGn+xTxhyLr3S9og13lX4IF87NPz9kjaQymFqgFEEAMDpZYS3gqcCPy5\nXnm6+p+UTkffDbwN+FQ6dCwGWhkkf8qrM0h+izQYvARnkOxfEfFMbhFcIWnjXHxORDyY+9zXS3qB\n1N2aMswuPgHMVkpBOgCcHhG3SPplPkX8kzwu85+AW3JL6g/AX0XEnZKuAu4iJWG/o0SV/5H06/ZM\n/lus0/8Dbge2BE6LiD9K+hZprObO/Ev7DGM4U9FIPbriNyJuJrUch5rX4TWVZ5BUNPy+CbP12VaT\nt4u3TDmm1Lbzfz9nYT/ex+WWjFmdIlpnjhrLQcasbg3vTTjImNUs3JIxs+p4Zjwzq1IA5U5P9y0H\nGbMaBRCetMrMKhPNn7TKQcasZk1vyfhiPLMaSfopKSNDGcsjYtSTR9XFQcbMKuUbJM2sUg4yZlYp\nBxkzq5SDjJlVykHGzCrlIGNmlXKQMbNKOciYWaUcZMysUv8fe2lm1W7IHUcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x117a50908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(lr, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model looks a little better based on the accuracy score of .75 but the area under the curve is pretty terrible. Area under the curve looks at all possible threshold values. Accuracy is just based on a single threshold value of .5. It is a pretty poor metric of classifications model performance. and one of the least important to determining positive predicitive power."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Under the time constraints (~10h) I can't progress further. The above just described the beginning of modeling the relationship between probability of a campaign being a staff pick and the campaign attributes determined at launch. I just took a peak and I hope you can see where we can go from here. I plan to improve the model by using cross validation or grid searching to select hyperparameters for each model. If we wanted to we could also take a look at stacking models of the textual information, profile information of the campaign creator and possibly web scraped data about the domains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}