basic_resources/intro_sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit-Learn\n",
    "\n",
    "### Quick Note\n",
    "\n",
    "This tutorial is part of the [Python Basic Resources](https://github.com/GalvanizeOpenSource/python-resources/tree/master/basic_resources) 4-part series produced by [Galvanize](http://www.galvanize.com/).\n",
    "\n",
    "This tutorial moves very fast. If you would like a slower, more in-depth intro to Python, we suggest you take our [Intro to Python Evening Course](http://www.galvanize.com/courses/intro-to-python/). It's the perfect way to become familiar with Python + gain experience using Python to solve challenging problems.\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "This document is a *Jupyter notebook*. If you are viewing it using GitHub, then you cannot execute the cells that contain Python code. To view **and run** this notebook you'll need to install Jupyter on your computer before you continue. See [these installation instructions](https://github.com/GalvanizeOpenSource/python-resources/tree/master/installation) for help!\n",
    "\n",
    "### Overview\n",
    "\n",
    "Here we're going to walk through running a model, and looking at the results. Before we get started let's go over some terminology...\n",
    "\n",
    "### Terminology\n",
    "\n",
    "1.) `features` - Another word for the X variables, independent variables, predictors, regressors...   \n",
    "2.) `target` - Another word for the Y variable, dependent variable, outcome variable, response...  \n",
    "3.) `model` - What we use to relate one set of variables to another.      \n",
    "4.) `feature engineering` - Refers to the process of manipulating your features, creating new ones, etc. before feeding the data into your model.   \n",
    "5.) `training data set` - Refers to the observations from your data set that are used to train/learn the statistical model.   \n",
    "6.) `testing data set` - Refers to the observations from your data that are **not** used to train/learn the statistical model. They are held out, and not seen by the model during training.  \n",
    "7.) `hyperparameters` - Stay tuned... it's a little hard to put into words, but remind me if I don't discuss it later. \n",
    "\n",
    "### Scikit-learn import\n",
    "\n",
    "```python\n",
    "import sklearn \n",
    "```\n",
    "\n",
    "Typically we're actually going to be importing something from one of the modules/libraries in `sklearn`. The [sklearn main page](http://scikit-learn.org/stable/) can help you determine where you might find something that you are looking for, and the [API reference](http://scikit-learn.org/stable/modules/classes) is also pretty helpful. A large majority of all of the machine learning algorithms you might run can be found somewhere within `sklearn`. Today we're going to talk through using a `Random Forest Regressor`.\n",
    "\n",
    "### General workflow\n",
    "\n",
    "Here are the steps by which we train a model... \n",
    "\n",
    "1.) Import whatever model you'll be fitting.  \n",
    "2.) Instantiate the model (i.e. create a variable that holds your model object). Set any hyperparameters as you see fit (we'll discuss what these are shortly).   \n",
    "3.) Feed in the X and Y variables (features and target) to the `.fit()` method.   \n",
    "4.) Call the `.score()` or `.predict()` method to see how well the model does on the training data (or new data). \n",
    "\n",
    "##### What would be another word/term we might use to describe the `new data` from step (4) above?\n",
    "\n",
    "We'll be working with a `RandomForestRegressor` tonight, which you can see the documentation for [here](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>FFMC</th>\n",
       "      <th>DMC</th>\n",
       "      <th>DC</th>\n",
       "      <th>ISI</th>\n",
       "      <th>temp</th>\n",
       "      <th>RH</th>\n",
       "      <th>wind</th>\n",
       "      <th>rain</th>\n",
       "      <th>area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>mar</td>\n",
       "      <td>fri</td>\n",
       "      <td>86.2</td>\n",
       "      <td>26.2</td>\n",
       "      <td>94.3</td>\n",
       "      <td>5.1</td>\n",
       "      <td>8.2</td>\n",
       "      <td>51</td>\n",
       "      <td>6.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>oct</td>\n",
       "      <td>tue</td>\n",
       "      <td>90.6</td>\n",
       "      <td>35.4</td>\n",
       "      <td>669.1</td>\n",
       "      <td>6.7</td>\n",
       "      <td>18.0</td>\n",
       "      <td>33</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>oct</td>\n",
       "      <td>sat</td>\n",
       "      <td>90.6</td>\n",
       "      <td>43.7</td>\n",
       "      <td>686.9</td>\n",
       "      <td>6.7</td>\n",
       "      <td>14.6</td>\n",
       "      <td>33</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>mar</td>\n",
       "      <td>fri</td>\n",
       "      <td>91.7</td>\n",
       "      <td>33.3</td>\n",
       "      <td>77.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.3</td>\n",
       "      <td>97</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>mar</td>\n",
       "      <td>sun</td>\n",
       "      <td>89.3</td>\n",
       "      <td>51.3</td>\n",
       "      <td>102.2</td>\n",
       "      <td>9.6</td>\n",
       "      <td>11.4</td>\n",
       "      <td>99</td>\n",
       "      <td>1.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   X  Y month  day  FFMC   DMC     DC  ISI  temp  RH  wind  rain  area\n",
       "0  7  5   mar  fri  86.2  26.2   94.3  5.1   8.2  51   6.7   0.0   0.0\n",
       "1  7  4   oct  tue  90.6  35.4  669.1  6.7  18.0  33   0.9   0.0   0.0\n",
       "2  7  4   oct  sat  90.6  43.7  686.9  6.7  14.6  33   1.3   0.0   0.0\n",
       "3  8  6   mar  fri  91.7  33.3   77.5  9.0   8.3  97   4.0   0.2   0.0\n",
       "4  8  6   mar  sun  89.3  51.3  102.2  9.6  11.4  99   1.8   0.0   0.0"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('data/forestfires.csv') # Get the data. \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor # Import our model. \n",
    "random_forest = RandomForestRegressor(n_estimators=100) # Instantiate it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create our features (X variables) and target (Y variable). I'm using the forest-fire \n",
    "data, and for now am only going to use the `X` and `Y` columns (which are the spatial coordinates of the fires) for the features, and the `area` column for the target (this is defined as the dependent variable on the UCI website where I got this data). A link to the data and it's description can be found [here](https://archive.ics.uci.edu/ml/datasets/Forest+Fires). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### How do I pull the X and Y columns from our df to use as the features? How about the area?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features = df[['X','Y']]\n",
    "target = df['area']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='auto', max_leaf_nodes=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=100, n_jobs=1, oob_score=False, random_state=None,\n",
       "           verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit/train the model (i.e. build the model based off the training data)\n",
    "random_forest.fit(features, target) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.032062770222170189"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_forest.score(features, target) # This .score() gives you the R^2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions = random_forest.predict(features) # This gives us back a vector of predictions\n",
    "                                              # (one for each observation). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In terms of metrics, the [sklearn.metrics](http://scikit-learn.org/stable/modules/classes#sklearn-metrics-metrics) documentation will give you an idea of any of the metrics you can use to judge a model. The majority of these take the format of a fuction call where you input `(y_predictions, y_observations)`, and they output the calculated metric. We'll look at mean squared error below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3914.55650291\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "mse = mean_squared_error(predictions, target)\n",
    "print mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks terrible! We're doing awful. Let's see if we can add in something else and get better. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.203887096898\n"
     ]
    }
   ],
   "source": [
    "features = df[['X','Y', 'wind']]\n",
    "target = df['area']\n",
    "random_forest.fit(features, target)\n",
    "print random_forest.score(features, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3219.66016598\n"
     ]
    }
   ],
   "source": [
    "predictions = random_forest.predict(features)\n",
    "print mean_squared_error(predictions, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better! But let's try one more variable. I imagine the `month` could be pretty important. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "could not convert string to float: nov",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-5b5ffb984971>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mfeatures\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'X'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'Y'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'wind'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'month'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mtarget\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'area'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mrandom_forest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/sallamander/anaconda/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m    193\u001b[0m         \"\"\"\n\u001b[1;32m    194\u001b[0m         \u001b[0;31m# Validate or convert input data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 195\u001b[0;31m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mDTYPE\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"csc\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    196\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0missparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    197\u001b[0m             \u001b[0;31m# Pre-sort indices to avoid that each individual tree of the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/sallamander/anaconda/lib/python2.7/site-packages/sklearn/utils/validation.pyc\u001b[0m in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features)\u001b[0m\n\u001b[1;32m    342\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    343\u001b[0m                 \u001b[0mdtype\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 344\u001b[0;31m         \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    345\u001b[0m         \u001b[0;31m# make sure we actually converted to numeric:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    346\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mdtype_numeric\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0marray\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkind\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"O\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: could not convert string to float: nov"
     ]
    }
   ],
   "source": [
    "features = df[['X','Y', 'wind', 'month']]\n",
    "target = df['area']\n",
    "random_forest.fit(features, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### What do you think went wrong here? Don't look ahead!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What went wrong here? It turns out that most of the algorithms we use don't accept strings as inputs, but rather expect numeric values. The way to fix this would be to create dummy variables for the months (I'll leave that as an exercise for you). \n",
    "\n",
    "So, we have run two models now, and saw that the second one performed better. We could keep adding variables into our model, checking the R^2 and our MSE after adding in any variable. But we're not actually running our model on any data that we aren't training it on, so how do we know that what we are putting into our model would actually help on data that we've never seen. In other words, how do we tell if our model will generalize well? The answer is **cross validation**.\n",
    "\n",
    "The way that **cross validation** works is that we break our data into `k` number of folds (typically 5 or 10). We train our model on `k-1` of those folds, and then predict on the `kth` fold. We take those predictions, and then get our scoring metric (`mean squared error`, in our case) using those predictions. We then do this again, and again, and again, until each of the `k` folds has been used for predictions (so with 5 folds, we do this 5 times, and with 10 folds, 10 times, etc.)\n",
    "\n",
    "![cross-val-image](http://i.stack.imgur.com/1fXzJ.png)\n",
    "\n",
    "Using **cross validation**, we can get an idea of how our model would perform on data it hasn't seen before, and then when we add in variables into our model (or change model hyperparameters), we can be more sure that they were actually worth putting into our model.\n",
    "\n",
    "Best of all, it turns out that sklearn has a library we can use for this! Check out the [cross validation library](http://scikit-learn.org/stable/modules/classes#module-sklearn.cross_validation) for all the details. Today we'll be looking at the `cross_val_score` function, which allows you to pass in a model, a target (Y), a feature set (X), a number of folds (5 or 10, for example), and a scoring function (we'll use our mean_squared_error). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  -375.47767068,   -162.20055005,   -603.25384842,   -536.51811759,\n",
       "          -62.73033068,   -280.80067275,   -341.6422815 ,   -173.85800086,\n",
       "        -1912.14482145, -50478.40726929,   -118.81598123,   -528.28201791,\n",
       "         -419.59203963,   -139.91659542,  -1225.51294279, -21430.62382299,\n",
       "        -1533.13269162,   -509.65316076,  -3340.51172585,   -345.3483658 ])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "features = df[['X','Y']]\n",
    "target = df['area']\n",
    "results = cross_val_score(random_forest, features, target, cv=20, scoring='mean_squared_error')\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Anybody want to take a guess at why we're getting negative mean squared error values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4225.9211453648695"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-results.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5270.6323296025976"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = df[['X', 'Y', 'wind']]\n",
    "target = df['area']\n",
    "results = cross_val_score(random_forest, features, target, cv=20, scoring='mean_squared_error')\n",
    "-results.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it looks like `wind` might not have been as helpful as we thought. Good think we used cross-validation!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cross-validation is a crucial part of a data-scientists workflow. We have to make sure that our model will generalize well, and cross-validation is a way to make sure that we are putting the right variables into our model. It can also be used to check our model hyperparameters (for a random forest, this might be the number of trees, the depth of each tree, etc.). `Sklearn` also has a built in to perform cross-validation over hyperparameters. It is located in the `sklearn.grid_search` module, and it is called `GridSearchCV`. As arguments, it takes an estimator/model (such as our Random Forest) and a parameter grid (dictionary). We instantiate it with these, and then we call the `.fit()` method on it, passing it our features and target. It returns back to us the best parameters to use for our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "random_forest = RandomForestRegressor()\n",
    "param_grid = {'n_estimators': [10, 100, 500], 'max_depth': [1, 3, 5]}\n",
    "grid_search_cv = GridSearchCV(random_forest, param_grid, scoring='mean_squared_error')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='auto', max_leaf_nodes=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "           verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, loss_func=None, n_jobs=1,\n",
       "       param_grid={'n_estimators': [10, 100, 500], 'max_depth': [1, 3, 5]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, score_func=None,\n",
       "       scoring='mean_squared_error', verbose=0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = df[['X', 'Y', 'wind']]\n",
    "target = df['area']\n",
    "grid_search_cv.fit(features, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_model = grid_search_cv.best_estimator_ # Get a copy of the best model. \n",
    "best_params = grid_search_cv.best_params_ # Get a dictionary of the best parameters. \n",
    "best_score = grid_search_cv.best_score_ # Get the best score of scoring function we passed in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 1, 'n_estimators': 100}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-4173.1288852755588"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='auto', max_leaf_nodes=None, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "           verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, loss_func=None, n_jobs=1,\n",
       "       param_grid={'n_estimators': [10, 100, 500], 'max_depth': [1, 3, 5]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, score_func=None,\n",
       "       scoring='mean_squared_error', verbose=0)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = df[['X', 'Y']]\n",
    "target = df['area']\n",
    "grid_search_cv.fit(features, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_model = grid_search_cv.best_estimator_ # Get a copy of the best model. \n",
    "best_params = grid_search_cv.best_params_ # Get a dictionary of the best parameters. \n",
    "best_score = grid_search_cv.best_score_ # Get the best score of scoring function we passed in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 1, 'n_estimators': 500}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-4171.6458630324541"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With all this being said, we can kind of re-define/re-work the steps in our general workflow... \n",
    "\n",
    "1.) Import whatever model you'll be fitting.  \n",
    "2.) Instantiate the model (i.e. create a variable that holds your model object). Set any hyperparameters as you see fit.   \n",
    "3.) Feed in the X and Y variables (features and target) to the `.fit()` method.   \n",
    "4.) Call the `.score()` or `.predict()` method to see how well the model does on the training data (or new data).   \n",
    "5.) Repeat steps (2) - (4) to find the best model given your chosen scoring metric.\n",
    "\n",
    "**Note**: This assumes that all of your feature engineering/variable manipulation is done. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Want some practice?\n",
    "\n",
    "Have a look at the `intro_sklearn_practice.ipynb` notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Next Steps\n",
    "\n",
    "Congrats on finishing this [Python Basic Resources](https://github.com/GalvanizeOpenSource/python-resources/tree/master/basic_resources) series!\n",
    "\n",
    "Next, you can see Python in-action exploring a real dataset, have a look at [Exploring Data with Python using Jupyter Notebooks](https://github.com/GalvanizeOpenSource/python-resources/tree/master/exploring_data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}