08_grid_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Efficiently searching for optimal tuning parameters ([video #8](https://www.youtube.com/watch?v=Gol_qOgRqfA&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=8))\n",
    "\n",
    "Created by [Data School](http://www.dataschool.io/). Watch all 9 videos on [YouTube](https://www.youtube.com/playlist?list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A). Download the notebooks from [GitHub](https://github.com/justmarkham/scikit-learn-videos).\n",
    "\n",
    "**Note:** This notebook uses Python 3.6 and scikit-learn 0.19.1. The original notebook (shown in the video) used Python 2.7 and scikit-learn 0.16, and can be downloaded from the [archive branch](https://github.com/justmarkham/scikit-learn-videos/tree/archive)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "- How can K-fold cross-validation be used to search for an **optimal tuning parameter**?\n",
    "- How can this process be made **more efficient**?\n",
    "- How do you search for **multiple tuning parameters** at once?\n",
    "- What do you do with those tuning parameters before making **real predictions**?\n",
    "- How can the **computational expense** of this process be reduced?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review of K-fold cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Steps for cross-validation:\n",
    "\n",
    "- Dataset is split into K \"folds\" of **equal size**\n",
    "- Each fold acts as the **testing set** 1 time, and acts as the **training set** K-1 times\n",
    "- **Average testing performance** is used as the estimate of out-of-sample performance\n",
    "\n",
    "Benefits of cross-validation:\n",
    "\n",
    "- More **reliable** estimate of out-of-sample performance than train/test split\n",
    "- Can be used for selecting **tuning parameters**, choosing between **models**, and selecting **features**\n",
    "\n",
    "Drawbacks of cross-validation:\n",
    "\n",
    "- Can be computationally **expensive**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review of parameter tuning using `cross_val_score`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goal:** Select the best tuning parameters (aka \"hyperparameters\") for KNN on the iris dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in the iris data\n",
    "iris = load_iris()\n",
    "\n",
    "# create X (features) and y (response)\n",
    "X = iris.data\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.         0.93333333 1.         1.         0.86666667 0.93333333\n",
      " 0.93333333 1.         1.         1.        ]\n"
     ]
    }
   ],
   "source": [
    "# 10-fold cross-validation with K=5 for KNN (the n_neighbors parameter)\n",
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9666666666666668\n"
     ]
    }
   ],
   "source": [
    "# use average accuracy as an estimate of out-of-sample accuracy\n",
    "print(scores.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.96, 0.9533333333333334, 0.9666666666666666, 0.9666666666666666, 0.9666666666666668, 0.9666666666666668, 0.9666666666666668, 0.9666666666666668, 0.9733333333333334, 0.9666666666666668, 0.9666666666666668, 0.9733333333333334, 0.9800000000000001, 0.9733333333333334, 0.9733333333333334, 0.9733333333333334, 0.9733333333333334, 0.9800000000000001, 0.9733333333333334, 0.9800000000000001, 0.9666666666666666, 0.9666666666666666, 0.9733333333333334, 0.96, 0.9666666666666666, 0.96, 0.9666666666666666, 0.9533333333333334, 0.9533333333333334, 0.9533333333333334]\n"
     ]
    }
   ],
   "source": [
    "# search for an optimal value of K for KNN\n",
    "k_range = list(range(1, 31))\n",
    "k_scores = []\n",
    "for k in k_range:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')\n",
    "    k_scores.append(scores.mean())\n",
    "print(k_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cross-Validated Accuracy')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xucm2d54P3fNUeNZ0aSD+OR7ZHjJOTk2J4xmJRzQlog9AAJAUqWtsCypdstvO0WWKD0ZWlKXlqgpe9uWbq0UMiWEiCcUhoaaEjoiUIcLNlOnBjHSayxZ+zxQZqTNTOSrv3jeTSWZUnz6DQaaa7v5zOfkZ6T7sca69J9um5RVYwxxphKtTW6AMYYY5qbBRJjjDFVsUBijDGmKhZIjDHGVMUCiTHGmKpYIDHGGFMVCyTGGGOqYoHEGGNMVSyQGGOMqUpHowuwHDZs2KDbtm1rdDGMMaapPProo6dVdWCp41ZFINm2bRt79+5tdDGMMaapiMizXo6zpi1jjDFVsUBijDGmKhZIjDHGVMUCiTHGmKpYIDHGGFOVugYSEblFRJ4UkSMi8v4C+y8TkQdFZL+IPCwiQzn7PiYij4nIIRH5HyIi7vbnicgB95qL240xxjRG3QKJiLQDnwJeDWwH7hCR7XmHfQK4W1V3AXcCH3XPfRHwYmAXsAN4PnCje86ngXcAV7k/t9TrHowxxiytnjWSG4AjqnpUVeeBe4DX5h2zHXjQffxQzn4FfEAX0A10AidFZBPgV9UfqrNG8N3ArXW8B7MKPfrsOSKxeKOLUTNzqTRf+vEx0pnaLaudXEhzz4+PkanhNU3zqmcg2QLEcp6PuttyRYHb3ce3Af0isl5Vf4gTWMbcnwdU9ZB7/ugS1wRARN4hIntFZO/ExETVN2NWj/d9bT8f/MaBRhejZu4/MMYHvn6Af/5p7f4ffHv/GO//+gH2xc7V7JqmedUzkBTqu8j/+vIe4EYR2YfTdHUcSInIc4DrgCGcQHGziLzM4zWdjaqfUdU9qrpnYGDJGf7GADCZXOCpiWmeGJ/i/Hy60cWpicgxp3ZVy1pWxA0gJ+LJml3TNK96BpJRIJzzfAg4kXuAqp5Q1dep6m7gg+62BE7t5N9VdVpVp4HvAC9wrzlU6prGVOPAaAJVSGeUx04kGl2cmoiMOvcRrWEgicaca56ctEBi6htIHgGuEpHLRaQLeBNwX+4BIrJBRLJl+ADwOffxMZyaSoeIdOLUVg6p6hgwJSIvcEdr/RrwrTreg1llcr+1t0I/yVwqzaETkwBERxM4XYvVSS6kOTTmXHMsYYHE1DGQqGoKeCfwAHAI+IqqPiYid4rIa9zDbgKeFJHDwCBwl7v9XuAp4ABOP0pUVf/O3febwF8BR9xjvlOvezCrTyQW5/INvWwJ9rREIDk0NsV8OsNLr9rA2Zl5YmfPV33Nx05MknI72cetRmKoc/ZfVb0fuD9v24dyHt+LEzTyz0sDv1HkmntxhgQbU1OqSiQW5yXP2cB8KkN0tPkDSbY56y0v3MY///Q0kdE4W9evqck1r9jQy0mrkRhsZrsxi8Ynk0xMzTE8FGA4HCB29jxnpucaXayqRGNxBvq7ufGaAXydbTXpJ4mOxgn5fewaCliNxAAWSIxZlB3dNBwOMjwUBGj6WkkkFmd4KEhnexs7Ngdq0lwXicUZDgcIBXo4NTlnc0mMBRJjsiKjcTrbhe2b/ewcCtAmEIk178itxOwCR0/PsHurExRHwkEOHk+wkM5UfM1zM/M8e2aWkfBaQv5u5tMZzs7O16rIpklZIDHGFTkWZ/smP90d7azp6uDqwf6m7nDP1qaytavhcJC5VIYnx6cqvmYke81wgFDAB8C49ZOsehZIjMGZN3LgeIKRcHBx2+6tQaKxeE2GzDZCtj9kVzgAsHhv1QTHaCyOCOzcEmDQ7wQSm0tiLJAYAxw5Nc3sfJrhnEAyPBQkcX6BZ8/MNrBklYuOxrlyoBe/rxOAobU9rO/tqqrDPRqLc9XGPvp9nRdqJBZIVj0LJMZwIeXHRYGkBt/gGyU7lDn3fkSE4XCw4vtZvKbbVDbQ102bYEOAjQUSY8DpVPf7Orh8fe/itqsH+1nT1d6UgeR4/Dynp+fZnRNIwGneOjIxzVRyoexrxs6e59zswmJw6mhvY6C/22okxgKJMeA02QyHg7S1XcgL2t4m7NhSmyGzyy1b5uG8QDIcDqIKB46XPxot29Ge248U8vssTYqxQGLM+fk0T56cWmyyyTUSDvL4iUnmU5UPmW2EaCxOV0cb14b8F20fHnI63isJjpFjcbo72rgm1L+4bdDvs852Y4HEmIMnEqQzetE37ayRcJD5dIYnxicbULLKRWMJrt/sp6vj4v/iwTVdXL6ht6IO9+honJ1bAnS2X7hmKOCz4b/GAokx0SLNQLnbmql5K5XOcOB4omANC5xaSbTMiZYL6QwHjycu+Tca9PuYTKZaZu0WUxkLJGbV2xeLsyXYw0B/9yX7Ngd8DPR3N1UgOXxymvML6cUZ7flGwkHGJ5Nl1SSeHJ9iLpW5JJBssiHABgskxhCNxQs2a4E7ZHYoWNNFoeotf0Z7vkpqWdlj80eBhdxJiWOJ6tPTm+ZlgcSsaqen5xg9d55hd/Z3ISPhAE9NzJA4X/6Q2UaIHIsTXNPJZUXSxV+3yU9nu5QdSNb1djG0tuei7YMBm91uLJCYVS5b0xgJry16THbfgdHmSOAYHXUmDTqLiF7K19nO9k3+smpZ0Vic4aHAJdfM1kjGE82dbt9UxwKJWdWisThtAju2+Ises3NxyOy55SpWxWbmUhw+OVVw4ECu4XCQA8ed0WpLmUoucGRiumCw7e3uoL+7w2okq5wFErOq7YvF3RnsxRcLDfR0csVAb1OklD9wPEFGnea4UoaHgkzPpXhqYnrpa44mUKVo89+gDQFe9SyQmFVLVYnG4kVHN+UacXNUrfRMwItDmYt0tGeNbPXe4V5oRnuuTQEfY1YjWdUskJhV65kzs0wmU0t+6ILzIXp6eo4TK/ybdyQWJ7yuh/V9lw5lznX5+l76fR2e+kmisTjb1q8huKar4P5Bv88SN65yFkjMqpXt8xjxWCMBVvwwYGcoc/GBA1ltbbJYy1pKfhbhfCG/j4npOU/9LaY1WSAxq1Y0lmBNVztXbexf8thrQ3662ttWdCA5NZnkRCK5mE9rKcNDQZ4YnyK5UHxW+ngiycnJuaLNWuD0kaQzyulpG7m1WlkgMavWvlicHVsCtLcVHiabq6ujje2b/exbwYEkEivdl5FvOBwknVEeO1F8EEGxLMK5LgwBtuat1coCiVmV5lJpDp2YvGSmdikj4SAHRhOk0iszE3B0NL6Y+t6L7CisfceKB8dILE5nu7B9U/Hh0dk0KZZOfvWyQGJWpSfGpphPX5o7qpSRcJDzC2l+emrpIbONEInFuTbUj6+z3dPxG/t9bAn2EC0x0TIai3PdJn/Ja9ra7cYCiVmVvDTZ5BtewR3umYyyP3Zpdt6lDIcDRSdapjPK/tH4kqPa1vd20dkulrhxFbNAYlalaCzOQH83m91mGS+2rV9DoKdzMSniSnL09AxTcynP/SNZI+EgsbPnOVOgo/ypiWlm5tNLXrOtTdjYb0OAVzMLJGZViiyRj6oQEWE4HCzZp9Ao5Xa0Z2VrG/sLNG+VU2sb9Nva7atZXQOJiNwiIk+KyBEReX+B/ZeJyIMisl9EHhaRIXf7y0UkkvOTFJFb3X2fF5Gnc/aN1PMeTOtJzC5wdGLG04z2fCPhIIdPTjE7n6pDySoXjcXp6+7gyoG+ss7bORSgTSg4Gi0Si9Pv6+CKDb1LXicU8FkgWcXqFkhEpB34FPBqYDtwh4hszzvsE8DdqroLuBP4KICqPqSqI6o6AtwMzALfzTnvvdn9qhqp1z2Y1rT/uLc0IoWMhANkFA4eX1lL72aXwfUylDnXmq4Orh7sL9jv42T8DdLm4Zohfw/jieSKTyFj6mPJQCIivygilQScG4AjqnpUVeeBe4DX5h2zHXjQffxQgf0Arwe+o6qzFZTBmEtkPzR3epy4lysbfFZSJuDkQppDY5Nld7RnjYSDREcvziOWXEjzxPhUyXVacoUC3czOp5maW1k1NbM8vASINwE/FZGPich1ZVx7CxDLeT7qbssVBW53H98G9IvI+gKv/6W8bXe5zWGfFJGCSYVE5B0isldE9k5MTJRRbNPqIrE4Vw70EujpLPvc9X3dhNf1lL3meT09PjbJQlrL7h/JGgkHic8u8OyZC9/VDrop5r2kW4GcIcDW4b4qLRlIVPVXgN3AU8Bfi8gP3Q/ppfJKFKoP59d73wPcKCL7gBuB48DiVxoR2QTsBB7IOecDwLXA84F1wPuKlPszqrpHVfcMDAwsUVSzWqgqkQqGyeYaHvKWo2q5RI5V1tGetTisOWc02mJHu8da2+LsdusnWZU8NVmp6iTwNZzmqU04tYefiMi7Spw2CoRzng8BJ/Kue0JVX6equ4EPuttyv+q9EfiGqi7knDOmjjngr3Ga0Izx5Hj8PKenS+eOWspIOMjx+HlOTa2MD83oaJxBfzehMoYy57pqYx89ne0XjUaLxOJsDvjY6Pd2zexrW5qU1clLH8kvicg3gO8DncANqvpqYBinRlHMI8BVInK5iHThNFHdl3ftDTn9Lx8APpd3jTvIa9ZyaymIM27zVuDgUvdgTFa2SaraQAKwf4U0bzkZfyu/n472NnYOBS6qkURH456yImcNWr6tVc1LjeQNwCdVdZeqflxVTwG4nd//sdhJqpoC3onTLHUI+IqqPiYid4rIa9zDbgKeFJHDwCBwV/Z8EdmGU6P5Qd6lvygiB4ADwAbgIx7uwRjA+YDs6mjj2lDx3FFLuX6zMzpqJTRvnZuZ55kzs1U11YETHB87Mcl8KsOZ6TliZ8+XNarN19nO2jWd1rS1ShVfX/SC/w6MZZ+ISA8wqKrPqOqDxU8DVb0fuD9v24dyHt8L3Fvk3Ge4tHMeVb3ZQ5mNKShyLM71m/10dVQ+8r2nq51rQ/0rYoZ7dInVC70aCQeZT2V4YnxyMR18ucFp0O+zfFurlJf/TV8FctOdpt1txjSVVDrDgeOJiuaP5BsOB4nG4mQavJhTNJZABHZ6zPhbTG4esUgsQVsF17RJiauXl0DS4c4DAcB9XHjNTWNWsJ+emub8wtK5o7wYGQoymUzx9JmZGpSscpHYOZ4z0Ee/r/yhzLk2B3xs6OsmEksQjcW5erCf3m4vDRYXhPw+xhO2uNVq5CWQTOT0aSAirwVO169IxtRHpfmoCsl2RDcyE7CqEh1N1OR+RJyld/fFzjkd7RVcc9Dv4/T0HPOplblei6kfL4HkPwO/JyLHRCSGM2/jN+pbLGNqLxqLE+jp5LL1a6q+1pUDffR2tTe0wz129jxnZ+ar7mjPGgkHODoxQ3x2oaJrZhe4WinDos3yWbLuqqpPAS8QkT5AVHWq/sUypvYisTjD4fIy/hbT3ibOkNkGBpJIjTras3KDRyX9SIOBCwtcDa2tPlib5uGpEVREfgG4HvBl/xOq6p11LJdpQaqKKp6SANbazFyKwyeneOX1oZpdcyS8ls/+y1HGEufpaFv+FRl+/PQZujvauCa0VJIJb3a5waOns52rB8vLIgy5a7fXv58kk1FEqMmXAlO9JQOJiPwFsAZ4OfBXOEkUf1zncpkW9K4v7UMVPvXm5y77ax88niCj3lN+eLF7a5CFtPLCj36/Ztcs157L1tLZXpsgFujp5Dkb+1jf20VHBddczjQpb/38I1y2bg1/eOuOur+WWZqXGsmLVHWXiOxX1T8QkT8Bvl7vgpnWkskoPzg8Aeo8Xu5aSS072rNuvnYjn3jDMOcX0jW7ZrlecPm6ml7vU//huXRXOMcmuKaTro62us8lSS6k+eFTp5k8X7svBaY6XgJJ9q9iVkQ2A2eAy+tXJNOKnj4zw1TSycd59PQ0z9lYm+YYr6KjccLreljfVzBZdEU629t4/fOGana9laCaZjIRYVPAx1id06Rksx3b5MeVw8tXj78TkSDwceAnwDNcmtbdmJJyO6UjDchRFY3VZiKiKW3QX/+127N/S6em5kg3eEKocZQMJG5CxQdVNa6qXwMuA67NTXNijBeRWJzernb6ujuWfVGoU1NJjsfP17RZyxQW8td/dnu2mTKdUc5M2wTIlaBkIFHVDPAnOc/n8tK8G+NJNBZn11CQXUOBZV8UqhYZf4032TQp9VxyNxqLs6arHaDuzWjGGy9NW98VkdvFxtmZCiUX0jzuLgU7HA5yaGyS5DJ2UEdjcdrbhOs3W+dsvQ36fcynMsRnF5Y+uALZbMcvv2YjYAtprRReAsnv4iRpnBORSRGZEpHJOpfLtJBDOUvBjoSDpDLKYyeW708oEotzbaifHvdbrKmf7BDgetUUstmOX7XDmQ9kHe4rg5eldvtVtU1Vu1TV7z6vfDEHs+pEc4bejoSXN0dVJqNER+M1SyNiSgvlzG6vh2y245uuGaCjTWwhrRXCy4TElxXarqr/VPvimFYUiV28FGzI71u2HFVHTzvDjkdsxNayWFxyt06BJBI7x1Ub+/D7Ohlcho59442XeSTvzXnsw1kj/VHAFpgynuRnqB0JB5dtUajF2lAZy8aaym3s70akPkvuZrMd/9x1Tv/IoL/baiQrhJemrV/K+XkFsAM4Wf+imVYQn53n6dMzFycEDAd59sws52bmS5xZG9FRZ9jxlQPl544y5etsb2N9b3ddmrZGz12c7dgW0lo5KsmFMIoTTIxZUnT00qG32ceRZaiVRNxhx+0NSBS5WoUC3XXpbN/n1i6zE0uXY/Kj8cZLH8n/BLKDwtuAESBaz0KZ1hGNxS9ZCnbnUAARZ192GGc9JBfSHBqb5O0vuaJur2EuFfL3MHputubXjcbi+DovZDsO+X3MzKeZSi5UvUKkqY6XPpK9OY9TwJdU9V/rVB7TYiKx+CVLwfZ1d3DVxr66d7hfGHZs80eWUyjQzd5nz9b8upFYnB2bA4vZjnNHiFkgaSwvgeReIKmqaQARaReRNapa+68cpqWoKtFYnJuvvbTWMRIO8r3HT6KqdVtT4kLG37V1ub4pLOT3EZ9dILmQxtdZm7k7C+kMB48n+JUXXHbR64AzZ2W5k4Cai3npI3kQ6Ml53gP8Y32KY1rJ6LnznCmyFOxwOMi52QViZ8/X7fWjecOOzfIY9Nd+LsmT41PMpTIX9bUtDjW2fpKG8xJIfKo6nX3iPrZ1NM2SSq0Bku0w3VfHBI7RUcv42wjZD/hadrgX+luqR8AylfESSGZEZHFJOxF5HlC/r5GmZURj8aJLwV4T6sfX2Va3BI7ZYcc2f2T5herwAR+NxVnX28XQ2guNI77OdoJrOm0I8ArgpY/kd4CvisgJ9/km4JfrVyTTKiKxODu2BAouBdvZ3saOzYG6pZRfHHZsNZJlV48mp0gszkg4eEl/WsjvW5Y14k1pXiYkPgJcC/wm8F+A61T10XoXzDS3hXSGgycSJVO3j4SDHDwxyUI6U/PXjxxzhx3XcI12402/r5Pervaa1RSmkgscmZgu2EzpTEq0BpJGWzKQiMhvAb2qelBVDwB9IvJf6l8008wOn5wiuZApmSxxOBxkPpXhyfGpmr9+dPTSYcdm+QwGfDVr2jpwPIEqDBcYxm01kpXBSx/Jr6vq4oB/VT0H/LqXi4vILSLypIgcEZH3F9h/mYg8KCL7ReRhERlyt79cRCI5P0kRudXdd7mI/EhEfioiXxaRLm+3apbTYudoiaalbG1lX43nk2SHHVvG38ZxPuBrE0hKDdoY9Ps4MzNXl1qt8c5LIGnLXdRKRNqBJT+83eM+Bbwa2A7cISLb8w77BHC3qu4C7gQ+CqCqD6nqiKqO4CSHnAW+657zx8AnVfUq4Bzwdg/3YJZZtnM0vK6n6DFDa3tY39tV85Ty2WHHtiJi49QykERjcbatX0NwzaUfO6GAD1Vn/XbTOF4CyQPAV0TkZ0XkZuBLwD94OO8G4IiqHlXVeeAe4LV5x2zHmacC8FCB/QCvB76jqrNuQLsZZ5IkwBeAWz2UxSyzaCzB8FCg5GRDEWE4HKx5ICn1DdYsj1DAx6mpOTKZ6pfcjcYSRWuX2RFiNpeksbwEkvfhfNj/JvBb7uP3ljzDsQWI5TwfdbfligK3u49vA/pFZH3eMW/CCV4A64G4qqZKXBMAEXmHiOwVkb0TExMeimtqZXouxeFTU56aloaHghyZmGYqWbulWUsNOzbLIxTwkcoop2eqqymMJ5KMTyaLfikYtECyIngZtZVR1b9Q1der6u3A/cC7PVy70FfR/K8n7wFuFJF9wI3AcZx8Xs4FRDYBO3FqRV6vmS33Z1R1j6ruGRgY8FBcUysHRp3OUS81gpGtQVSdc2ql1LBjszwWJwtW2RGerV0W+1Kyqc4LaRlvPP1PE5ENIvKbIvJPwMPAoIfTRoFwzvMh4ETuAap6QlVfp6q7gQ+623I/Ud4IfENVs19XTwNBEcnOf7nkmqbxInnpvksZdofn1qrDPTvs2Ga0N9Zik1OVH/CRWJzOdmH7psKrewfXdNLV0Waz2xusaCARkX4R+TUR+Qfgx8BzgCtU9UpVfY+Haz8CXOWOsurCaaK6L+81NohItgwfAD6Xd407uNCshaoqTl/K691NbwG+5aEsZhllO0fX9i49oC64povLN/TWrJ8kO+zYZrQ31oVJidXN8YjG4ly3yV80+aOI1LRj31SmVI3kFM6IqLuAK1X13YDnJe3cfox34jRLHQK+oqqPicidIvIa97CbgCdF5DBOLeeu7Pkisg2nRvODvEu/D/hdETmC02fyWa9lMssjOlre0NvhoUDNlt71MuzY1N+Gvm7a26SqGkk6oxw4vnTtMmRrtzdcqRQpv4dTi/g08Lci8uVyL66q9+P0qeRu+1DO43u5MAIr/9xnKNCRrqpHcUaEmRXo5GSSsUSyrKal4XCQb0ZOMJY4z6ZA8eHCXngZdmzqr71N2NjfXdVkwacmppmeSy3Z1zYY8LF/GVbbNMUVrZGo6idV9WeA1+B0cn8T2Cwi7xORq5ergKa5LNYIymhayn5Q1KJ5y8uwY7M8Bv3VzW5fqqM9a1PAx1giidPybRrBy6ito6p6l6ruBJ4PBIDv1L1kpilFYnE62op3jhZy3SY/ne1SdYd7OcOOTf1V2+QUicXp93VwxYbekscN+n3MpzLEZ2s3hNyUp6zxkap6QFV/T1WvrFeBTHNbqnO0EF9nO9dt8lddI8kOO7ZAsjKEAtV1gkdjcYaHgrS1la5d1mqEmKmcDbQ3NZPJKPtHS2f8LWYkHOTAaIJ0FTOhraN9ZRn0+5ieSzE9l1r64DzJhTRPjE8VTNSYLxToBiyQNJIFElMz2c7RSmoEw0NBZubTHDk1vfTBRURjcS7zOOzY1N+mKtYlOXjc+VLhZdDGhcmPFkgaxQKJqZlqclxlO+erad6KjsYtv9YKUs1SuOX8LW3sr/3SvqY8RYf/isgBiqQfAXAz9hqzKDoap7976c7RQi5f30u/r4PIaJw3Pj+89Al5Khl2bOqrmpUSo6MJNgd8bHSDUSldHW1s6Ou22e0NVGoeyS+6v3/L/f1/3N9vxknrbsxFIrE4u8KBJTtHC2lrE4aHgkSOVVYj8TpU1CyfajrBI7FzZb2XoUC39ZE0UKl5JM+q6rPAi1X1v7kjtg6o6vuBVy1fEU0zSC6keWJsqqqmpZFwkCdPTnF+Pl32uVF32PH1m70POzb11dPVjt/XUXaN5Mz0HLGz58v6W7I0KY3lpY+kV0Rekn0iIi8Cym+7MC3tsRMJUh47R4sZDgdJZ5SDJ8rPBBypYNixqb9NgZ6yawrZdDnl1EiqnfxoquMlkLwd+JSIPCMiTwP/C/iP9S2WaTaRmPPhX02NJDvUs9wO9+ywYy9DRc3yqmTt9kgsQZvAzi3e38+Q38e52QWSC+XXZk31SvWRAKCqjwLDIuIHJC/NuzGA8+HvtXO0mI39PrYEexb7O7w6ejqbk2ltxa9t6iPk7+aJscmyzonG4lw92E9v95IfTxdeJ3BhhNhl663BZLktWSMRkUER+SzwZVVNiMh2EbF10s1FIrHyMv4WMxwOlB1I9h3LDhW1GslKE/L7mJieYyGd8XS8qjrZo8tsIq1mhJipnpemrc/jpILf7D4/DPxOvQpkms/ZmXmOnZ2tyRyOkXCQ0XPnOT3tPWvshWHHfVW/vqmtwYAPVZiY8vZ+PntmlvjsQtnryVialMbyEkg2qOpXgAwsrjNiDZFmUbSGQ2+z30TL6SepZtixqa9yP+DLWV0z12Cg8smPpnpeAsmMiKzHnZwoIi8ArJ/ELIrE4mV3jhazY0uANvEeSLLDjm0i4sq02HfhsckpEovT09nO1YPl1S77uztY09Vus9sbxEtv1u/iLJF7pYj8KzAAvKGupTJNJTpafudoMb3dHVw92E9k1Nt3lcdOTJLKqKVGWaHKrZFER+Ps3BKgo7287E3ZJXetRtIYXt6tx4AbgRcBvwFcDzxRz0KZ5qGqi+m+a2UkHCQai3taqKia/F6m/tb1dtHV3uYpkMynMjx2YrLiYdzVpq03lfPyFfKHqvpcnIACgIj8BHhu3UrVwr5zYIz7oicaXYyaWUhnODe7UNPUJMPhIPc8EuM/fWEvXR2lv+s8dmKSTVUOOzb1IyJs9Hfz7egYx86Uzqw0O59mPpWp+G8p5Pfxo6fPVnQuwFf3xljf18XN1w5WfI1C1xzo7+amazbW7JorUamkjSGcNdN7RGQ3znK7AH5gzTKUrSV99l+e5tDYJFvWts6a4sNDAW66ZqBm17vpmgGGw0Fi55ZO6ebrbOOXK0jyaJbPbbu38MBj4zw1sfQSAc/dGuTFV26o6HWykx8zGS174IWq8pG/P8S2Db01CySZjPKH336c4XBw9QYSnHxabwWGgD/N2T4F/F4dy9TSxhJJXnl9iE/+8kiji7JibQr08K3fenGji2Fq5N2vvIZ3v/Kaur9OyO8jlVFOz8wtppb36tkzsyTOL3DoxCRzqTTdHdWn2nnmzAyTydSq6LcpGkhU9QvAF0TkdlX92jKWqWVPWac8AAAfnklEQVRlMsqpqeTiSBZjTO1cWOCq/ECS7WubT2c4VGXy0fxrroZ+Gy8pUr4mIr+A08nuy9l+Zz0L1orOzs6zkNbFkSzGmNpZXJFxMslOyuuwj7jZo1MZZ/BILQJJdgj7ZDLF+fk0PV2tm1DUS4qUvwB+GXgXTj/JG4DL6lyulpT9ZjJogcSYmgsFKp/dHh2N89ytaxno765qlc5cuUPYW33GvZfhvy9S1V8DzqnqHwAvBKx3swLZtlJr2jKm9jb0ddPeJmWv3Z4ddjyyNchIOFh2rrdC5lJpDp2YXJyk2+rNW14CyXn396yIbAYWgMvrV6TWlZ11a01bxtRee5sw0Ndd9uz2J8YnnWHHQ04gOXp6hsTsQlVlOTQ2xXw6wy07QgCMT55f4ozm5iWQfFtEgsDHgZ8AzwD31LNQrerkZJI2gQ19XY0uijEtqZL1Ty7kigtcyPU2Wl2tJHLsHACvut4ZSjye8J6EtBl56Wz/Q/fh10Tk24DP1iSpzHgiycZ+X9npH4wx3oT83Tw1MVPWOfticTb0dbMl2IO/pxNwgsvLrq58blR0NMHG/m6uHOijr7uj5YcAl5qQ+LoS+1DVry91cRG5Bfj/gXbgr1T1j/L2XwZ8Did/11ngV1R11N23FfgrnP4YBX5eVZ8Rkc/jpGzJBrO3qmpkqbKsBOOTycUspcaY2tsU6OHfjpwp6xxnlFYAEcHv6+TKgd6qayTZkV8iwqC/u+X7SErVSH7J/b0RJ8/W993nLwceBkoGEhFpBz4FvAIYBR4RkftU9fGcwz4B3K2qXxCRm4GPAr/q7rsbuEtVvycifbhp7F3vVdV7l7q5lebkZJLLN9jqbcbUy6Dfx9Rcipm5lKckopPJBZ6amOG23VsWt42E1/KDw6dQVUTKX5ogMbvA0dMz3P68IaCydeubTdE2FlV9m6q+Dac2sF1Vb1fV23Hmk3hxA3BEVY+q6jxOv8pr847ZDjzoPn4ou19EtgMdqvo9tyzTqrp0vowVbiyRtI52Y+ooFOgGvA+33R9zGjZy83uNhAOcnp7neLyyDvJsbSY7F2VwFWQl9tJYv01Vx3KenwSu9nDeFiCW83zU3ZYrCtzuPr4N6HfXPrkaiIvI10Vkn4h83K3hZN0lIvtF5JMi0l3oxUXkHSKyV0T2TkxMeChufc3Op5hKpqxpy5g6ys7R8tqUlP3Q35WTvTobVKKxyrqCo7E4IrBzyBn6Gwp0c2pqjnRm6WzWzcpLIHlYRB4QkbeKyFuAv8epPSylUJ0w/1/yPcCNIrIPp9/jOJDCaXJ7qbv/+cAVOHm/AD4AXOtuXwe8r9CLq+pnVHWPqu4ZGKhdQsFKjdvQX2PqLlRmINl3LM4VA70E3E52gGtDfro62ojEzlVUhkgszpUDffh9nYtlSme0rOWjm82SgURV3wn8b2AYGAE+o6rv8nDtUS6euDgEXJQ/XVVPqOrrVHU38EF3W8I9d5/bLJYCvombtl5Vx9QxB/w1ThPaijdukxGNqbtyZrerKpFYnJG8tXS6Otq4frO/ohqJqhIdvTjFSrm1pGbkaRyqqn5dVf+r+/MNj9d+BLhKRC4XkS7gTTgrLS4SkQ0iki3DB3BGcGXPXSsi2arEzcDj7jmb3N8C3Aoc9Fiehlqc1W41EmPqZk1XB36ft+G2Y4kkp6fnCq5/MjwU5MDxBKl0psCZxR2Pn+f09PxF19wUcJaMaOUO96KBRET+xf09JSKTOT9TIjK51IXdmsQ7gQeAQ8BXVPUxEblTRF7jHnYT8KSIHAYGgbvcc9M4zVoPisgBnGayv3TP+aK77QCwAfhI2XfdANkJSVYjMaa+vK6UWGp1zd1bg5xfSHP45NJrqBS8Zk4tZ9AdANDKHe6l0si/xP3dX+nFVfV+4P68bR/KeXwvUHAYrztia1eB7TdXWp5GGk+cp9/XwZqu6tc1N8YUN+j3efr2H43F6Wpv49pNl37E5c5w377Z7/m1o7E4XR0XX3NDbzcdbVJ26pZmUqpGsq7Uz3IWshWMT9rQX2OWQ8jvvUZy3WZ/wUWsLlu/huCaTiLHypuYGInF2bHZT2dO9oq2NmFjf3fZySSbSamvx4/ijLIqNvrqirqUqEWNT85Zs5YxyyAU8HF6eo5UOlM0HVE6oxw4nuCNewonMhcRhoeCZc1wT6UzHDie4I4btl6ybzDgrZbUrEpNSLxcVa9wf+f/WBAp00mbjGjMsggFfGQUJkoMt/3pqSlm59MMh4svgDUcDnL45BQzcylPr3v45DTJhUzBPpdNqzWQ5BKRtSJyg4i8LPtT74K1klQ6w8S01UiMWQ5e5pJEFzva1xY9Znc4SEbhwHFvw4BLdd4P+n0t3bTlZYXE/wT8E87oqz9wf3+4vsVqLaen50ln1FZGNGYZeJm3EYnF8fs62LZ+TdFjdrkz072umBiNxVm7ppOt6y69ZsjvY2Y+zVSyunVOViovNZLfxplF/qyqvhzYDTQ+50gTGbc5JMYsGy+TEiOxBMNudt5i1vd1E17X47mfJDoaL3rNxTK1aK3ESyBJqmoSQES6VfUJ4Jr6Fqu1LKZHsaYtY+pu3ZouOtulaCCZnU9x+OQUuws0QeUbCa/1NHJrZs655vBQ4Wsu1pJatJ/ESyAZdVdI/CbwPRH5FnmpTkxptla7McunrU1K9kkcPD5JOqMFZ7TnGx4KcCKR5NQSAeDA8QQZLdw/AuXnAGs2XlZIvM19+GEReQgIAP9Q11K1mPHJJJ3twro1tsSuMcshVGJS4oWldb3USJxjIrE4r7w+VPS4yBLXzH6JbNXZ7aUmJP69iLxZRBZXYlLVH6jqfe76Isajk+4Su21t5S+SY4wp32CJNCmRWJyhtT1s6Cu4AsVFdmwJ0N4mS/aTRGNxtq5bw7rewl8WfZ3tBNd0rsqmrc8Avwg8IyJfFpFb3eSLpkxjiaQ1axmzjLI1EtVL1wCJxOKeaiPgBIBrQ/1LZgLOLq27ZJkSrZlKvtSExG+p6h3AVpxldd8CHBORz4nIK5argK3gpKVHMWZZhfw+kgsZJs9fPJlwYmqO4/Hznjras0bCQaKxOJkiC1OdmkxyIpFcMjg5OcAqW3VxpfOyHsl5Vf2y21fySpzhv9ZH4pGqOnm2rEZizLIZLDIEuJz+kazhcJCpuRRHT88U3H9hImLxWfKwSmskWSIyKCLvEpF/xRm59V3geXUvWYuYmksxO5+2Gokxy2hTsUAyGqe9TdixufSHfq6RxaV3C/eTREfjdLQJ1y9xzVDAx5mZORbKXOOkGZTqbP91Efk+8BOcNdT/m5t7632qGlm2Eja57BBEW6vdmOVzYbjtxU1JkVicawb76em6NONvMVcO9NHX3bFY88gXicW5dlM/vs7S1wwFfKjCqanWq5WUqpG8CPgjIKyq71LVf12mMrWUMVur3Zhlt9HvjMjKbUrKZJRoGR3tWe1tws4tgYIjtzIZZX8sUXQiYq5WnktSqrP9bar6XVVdrIeJyIeXpVQtxNKjGLP8ujvaWdfbdVHT1jNnZphMppbsyyhkOBzk0NgkyYX0RduPnp5mai615IgtaO212z1l/83xmqUPMbmyTVvZb0jGmOUx6PddNAEwW6MolfG3mJFwkIW08vjYxauMR9xhwV4CiZccYM2q3EBiM+rKND6ZZF1v15Ltp8aY2tqUNykxcixOb1c7z9nYV/a1inW4R2Nx+ro7uGJg6WuuXdNJV0dbS85uLzeQ2GitMp2cTFr6eGMaIL9GEhlNsHPImalerlDAR8jvu6TDPRKLs8vjNUXE8zLAzcbL8N+PiYhfRDpxkjaeFpFfWYaytQRnrXZr1jJmuYX8Ps7MzDOXSjOXSnPoxGTZHe25hsOBi2okyYU0h8bKu2apHGDNzEuN5JWqOomTLmUUZyjwe+taqhYybulRjGmIUMD5Andqco5DY1PMpzOMeBhdVcxwOMgzZ2aJzzqpBh8fmySVUU8jtrIGA75V27TV6f7+eeBLqnq2juVpKfOpDKen561py5gGyF0DZHFp3a2VB5LcTMDA4jolu8u4ZsjfzViicA6wZuYlkPydiDwB7AEeFJEBoPVCah2cmrKhv8Y0yqZAD+C0CkRicTb2d1f1f3HnlgAiLCZwjI7GCfl9ZX1RDAV6mE9liM+21pK7XnJtvR94IbBHVReAGeC19S5YK7AFrYxpnGzQOOnWSJZaWncp/b5OnjPQRyR2DnBqJl6G/RYqU6v1k3jpbH8DkFLVtIj8PvA3wOa6l6wFZGfVWiAxZvn5ezrwdbbx5PgUR0/PlP2hX8hIOEh0NMHZmXmePTNbdud9tt9m1QUS4P9V1SkReQnwKuALwKfrW6zWMObm+bGmLWOWX3a47YNPnAK8TRpcynA4yNmZef7+wJj7vLxZ8tlmsGLLADcrL4EkmxPgF4BPq+q3AFvgyoOTk0m6O9oI9HQufbAxpuYG/T7OzswjAjuHyk+Nki8bjO7+t2cQgV1ljgLb2O8EkrFVGEiOi8j/Bt4I3C8i3R7PQ0RuEZEnReSIiLy/wP7LRORBEdkvIg+LyFDOvq0i8l0ROSQij4vINnf75SLyIxH5qbty44oNauOTc4QCvqraZY0xlcs2K1850IffV/0XumtC/XR3tPHTU9NctdHJClyOro42NvR1t9wQYC8B4Y3AA8AtqhoH1uFhHomItAOfAl4NbAfuEJHteYd9ArhbVXcBdwIfzdl3N/BxVb0OuAE45W7/Y+CTqnoVcA54u4d7aIiTCVsZ0ZhGygaScuZ6lNLZ3saOLYGqrhkKdK++PhJVnQWeAl4lIu8ENqrqdz1c+wbgiKoeVdV54B4uHe21HXjQffxQdr8bcDpU9XtuGaZVdVacr/Y3A/e653wBuNVDWSry9OkZfnLsXMXn28qIxjRW9otcNfNH8mWbtyq9ZiumSfEyauu3gS8CG92fvxGRd3m49hYglvN81N2WKwrc7j6+DegXkfU4s+fjIvJ1EdknIh93azjrgbiqpkpcM1vud4jIXhHZOzEx4aG4l/rQtw7y+984WNG5i0vsWo3EmIa5cqAPEbhh27qaXfNnLl+HCDy/wmvm5wBrBV6att4O/IyqfkhVPwS8APh1D+cV6hjIn875HuBGEdkH3AgcB1JAB/BSd//zgSuAt3q8prNR9TOqukdV9wwMDHgo7qWGh4I8eXKK8/PppQ/Oc252gflUxma1G9NAL71qAw+/5yauCfXX7Jqv2D7Iw++5iasHK7tmyO/j3OzCJWubNDMvgUS4MHIL97GX3uNRIJzzfAg4kXuAqp5Q1dep6m7gg+62hHvuPrdZLIWzVvxzgdNAUEQ6il2zlkbCQdIZ5eCJRNnnZquu1rRlTOOICJet711R18x+JrRSrcRLIPlr4Eci8mF3hcR/Bz7r4bxHgKvcUVZdwJuA+3IPEJENIpItwweAz+Wcu9ZNxwJOv8jj6iSoeQh4vbv9LcC3PJSlIrvcMeL5axB4kf0jsRqJMSbX4gJXLdRP4qWz/U+BtwFncUZJvU1V/8zDeSngnTgjvg4BX1HVx0TkThHJrrR4E/CkiBwGBoG73HPTOM1aD4rIAZwa0F+657wP+F0ROYLTZ+IlqFVkY7+PLcEe9lUQSLKjMjZZjcQYk6MV06SUHATt1hb2q+oO4CflXlxV7wfuz9v2oZzH93JhBFb+ud8DdhXYfhRnRNiyGAkHK6qRjCeSiMBAv61FYoy5YHC1NW2pagaIisjWZSrPijMcDjB67jynp+fKOm88kWRDXzed7eUuQmmMaWX93R2s6WpfzMXXCrxMy9wEPCYiP8bJ/AuAqr6m+CmtIzvpKBqL87PXDXo+z4b+GmMKyeYAa6UaiZdA8gd1L8UKtnMoQJuUH0hOTiYZWrumjiUzxjSrUMC3mNS1FRQNJCLyHGBQVX+Qt/1lOPM9VoU1XR1cPdhfdof7+GSSPdvW1qlUxphmFvL7+NHTrbPYbKkG/D8Dpgpsn3X3rRq7tzod7l6Xx0wupInPLiyu0GaMMbmya7dnMq2x5G6pQLJNVffnb1TVvcC2upVoBRoeCjKZTPHMmVlPx9scEmNMKSG/j1RGOTMz3+ii1ESpQFLqU3BVfdXOroKWXWJzKdm1Bqyz3RhTyKC/tYYAlwokj4jIJTm1ROTtwKP1K9LKc/VgP2u62onGvKVKubBWu80hMcZcKjtRuVUWuCo1aut3gG+IyJu5EDj24KyOeFu9C7aStLcJO7YEiHjscM+mPrCmLWNMIYtpUlqkRlI0kKjqSeBFIvJyYIe7+e9V9fvLUrIVZiQc5PP/+gxzqTTdHe0ljx2fTNLX3UF/DVZkM8a0ng193bS3Scus3b7kPBJVfQgnUeKqNhIOMp/O8MTY1GKfSTEnJ5MM+q1ZyxhTWHubMNDXOislWv4Ojy50uC/dvDWWsJURjTGlZYcAtwILJB5tDvgY6O/2lMDxZCJp/SPGmJI2tdCSuxZIPBIRhoeCREZLB5JMRjk1NWdDf40xJYUCFkhWpZFwgKMTMyRmF4oec3pmjlRGrWnLGFPSoN/H1FyKmblUo4tSNQskZRgJO7mz9h8vXis56aaGthqJMaaU7DyzVuhwt0BShp1DSy+9Oz5pa7UbY5a2OLu9BZq3LJCUIdDTyRUDvSVHbo27qaGtRmKMKSWb1NVqJKvQSDhIJJYomgl4fDJJe5uwvs/mkRhjimultdstkJRpJBzk9PQcx+OFF6UZT8yxsd+ZtWqMMcX0dLXj93W0xMgtCyRlGglnl94tnMDx5KRNRjTGeNMqQ4AtkJTp2pCfrvY2okXmk9ha7cYYrwZbZO12CyRl6upoY/tmP5FjRQKJzWo3xni0KeCzPpLVaiQc5MDxBKl05qLt03MppudS1rRljPEk5PcxMTV3yWdJs7FAUoGRcJDzC2kOn5y+aPu4rYxojCnDYMBHRmFieq7RRamKBZIKZDMB5/eT2FrtxphyLA4BbvIOdwskFdi2fg2Bns5LZrhn/xg2WdOWMcaDVlm73QJJBUSE4XDwkhnulh7FGFOO7JdOq5GUICK3iMiTInJERN5fYP9lIvKgiOwXkYdFZChnX1pEIu7PfTnbPy8iT+fsG6nnPRQzEg5y+OTURZk7xxNJAj2d+DpLL8VrjDEA63q76GpvY3zS+kgKEpF24FPAq4HtwB0isj3vsE8Ad6vqLuBO4KM5+86r6oj785q8896bsy9Sr3soZSQcIKNw8PiFiYk2h8QYUw4RYaO/25q2SrgBOKKqR1V1HrgHeG3eMduBB93HDxXYv2IND1269O7JySSD1qxljClDyO9jLFE45VKzqGcg2QLEcp6PuttyRYHb3ce3Af0ist597hORvSLy7yJya955d7nNYZ8UkYZkR1zf1014Xc9FI7fGE0lCfkvWaIzxzlm73Zq2iimUtTA/Ze57gBtFZB9wI3AcyHY6bFXVPcB/AP5MRK50t38AuBZ4PrAOeF/BFxd5hxuI9k5MTFR3J0UMDwUXZ7gvpDNMTM8RclNDG2OMFyF37fZiGcWbQT0DySgQznk+BJzIPUBVT6jq61R1N/BBd1siu8/9fRR4GNjtPh9Txxzw1zhNaJdQ1c+o6h5V3TMwMFDTG8saCQc5kUhyajLJxNQcqjYZ0RhTnk0BH+cX0kwmm3fJ3XoGkkeAq0TkchHpAt4E3Jd7gIhsEJFsGT4AfM7dvjbbZCUiG4AXA4+7zze5vwW4FThYx3soaTET8GgiZ+ivNW0ZY7xrhbkkdQskqpoC3gk8ABwCvqKqj4nInSKSHYV1E/CkiBwGBoG73O3XAXtFJIrTCf9Hqvq4u++LInIAOABsAD5Sr3tYyvWbA7S3CZHYucXlMm1WuzGmHNl5Z2NNPJeko54XV9X7gfvztn0o5/G9wL0Fzvs3YGeRa95c42JWrKernWtD/URjCTa4KyJa05YxphyhFli73Wa2V2k4HCQ6Gmc8kaSrvY11vV2NLpIxpolsdEd6NnM6eQskVRoZCjKVTPFvT51hMNCN03VjjDHedHe0s763ywLJajay1elwP3A8Yc1axpiKDPp91rS1ml050Edvl5NbyzrajTGVCDX5SokWSKrU3ibsctOlWI3EGFOJQXdSYrOyQFID2YWuLH28MaYSIb+PMzPzzKXSjS5KReo6/He1GAkHAGvaMsZUJrsuyav/7J9pb6vtgJ3PvuX5bF2/pqbXzGeBpAZuvHojv/7Sy3nZ1fVJxWKMaW03XjPAbbu31KVG0tVR/4YnaeZEYV7t2bNH9+7d2+hiGGNMUxGRR93kuSVZH4kxxpiqWCAxxhhTFQskxhhjqmKBxBhjTFUskBhjjKmKBRJjjDFVsUBijDGmKhZIjDHGVGVVTEgUkQng2bzNG4DTDShOvbTa/UDr3ZPdz8rXavdU7f1cpqpLpuxYFYGkEBHZ62XGZrNotfuB1rsnu5+Vr9Xuabnux5q2jDHGVMUCiTHGmKqs5kDymUYXoMZa7X6g9e7J7mfla7V7Wpb7WbV9JMYYY2pjNddIjDHG1MCqCyQicouIPCkiR0Tk/Y0uTy2IyDMickBEIiLSdAuviMjnROSUiBzM2bZORL4nIj91f69tZBnLVeSePiwix933KSIiP9/IMpZDRMIi8pCIHBKRx0Tkt93tTfk+lbifZn6PfCLyYxGJuvf0B+72y0XkR+579GUR6ar5a6+mpi0RaQcOA68ARoFHgDtU9fGGFqxKIvIMsEdVm3L8u4i8DJgG7lbVHe62jwFnVfWP3IC/VlXf18hylqPIPX0YmFbVTzSybJUQkU3AJlX9iYj0A48CtwJvpQnfpxL380aa9z0SoFdVp0WkE/gX4LeB3wW+rqr3iMhfAFFV/XQtX3u11UhuAI6o6lFVnQfuAV7b4DKteqr6T8DZvM2vBb7gPv4Czn/yplHknpqWqo6p6k/cx1PAIWALTfo+lbifpqWOafdpp/ujwM3Ave72urxHqy2QbAFiOc9HafI/HpcC3xWRR0XkHY0uTI0MquoYOP/pgY0NLk+tvFNE9rtNX03RDJRPRLYBu4Ef0QLvU979QBO/RyLSLiIR4BTwPeApIK6qKfeQunzmrbZAIgW2tULb3otV9bnAq4HfcptVzMrzaeBKYAQYA/6kscUpn4j0AV8DfkdVJxtdnmoVuJ+mfo9UNa2qI8AQTgvMdYUOq/XrrrZAMgqEc54PAScaVJaaUdUT7u9TwDdw/oCa3Um3HTvbnn2qweWpmqqedP+jZ4C/pMneJ7fd/WvAF1X16+7mpn2fCt1Ps79HWaoaBx4GXgAERaTD3VWXz7zVFkgeAa5yRzF0AW8C7mtwmaoiIr1uZyEi0gu8EjhY+qymcB/wFvfxW4BvNbAsNZH9wHXdRhO9T25H7meBQ6r6pzm7mvJ9KnY/Tf4eDYhI0H3cA/wcTt/PQ8Dr3cPq8h6tqlFbAO5wvj8D2oHPqepdDS5SVUTkCpxaCEAH8LfNdk8i8iXgJpxMpSeB/w58E/gKsBU4BrxBVZum87rIPd2E02SiwDPAb2T7F1Y6EXkJ8M/AASDjbv49nH6FpnufStzPHTTve7QLpzO9HaeS8BVVvdP9jLgHWAfsA35FVedq+tqrLZAYY4yprdXWtGWMMabGLJAYY4ypigUSY4wxVbFAYowxpioWSIwxxlTFAolpCSLysIi8Km/b74jI/1rivOlS+2tQrgE38+o+EXlp3r6HRWSP+3ibm531VQWu8XE3m+vHKyzDTSLy7ZznHxGRB0Sk2y3D3px9e0Tk4ZzzVER+KWf/t0XkpkrKYVqXBRLTKr6EM8E015vc7Y30s8ATqrpbVf+50AEiMgQ8ALxbVR8ocMhvAM9V1fd6ecGcWcyF9n0QeDFwa85cgo0i8uoip4wCH/Tyumb1skBiWsW9wC+KSDcsJuLbDPyLiPSJyIMi8hNx1m25JONzgW/tfy4ib3UfP09EfuAmxXwgb/Zz9vjL3NfY7/7eKiIjwMeAnxdnbYueAuUOAd8Ffl9VL8myICL3Ab3Aj0Tklwu9jnvc50XkT0XkIeCPC/0Dici7gZ8HfklVz+fs+jjw+4XOAaJAQkReUWS/MRZITGtQ1TPAj4Fb3E1vAr6szozbJHCbm9jy5cCfuCkyluTmY/qfwOtV9XnA54BCmQP+HGftkV3AF4H/oaoR4ENuOUbyPryz7gb+XFW/WuS+XgOcd8//cqHXyTn8auDnVPXdBS71YuA/A6/OSTWe9UNgTkReXqgMwEcoHmiMsUBiWkpu81Zus5YA/5+I7Af+ESeN9qDHa14D7AC+56bn/n2cxHf5Xgj8rfv4/wAv8Xj9fwR+VUTWeDy+1Ot8VVXTRc47gvPv8Moi+4sGi2yTXH4fjzFZFkhMK/km8LMi8lygJ7twEfBmYAB4npti+yTgyzs3xcX/H7L7BXjMrRGMqOpOVS32YZzLa+6hj+Hkq/pqqb4Nj68zU+K4kzjNWp8sVPNQ1e/j3PMLipx/F9ZXYoqwQGJahttk8zBO81NuJ3sAOKWqC+6H6GUFTn8W2O6OZArgdJIDPAkMiMgLwWnqEpHrC5z/b1yoDb0ZZ5lTr/4rMAl81kOTW8Wvo6qHgdcBf+P23+S7C/hvRc79LrAWGPb6emb1sEBiWs2XcD7s7snZ9kVgjzvM9c3AE/knqWoMJ4vtfvf4fe72eZwU3H8sIlEgAryowOv+P8Db3OazX8VZK9sTtx/nLcAmnBpKKRW/jvtajwBvA+4TkSvz9t0PTJQ4/S4KN+uZVc6y/xpjjKmK1UiMMcZUxQKJMcaYqlggMcYYUxULJMYYY6pigcQYY0xVLJAYY4ypigUSY4wxVbFAYowxpir/F19pSpGIa+FNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)\n",
    "plt.plot(k_range, k_scores)\n",
    "plt.xlabel('Value of K for KNN')\n",
    "plt.ylabel('Cross-Validated Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More efficient parameter tuning using `GridSearchCV`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Allows you to define a **grid of parameters** that will be **searched** using K-fold cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]\n"
     ]
    }
   ],
   "source": [
    "# define the parameter values that should be searched\n",
    "k_range = list(range(1, 31))\n",
    "print(k_range)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]}\n"
     ]
    }
   ],
   "source": [
    "# create a parameter grid: map the parameter names to the values that should be searched\n",
    "param_grid = dict(n_neighbors=k_range)\n",
    "print(param_grid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate the grid\n",
    "grid = GridSearchCV(knn, param_grid, cv=10, scoring='accuracy', return_train_score=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You can set **`n_jobs = -1`** to run computations in parallel (if supported by your computer and OS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, error_score='raise',\n",
       "       estimator=KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=30, p=2,\n",
       "           weights='uniform'),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "       scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit the grid with data\n",
    "grid.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.053333</td>\n",
       "      <td>{'n_neighbors': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.052068</td>\n",
       "      <td>{'n_neighbors': 2}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 3}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 4}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 6}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 7}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 8}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 9}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 10}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 11}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 12}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 13}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 14}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 15}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 16}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 17}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 18}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 19}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 20}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 21}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 22}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 23}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 24}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 25}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 26}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 27}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'n_neighbors': 28}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'n_neighbors': 29}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'n_neighbors': 30}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score  std_test_score               params\n",
       "0          0.960000        0.053333   {'n_neighbors': 1}\n",
       "1          0.953333        0.052068   {'n_neighbors': 2}\n",
       "2          0.966667        0.044721   {'n_neighbors': 3}\n",
       "3          0.966667        0.044721   {'n_neighbors': 4}\n",
       "4          0.966667        0.044721   {'n_neighbors': 5}\n",
       "5          0.966667        0.044721   {'n_neighbors': 6}\n",
       "6          0.966667        0.044721   {'n_neighbors': 7}\n",
       "7          0.966667        0.044721   {'n_neighbors': 8}\n",
       "8          0.973333        0.032660   {'n_neighbors': 9}\n",
       "9          0.966667        0.044721  {'n_neighbors': 10}\n",
       "10         0.966667        0.044721  {'n_neighbors': 11}\n",
       "11         0.973333        0.032660  {'n_neighbors': 12}\n",
       "12         0.980000        0.030551  {'n_neighbors': 13}\n",
       "13         0.973333        0.044222  {'n_neighbors': 14}\n",
       "14         0.973333        0.032660  {'n_neighbors': 15}\n",
       "15         0.973333        0.032660  {'n_neighbors': 16}\n",
       "16         0.973333        0.032660  {'n_neighbors': 17}\n",
       "17         0.980000        0.030551  {'n_neighbors': 18}\n",
       "18         0.973333        0.032660  {'n_neighbors': 19}\n",
       "19         0.980000        0.030551  {'n_neighbors': 20}\n",
       "20         0.966667        0.033333  {'n_neighbors': 21}\n",
       "21         0.966667        0.033333  {'n_neighbors': 22}\n",
       "22         0.973333        0.032660  {'n_neighbors': 23}\n",
       "23         0.960000        0.044222  {'n_neighbors': 24}\n",
       "24         0.966667        0.033333  {'n_neighbors': 25}\n",
       "25         0.960000        0.044222  {'n_neighbors': 26}\n",
       "26         0.966667        0.044721  {'n_neighbors': 27}\n",
       "27         0.953333        0.042687  {'n_neighbors': 28}\n",
       "28         0.953333        0.042687  {'n_neighbors': 29}\n",
       "29         0.953333        0.042687  {'n_neighbors': 30}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the results as a pandas DataFrame\n",
    "import pandas as pd\n",
    "pd.DataFrame(grid.cv_results_)[['mean_test_score', 'std_test_score', 'params']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 1}\n",
      "0.96\n"
     ]
    }
   ],
   "source": [
    "# examine the first result\n",
    "print(grid.cv_results_['params'][0])\n",
    "print(grid.cv_results_['mean_test_score'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.96       0.95333333 0.96666667 0.96666667 0.96666667 0.96666667\n",
      " 0.96666667 0.96666667 0.97333333 0.96666667 0.96666667 0.97333333\n",
      " 0.98       0.97333333 0.97333333 0.97333333 0.97333333 0.98\n",
      " 0.97333333 0.98       0.96666667 0.96666667 0.97333333 0.96\n",
      " 0.96666667 0.96       0.96666667 0.95333333 0.95333333 0.95333333]\n"
     ]
    }
   ],
   "source": [
    "# print the array of mean scores only\n",
    "grid_mean_scores = grid.cv_results_['mean_test_score']\n",
    "print(grid_mean_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Cross-Validated Accuracy')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xucm2d54P3fNUeNZ0aSD+OR7ZHjJOTk2J4xmJRzQlog9AAJAUqWtsCypdstvO0WWKD0ZWlKXlqgpe9uWbq0UMiWEiCcUhoaaEjoiUIcLNlOnBjHSayxZ+zxQZqTNTOSrv3jeTSWZUnz6DQaaa7v5zOfkZ6T7sca69J9um5RVYwxxphKtTW6AMYYY5qbBRJjjDFVsUBijDGmKhZIjDHGVMUCiTHGmKpYIDHGGFMVCyTGGGOqYoHEGGNMVSyQGGOMqUpHowuwHDZs2KDbtm1rdDGMMaapPProo6dVdWCp41ZFINm2bRt79+5tdDGMMaapiMizXo6zpi1jjDFVsUBijDGmKhZIjDHGVMUCiTHGmKpYIDHGGFOVugYSEblFRJ4UkSMi8v4C+y8TkQdFZL+IPCwiQzn7PiYij4nIIRH5HyIi7vbnicgB95qL240xxjRG3QKJiLQDnwJeDWwH7hCR7XmHfQK4W1V3AXcCH3XPfRHwYmAXsAN4PnCje86ngXcAV7k/t9TrHowxxiytnjWSG4AjqnpUVeeBe4DX5h2zHXjQffxQzn4FfEAX0A10AidFZBPgV9UfqrNG8N3ArXW8B7MKPfrsOSKxeKOLUTNzqTRf+vEx0pnaLaudXEhzz4+PkanhNU3zqmcg2QLEcp6PuttyRYHb3ce3Af0isl5Vf4gTWMbcnwdU9ZB7/ugS1wRARN4hIntFZO/ExETVN2NWj/d9bT8f/MaBRhejZu4/MMYHvn6Af/5p7f4ffHv/GO//+gH2xc7V7JqmedUzkBTqu8j/+vIe4EYR2YfTdHUcSInIc4DrgCGcQHGziLzM4zWdjaqfUdU9qrpnYGDJGf7GADCZXOCpiWmeGJ/i/Hy60cWpicgxp3ZVy1pWxA0gJ+LJml3TNK96BpJRIJzzfAg4kXuAqp5Q1dep6m7gg+62BE7t5N9VdVpVp4HvAC9wrzlU6prGVOPAaAJVSGeUx04kGl2cmoiMOvcRrWEgicaca56ctEBi6htIHgGuEpHLRaQLeBNwX+4BIrJBRLJl+ADwOffxMZyaSoeIdOLUVg6p6hgwJSIvcEdr/RrwrTreg1llcr+1t0I/yVwqzaETkwBERxM4XYvVSS6kOTTmXHMsYYHE1DGQqGoKeCfwAHAI+IqqPiYid4rIa9zDbgKeFJHDwCBwl7v9XuAp4ABOP0pUVf/O3febwF8BR9xjvlOvezCrTyQW5/INvWwJ9rREIDk0NsV8OsNLr9rA2Zl5YmfPV33Nx05MknI72cetRmKoc/ZfVb0fuD9v24dyHt+LEzTyz0sDv1HkmntxhgQbU1OqSiQW5yXP2cB8KkN0tPkDSbY56y0v3MY///Q0kdE4W9evqck1r9jQy0mrkRhsZrsxi8Ynk0xMzTE8FGA4HCB29jxnpucaXayqRGNxBvq7ufGaAXydbTXpJ4mOxgn5fewaCliNxAAWSIxZlB3dNBwOMjwUBGj6WkkkFmd4KEhnexs7Ngdq0lwXicUZDgcIBXo4NTlnc0mMBRJjsiKjcTrbhe2b/ewcCtAmEIk178itxOwCR0/PsHurExRHwkEOHk+wkM5UfM1zM/M8e2aWkfBaQv5u5tMZzs7O16rIpklZIDHGFTkWZ/smP90d7azp6uDqwf6m7nDP1qaytavhcJC5VIYnx6cqvmYke81wgFDAB8C49ZOsehZIjMGZN3LgeIKRcHBx2+6tQaKxeE2GzDZCtj9kVzgAsHhv1QTHaCyOCOzcEmDQ7wQSm0tiLJAYAxw5Nc3sfJrhnEAyPBQkcX6BZ8/MNrBklYuOxrlyoBe/rxOAobU9rO/tqqrDPRqLc9XGPvp9nRdqJBZIVj0LJMZwIeXHRYGkBt/gGyU7lDn3fkSE4XCw4vtZvKbbVDbQ102bYEOAjQUSY8DpVPf7Orh8fe/itqsH+1nT1d6UgeR4/Dynp+fZnRNIwGneOjIxzVRyoexrxs6e59zswmJw6mhvY6C/22okxgKJMeA02QyHg7S1XcgL2t4m7NhSmyGzyy1b5uG8QDIcDqIKB46XPxot29Ge248U8vssTYqxQGLM+fk0T56cWmyyyTUSDvL4iUnmU5UPmW2EaCxOV0cb14b8F20fHnI63isJjpFjcbo72rgm1L+4bdDvs852Y4HEmIMnEqQzetE37ayRcJD5dIYnxicbULLKRWMJrt/sp6vj4v/iwTVdXL6ht6IO9+honJ1bAnS2X7hmKOCz4b/GAokx0SLNQLnbmql5K5XOcOB4omANC5xaSbTMiZYL6QwHjycu+Tca9PuYTKZaZu0WUxkLJGbV2xeLsyXYw0B/9yX7Ngd8DPR3N1UgOXxymvML6cUZ7flGwkHGJ5Nl1SSeHJ9iLpW5JJBssiHABgskxhCNxQs2a4E7ZHYoWNNFoeotf0Z7vkpqWdlj80eBhdxJiWOJ6tPTm+ZlgcSsaqen5xg9d55hd/Z3ISPhAE9NzJA4X/6Q2UaIHIsTXNPJZUXSxV+3yU9nu5QdSNb1djG0tuei7YMBm91uLJCYVS5b0xgJry16THbfgdHmSOAYHXUmDTqLiF7K19nO9k3+smpZ0Vic4aHAJdfM1kjGE82dbt9UxwKJWdWisThtAju2+Ises3NxyOy55SpWxWbmUhw+OVVw4ECu4XCQA8ed0WpLmUoucGRiumCw7e3uoL+7w2okq5wFErOq7YvF3RnsxRcLDfR0csVAb1OklD9wPEFGnea4UoaHgkzPpXhqYnrpa44mUKVo89+gDQFe9SyQmFVLVYnG4kVHN+UacXNUrfRMwItDmYt0tGeNbPXe4V5oRnuuTQEfY1YjWdUskJhV65kzs0wmU0t+6ILzIXp6eo4TK/ybdyQWJ7yuh/V9lw5lznX5+l76fR2e+kmisTjb1q8huKar4P5Bv88SN65yFkjMqpXt8xjxWCMBVvwwYGcoc/GBA1ltbbJYy1pKfhbhfCG/j4npOU/9LaY1WSAxq1Y0lmBNVztXbexf8thrQ3662ttWdCA5NZnkRCK5mE9rKcNDQZ4YnyK5UHxW+ngiycnJuaLNWuD0kaQzyulpG7m1WlkgMavWvlicHVsCtLcVHiabq6ujje2b/exbwYEkEivdl5FvOBwknVEeO1F8EEGxLMK5LgwBtuat1coCiVmV5lJpDp2YvGSmdikj4SAHRhOk0iszE3B0NL6Y+t6L7CisfceKB8dILE5nu7B9U/Hh0dk0KZZOfvWyQGJWpSfGpphPX5o7qpSRcJDzC2l+emrpIbONEInFuTbUj6+z3dPxG/t9bAn2EC0x0TIai3PdJn/Ja9ra7cYCiVmVvDTZ5BtewR3umYyyP3Zpdt6lDIcDRSdapjPK/tH4kqPa1vd20dkulrhxFbNAYlalaCzOQH83m91mGS+2rV9DoKdzMSniSnL09AxTcynP/SNZI+EgsbPnOVOgo/ypiWlm5tNLXrOtTdjYb0OAVzMLJGZViiyRj6oQEWE4HCzZp9Ao5Xa0Z2VrG/sLNG+VU2sb9Nva7atZXQOJiNwiIk+KyBEReX+B/ZeJyIMisl9EHhaRIXf7y0UkkvOTFJFb3X2fF5Gnc/aN1PMeTOtJzC5wdGLG04z2fCPhIIdPTjE7n6pDySoXjcXp6+7gyoG+ss7bORSgTSg4Gi0Si9Pv6+CKDb1LXicU8FkgWcXqFkhEpB34FPBqYDtwh4hszzvsE8DdqroLuBP4KICqPqSqI6o6AtwMzALfzTnvvdn9qhqp1z2Y1rT/uLc0IoWMhANkFA4eX1lL72aXwfUylDnXmq4Orh7sL9jv42T8DdLm4Zohfw/jieSKTyFj6mPJQCIivygilQScG4AjqnpUVeeBe4DX5h2zHXjQffxQgf0Arwe+o6qzFZTBmEtkPzR3epy4lysbfFZSJuDkQppDY5Nld7RnjYSDREcvziOWXEjzxPhUyXVacoUC3czOp5maW1k1NbM8vASINwE/FZGPich1ZVx7CxDLeT7qbssVBW53H98G9IvI+gKv/6W8bXe5zWGfFJGCSYVE5B0isldE9k5MTJRRbNPqIrE4Vw70EujpLPvc9X3dhNf1lL3meT09PjbJQlrL7h/JGgkHic8u8OyZC9/VDrop5r2kW4GcIcDW4b4qLRlIVPVXgN3AU8Bfi8gP3Q/ppfJKFKoP59d73wPcKCL7gBuB48DiVxoR2QTsBB7IOecDwLXA84F1wPuKlPszqrpHVfcMDAwsUVSzWqgqkQqGyeYaHvKWo2q5RI5V1tGetTisOWc02mJHu8da2+LsdusnWZU8NVmp6iTwNZzmqU04tYefiMi7Spw2CoRzng8BJ/Kue0JVX6equ4EPuttyv+q9EfiGqi7knDOmjjngr3Ga0Izx5Hj8PKenS+eOWspIOMjx+HlOTa2MD83oaJxBfzehMoYy57pqYx89ne0XjUaLxOJsDvjY6Pd2zexrW5qU1clLH8kvicg3gO8DncANqvpqYBinRlHMI8BVInK5iHThNFHdl3ftDTn9Lx8APpd3jTvIa9ZyaymIM27zVuDgUvdgTFa2SaraQAKwf4U0bzkZfyu/n472NnYOBS6qkURH456yImcNWr6tVc1LjeQNwCdVdZeqflxVTwG4nd//sdhJqpoC3onTLHUI+IqqPiYid4rIa9zDbgKeFJHDwCBwV/Z8EdmGU6P5Qd6lvygiB4ADwAbgIx7uwRjA+YDs6mjj2lDx3FFLuX6zMzpqJTRvnZuZ55kzs1U11YETHB87Mcl8KsOZ6TliZ8+XNarN19nO2jWd1rS1ShVfX/SC/w6MZZ+ISA8wqKrPqOqDxU8DVb0fuD9v24dyHt8L3Fvk3Ge4tHMeVb3ZQ5mNKShyLM71m/10dVQ+8r2nq51rQ/0rYoZ7dInVC70aCQeZT2V4YnxyMR18ucFp0O+zfFurlJf/TV8FctOdpt1txjSVVDrDgeOJiuaP5BsOB4nG4mQavJhTNJZABHZ6zPhbTG4esUgsQVsF17RJiauXl0DS4c4DAcB9XHjNTWNWsJ+emub8wtK5o7wYGQoymUzx9JmZGpSscpHYOZ4z0Ee/r/yhzLk2B3xs6OsmEksQjcW5erCf3m4vDRYXhPw+xhO2uNVq5CWQTOT0aSAirwVO169IxtRHpfmoCsl2RDcyE7CqEh1N1OR+RJyld/fFzjkd7RVcc9Dv4/T0HPOplblei6kfL4HkPwO/JyLHRCSGM2/jN+pbLGNqLxqLE+jp5LL1a6q+1pUDffR2tTe0wz129jxnZ+ar7mjPGgkHODoxQ3x2oaJrZhe4WinDos3yWbLuqqpPAS8QkT5AVHWq/sUypvYisTjD4fIy/hbT3ibOkNkGBpJIjTras3KDRyX9SIOBCwtcDa2tPlib5uGpEVREfgG4HvBl/xOq6p11LJdpQaqKKp6SANbazFyKwyeneOX1oZpdcyS8ls/+y1HGEufpaFv+FRl+/PQZujvauCa0VJIJb3a5waOns52rB8vLIgy5a7fXv58kk1FEqMmXAlO9JQOJiPwFsAZ4OfBXOEkUf1zncpkW9K4v7UMVPvXm5y77ax88niCj3lN+eLF7a5CFtPLCj36/Ztcs157L1tLZXpsgFujp5Dkb+1jf20VHBddczjQpb/38I1y2bg1/eOuOur+WWZqXGsmLVHWXiOxX1T8QkT8Bvl7vgpnWkskoPzg8Aeo8Xu5aSS072rNuvnYjn3jDMOcX0jW7ZrlecPm6ml7vU//huXRXOMcmuKaTro62us8lSS6k+eFTp5k8X7svBaY6XgJJ9q9iVkQ2A2eAy+tXJNOKnj4zw1TSycd59PQ0z9lYm+YYr6KjccLreljfVzBZdEU629t4/fOGana9laCaZjIRYVPAx1id06Rksx3b5MeVw8tXj78TkSDwceAnwDNcmtbdmJJyO6UjDchRFY3VZiKiKW3QX/+127N/S6em5kg3eEKocZQMJG5CxQdVNa6qXwMuA67NTXNijBeRWJzernb6ujuWfVGoU1NJjsfP17RZyxQW8td/dnu2mTKdUc5M2wTIlaBkIFHVDPAnOc/n8tK8G+NJNBZn11CQXUOBZV8UqhYZf4032TQp9VxyNxqLs6arHaDuzWjGGy9NW98VkdvFxtmZCiUX0jzuLgU7HA5yaGyS5DJ2UEdjcdrbhOs3W+dsvQ36fcynMsRnF5Y+uALZbMcvv2YjYAtprRReAsnv4iRpnBORSRGZEpHJOpfLtJBDOUvBjoSDpDLKYyeW708oEotzbaifHvdbrKmf7BDgetUUstmOX7XDmQ9kHe4rg5eldvtVtU1Vu1TV7z6vfDEHs+pEc4bejoSXN0dVJqNER+M1SyNiSgvlzG6vh2y245uuGaCjTWwhrRXCy4TElxXarqr/VPvimFYUiV28FGzI71u2HFVHTzvDjkdsxNayWFxyt06BJBI7x1Ub+/D7Ohlcho59442XeSTvzXnsw1kj/VHAFpgynuRnqB0JB5dtUajF2lAZy8aaym3s70akPkvuZrMd/9x1Tv/IoL/baiQrhJemrV/K+XkFsAM4Wf+imVYQn53n6dMzFycEDAd59sws52bmS5xZG9FRZ9jxlQPl544y5etsb2N9b3ddmrZGz12c7dgW0lo5KsmFMIoTTIxZUnT00qG32ceRZaiVRNxhx+0NSBS5WoUC3XXpbN/n1i6zE0uXY/Kj8cZLH8n/BLKDwtuAESBaz0KZ1hGNxS9ZCnbnUAARZ192GGc9JBfSHBqb5O0vuaJur2EuFfL3MHputubXjcbi+DovZDsO+X3MzKeZSi5UvUKkqY6XPpK9OY9TwJdU9V/rVB7TYiKx+CVLwfZ1d3DVxr66d7hfGHZs80eWUyjQzd5nz9b8upFYnB2bA4vZjnNHiFkgaSwvgeReIKmqaQARaReRNapa+68cpqWoKtFYnJuvvbTWMRIO8r3HT6KqdVtT4kLG37V1ub4pLOT3EZ9dILmQxtdZm7k7C+kMB48n+JUXXHbR64AzZ2W5k4Cai3npI3kQ6Ml53gP8Y32KY1rJ6LnznCmyFOxwOMi52QViZ8/X7fWjecOOzfIY9Nd+LsmT41PMpTIX9bUtDjW2fpKG8xJIfKo6nX3iPrZ1NM2SSq0Bku0w3VfHBI7RUcv42wjZD/hadrgX+luqR8AylfESSGZEZHFJOxF5HlC/r5GmZURj8aJLwV4T6sfX2Va3BI7ZYcc2f2T5herwAR+NxVnX28XQ2guNI77OdoJrOm0I8ArgpY/kd4CvisgJ9/km4JfrVyTTKiKxODu2BAouBdvZ3saOzYG6pZRfHHZsNZJlV48mp0gszkg4eEl/WsjvW5Y14k1pXiYkPgJcC/wm8F+A61T10XoXzDS3hXSGgycSJVO3j4SDHDwxyUI6U/PXjxxzhx3XcI12402/r5Pervaa1RSmkgscmZgu2EzpTEq0BpJGWzKQiMhvAb2qelBVDwB9IvJf6l8008wOn5wiuZApmSxxOBxkPpXhyfGpmr9+dPTSYcdm+QwGfDVr2jpwPIEqDBcYxm01kpXBSx/Jr6vq4oB/VT0H/LqXi4vILSLypIgcEZH3F9h/mYg8KCL7ReRhERlyt79cRCI5P0kRudXdd7mI/EhEfioiXxaRLm+3apbTYudoiaalbG1lX43nk2SHHVvG38ZxPuBrE0hKDdoY9Ps4MzNXl1qt8c5LIGnLXdRKRNqBJT+83eM+Bbwa2A7cISLb8w77BHC3qu4C7gQ+CqCqD6nqiKqO4CSHnAW+657zx8AnVfUq4Bzwdg/3YJZZtnM0vK6n6DFDa3tY39tV85Ty2WHHtiJi49QykERjcbatX0NwzaUfO6GAD1Vn/XbTOF4CyQPAV0TkZ0XkZuBLwD94OO8G4IiqHlXVeeAe4LV5x2zHmacC8FCB/QCvB76jqrNuQLsZZ5IkwBeAWz2UxSyzaCzB8FCg5GRDEWE4HKx5ICn1DdYsj1DAx6mpOTKZ6pfcjcYSRWuX2RFiNpeksbwEkvfhfNj/JvBb7uP3ljzDsQWI5TwfdbfligK3u49vA/pFZH3eMW/CCV4A64G4qqZKXBMAEXmHiOwVkb0TExMeimtqZXouxeFTU56aloaHghyZmGYqWbulWUsNOzbLIxTwkcoop2eqqymMJ5KMTyaLfikYtECyIngZtZVR1b9Q1der6u3A/cC7PVy70FfR/K8n7wFuFJF9wI3AcZx8Xs4FRDYBO3FqRV6vmS33Z1R1j6ruGRgY8FBcUysHRp3OUS81gpGtQVSdc2ql1LBjszwWJwtW2RGerV0W+1Kyqc4LaRlvPP1PE5ENIvKbIvJPwMPAoIfTRoFwzvMh4ETuAap6QlVfp6q7gQ+623I/Ud4IfENVs19XTwNBEcnOf7nkmqbxInnpvksZdofn1qrDPTvs2Ga0N9Zik1OVH/CRWJzOdmH7psKrewfXdNLV0Waz2xusaCARkX4R+TUR+Qfgx8BzgCtU9UpVfY+Haz8CXOWOsurCaaK6L+81NohItgwfAD6Xd407uNCshaoqTl/K691NbwG+5aEsZhllO0fX9i49oC64povLN/TWrJ8kO+zYZrQ31oVJidXN8YjG4ly3yV80+aOI1LRj31SmVI3kFM6IqLuAK1X13YDnJe3cfox34jRLHQK+oqqPicidIvIa97CbgCdF5DBOLeeu7Pkisg2nRvODvEu/D/hdETmC02fyWa9lMssjOlre0NvhoUDNlt71MuzY1N+Gvm7a26SqGkk6oxw4vnTtMmRrtzdcqRQpv4dTi/g08Lci8uVyL66q9+P0qeRu+1DO43u5MAIr/9xnKNCRrqpHcUaEmRXo5GSSsUSyrKal4XCQb0ZOMJY4z6ZA8eHCXngZdmzqr71N2NjfXdVkwacmppmeSy3Z1zYY8LF/GVbbNMUVrZGo6idV9WeA1+B0cn8T2Cwi7xORq5ergKa5LNYIymhayn5Q1KJ5y8uwY7M8Bv3VzW5fqqM9a1PAx1giidPybRrBy6ito6p6l6ruBJ4PBIDv1L1kpilFYnE62op3jhZy3SY/ne1SdYd7OcOOTf1V2+QUicXp93VwxYbekscN+n3MpzLEZ2s3hNyUp6zxkap6QFV/T1WvrFeBTHNbqnO0EF9nO9dt8lddI8kOO7ZAsjKEAtV1gkdjcYaHgrS1la5d1mqEmKmcDbQ3NZPJKPtHS2f8LWYkHOTAaIJ0FTOhraN9ZRn0+5ieSzE9l1r64DzJhTRPjE8VTNSYLxToBiyQNJIFElMz2c7RSmoEw0NBZubTHDk1vfTBRURjcS7zOOzY1N+mKtYlOXjc+VLhZdDGhcmPFkgaxQKJqZlqclxlO+erad6KjsYtv9YKUs1SuOX8LW3sr/3SvqY8RYf/isgBiqQfAXAz9hqzKDoap7976c7RQi5f30u/r4PIaJw3Pj+89Al5Khl2bOqrmpUSo6MJNgd8bHSDUSldHW1s6Ou22e0NVGoeyS+6v3/L/f1/3N9vxknrbsxFIrE4u8KBJTtHC2lrE4aHgkSOVVYj8TpU1CyfajrBI7FzZb2XoUC39ZE0UKl5JM+q6rPAi1X1v7kjtg6o6vuBVy1fEU0zSC6keWJsqqqmpZFwkCdPTnF+Pl32uVF32PH1m70POzb11dPVjt/XUXaN5Mz0HLGz58v6W7I0KY3lpY+kV0Rekn0iIi8Cym+7MC3tsRMJUh47R4sZDgdJZ5SDJ8rPBBypYNixqb9NgZ6yawrZdDnl1EiqnfxoquMlkLwd+JSIPCMiTwP/C/iP9S2WaTaRmPPhX02NJDvUs9wO9+ywYy9DRc3yqmTt9kgsQZvAzi3e38+Q38e52QWSC+XXZk31SvWRAKCqjwLDIuIHJC/NuzGA8+HvtXO0mI39PrYEexb7O7w6ejqbk2ltxa9t6iPk7+aJscmyzonG4lw92E9v95IfTxdeJ3BhhNhl663BZLktWSMRkUER+SzwZVVNiMh2EbF10s1FIrHyMv4WMxwOlB1I9h3LDhW1GslKE/L7mJieYyGd8XS8qjrZo8tsIq1mhJipnpemrc/jpILf7D4/DPxOvQpkms/ZmXmOnZ2tyRyOkXCQ0XPnOT3tPWvshWHHfVW/vqmtwYAPVZiY8vZ+PntmlvjsQtnryVialMbyEkg2qOpXgAwsrjNiDZFmUbSGQ2+z30TL6SepZtixqa9yP+DLWV0z12Cg8smPpnpeAsmMiKzHnZwoIi8ArJ/ELIrE4mV3jhazY0uANvEeSLLDjm0i4sq02HfhsckpEovT09nO1YPl1S77uztY09Vus9sbxEtv1u/iLJF7pYj8KzAAvKGupTJNJTpafudoMb3dHVw92E9k1Nt3lcdOTJLKqKVGWaHKrZFER+Ps3BKgo7287E3ZJXetRtIYXt6tx4AbgRcBvwFcDzxRz0KZ5qGqi+m+a2UkHCQai3taqKia/F6m/tb1dtHV3uYpkMynMjx2YrLiYdzVpq03lfPyFfKHqvpcnIACgIj8BHhu3UrVwr5zYIz7oicaXYyaWUhnODe7UNPUJMPhIPc8EuM/fWEvXR2lv+s8dmKSTVUOOzb1IyJs9Hfz7egYx86Uzqw0O59mPpWp+G8p5Pfxo6fPVnQuwFf3xljf18XN1w5WfI1C1xzo7+amazbW7JorUamkjSGcNdN7RGQ3znK7AH5gzTKUrSV99l+e5tDYJFvWts6a4sNDAW66ZqBm17vpmgGGw0Fi55ZO6ebrbOOXK0jyaJbPbbu38MBj4zw1sfQSAc/dGuTFV26o6HWykx8zGS174IWq8pG/P8S2Db01CySZjPKH336c4XBw9QYSnHxabwWGgD/N2T4F/F4dy9TSxhJJXnl9iE/+8kiji7JibQr08K3fenGji2Fq5N2vvIZ3v/Kaur9OyO8jlVFOz8wtppb36tkzsyTOL3DoxCRzqTTdHdWn2nnmzAyTydSq6LcpGkhU9QvAF0TkdlX92jKWqWVPWac8AAAfnklEQVRlMsqpqeTiSBZjTO1cWOCq/ECS7WubT2c4VGXy0fxrroZ+Gy8pUr4mIr+A08nuy9l+Zz0L1orOzs6zkNbFkSzGmNpZXJFxMslOyuuwj7jZo1MZZ/BILQJJdgj7ZDLF+fk0PV2tm1DUS4qUvwB+GXgXTj/JG4DL6lyulpT9ZjJogcSYmgsFKp/dHh2N89ytaxno765qlc5cuUPYW33GvZfhvy9S1V8DzqnqHwAvBKx3swLZtlJr2jKm9jb0ddPeJmWv3Z4ddjyyNchIOFh2rrdC5lJpDp2YXJyk2+rNW14CyXn396yIbAYWgMvrV6TWlZ11a01bxtRee5sw0Ndd9uz2J8YnnWHHQ04gOXp6hsTsQlVlOTQ2xXw6wy07QgCMT55f4ozm5iWQfFtEgsDHgZ8AzwD31LNQrerkZJI2gQ19XY0uijEtqZL1Ty7kigtcyPU2Wl2tJHLsHACvut4ZSjye8J6EtBl56Wz/Q/fh10Tk24DP1iSpzHgiycZ+X9npH4wx3oT83Tw1MVPWOfticTb0dbMl2IO/pxNwgsvLrq58blR0NMHG/m6uHOijr7uj5YcAl5qQ+LoS+1DVry91cRG5Bfj/gXbgr1T1j/L2XwZ8Did/11ngV1R11N23FfgrnP4YBX5eVZ8Rkc/jpGzJBrO3qmpkqbKsBOOTycUspcaY2tsU6OHfjpwp6xxnlFYAEcHv6+TKgd6qayTZkV8iwqC/u+X7SErVSH7J/b0RJ8/W993nLwceBkoGEhFpBz4FvAIYBR4RkftU9fGcwz4B3K2qXxCRm4GPAr/q7rsbuEtVvycifbhp7F3vVdV7l7q5lebkZJLLN9jqbcbUy6Dfx9Rcipm5lKckopPJBZ6amOG23VsWt42E1/KDw6dQVUTKX5ogMbvA0dMz3P68IaCydeubTdE2FlV9m6q+Dac2sF1Vb1fV23Hmk3hxA3BEVY+q6jxOv8pr847ZDjzoPn4ou19EtgMdqvo9tyzTqrp0vowVbiyRtI52Y+ooFOgGvA+33R9zGjZy83uNhAOcnp7neLyyDvJsbSY7F2VwFWQl9tJYv01Vx3KenwSu9nDeFiCW83zU3ZYrCtzuPr4N6HfXPrkaiIvI10Vkn4h83K3hZN0lIvtF5JMi0l3oxUXkHSKyV0T2TkxMeChufc3Op5hKpqxpy5g6ys7R8tqUlP3Q35WTvTobVKKxyrqCo7E4IrBzyBn6Gwp0c2pqjnRm6WzWzcpLIHlYRB4QkbeKyFuAv8epPSylUJ0w/1/yPcCNIrIPp9/jOJDCaXJ7qbv/+cAVOHm/AD4AXOtuXwe8r9CLq+pnVHWPqu4ZGKhdQsFKjdvQX2PqLlRmINl3LM4VA70E3E52gGtDfro62ojEzlVUhkgszpUDffh9nYtlSme0rOWjm82SgURV3wn8b2AYGAE+o6rv8nDtUS6euDgEXJQ/XVVPqOrrVHU38EF3W8I9d5/bLJYCvombtl5Vx9QxB/w1ThPaijdukxGNqbtyZrerKpFYnJG8tXS6Otq4frO/ohqJqhIdvTjFSrm1pGbkaRyqqn5dVf+r+/MNj9d+BLhKRC4XkS7gTTgrLS4SkQ0iki3DB3BGcGXPXSsi2arEzcDj7jmb3N8C3Aoc9Fiehlqc1W41EmPqZk1XB36ft+G2Y4kkp6fnCq5/MjwU5MDxBKl0psCZxR2Pn+f09PxF19wUcJaMaOUO96KBRET+xf09JSKTOT9TIjK51IXdmsQ7gQeAQ8BXVPUxEblTRF7jHnYT8KSIHAYGgbvcc9M4zVoPisgBnGayv3TP+aK77QCwAfhI2XfdANkJSVYjMaa+vK6UWGp1zd1bg5xfSHP45NJrqBS8Zk4tZ9AdANDKHe6l0si/xP3dX+nFVfV+4P68bR/KeXwvUHAYrztia1eB7TdXWp5GGk+cp9/XwZqu6tc1N8YUN+j3efr2H43F6Wpv49pNl37E5c5w377Z7/m1o7E4XR0XX3NDbzcdbVJ26pZmUqpGsq7Uz3IWshWMT9rQX2OWQ8jvvUZy3WZ/wUWsLlu/huCaTiLHypuYGInF2bHZT2dO9oq2NmFjf3fZySSbSamvx4/ijLIqNvrqirqUqEWNT85Zs5YxyyAU8HF6eo5UOlM0HVE6oxw4nuCNewonMhcRhoeCZc1wT6UzHDie4I4btl6ybzDgrZbUrEpNSLxcVa9wf+f/WBAp00mbjGjMsggFfGQUJkoMt/3pqSlm59MMh4svgDUcDnL45BQzcylPr3v45DTJhUzBPpdNqzWQ5BKRtSJyg4i8LPtT74K1klQ6w8S01UiMWQ5e5pJEFzva1xY9Znc4SEbhwHFvw4BLdd4P+n0t3bTlZYXE/wT8E87oqz9wf3+4vsVqLaen50ln1FZGNGYZeJm3EYnF8fs62LZ+TdFjdrkz072umBiNxVm7ppOt6y69ZsjvY2Y+zVSyunVOViovNZLfxplF/qyqvhzYDTQ+50gTGbc5JMYsGy+TEiOxBMNudt5i1vd1E17X47mfJDoaL3rNxTK1aK3ESyBJqmoSQES6VfUJ4Jr6Fqu1LKZHsaYtY+pu3ZouOtulaCCZnU9x+OQUuws0QeUbCa/1NHJrZs655vBQ4Wsu1pJatJ/ESyAZdVdI/CbwPRH5FnmpTkxptla7McunrU1K9kkcPD5JOqMFZ7TnGx4KcCKR5NQSAeDA8QQZLdw/AuXnAGs2XlZIvM19+GEReQgIAP9Q11K1mPHJJJ3twro1tsSuMcshVGJS4oWldb3USJxjIrE4r7w+VPS4yBLXzH6JbNXZ7aUmJP69iLxZRBZXYlLVH6jqfe76Isajk+4Su21t5S+SY4wp32CJNCmRWJyhtT1s6Cu4AsVFdmwJ0N4mS/aTRGNxtq5bw7rewl8WfZ3tBNd0rsqmrc8Avwg8IyJfFpFb3eSLpkxjiaQ1axmzjLI1EtVL1wCJxOKeaiPgBIBrQ/1LZgLOLq27ZJkSrZlKvtSExG+p6h3AVpxldd8CHBORz4nIK5argK3gpKVHMWZZhfw+kgsZJs9fPJlwYmqO4/Hznjras0bCQaKxOJkiC1OdmkxyIpFcMjg5OcAqW3VxpfOyHsl5Vf2y21fySpzhv9ZH4pGqOnm2rEZizLIZLDIEuJz+kazhcJCpuRRHT88U3H9hImLxWfKwSmskWSIyKCLvEpF/xRm59V3geXUvWYuYmksxO5+2Gokxy2hTsUAyGqe9TdixufSHfq6RxaV3C/eTREfjdLQJ1y9xzVDAx5mZORbKXOOkGZTqbP91Efk+8BOcNdT/m5t7632qGlm2Eja57BBEW6vdmOVzYbjtxU1JkVicawb76em6NONvMVcO9NHX3bFY88gXicW5dlM/vs7S1wwFfKjCqanWq5WUqpG8CPgjIKyq71LVf12mMrWUMVur3Zhlt9HvjMjKbUrKZJRoGR3tWe1tws4tgYIjtzIZZX8sUXQiYq5WnktSqrP9bar6XVVdrIeJyIeXpVQtxNKjGLP8ujvaWdfbdVHT1jNnZphMppbsyyhkOBzk0NgkyYX0RduPnp5mai615IgtaO212z1l/83xmqUPMbmyTVvZb0jGmOUx6PddNAEwW6MolfG3mJFwkIW08vjYxauMR9xhwV4CiZccYM2q3EBiM+rKND6ZZF1v15Ltp8aY2tqUNykxcixOb1c7z9nYV/a1inW4R2Nx+ro7uGJg6WuuXdNJV0dbS85uLzeQ2GitMp2cTFr6eGMaIL9GEhlNsHPImalerlDAR8jvu6TDPRKLs8vjNUXE8zLAzcbL8N+PiYhfRDpxkjaeFpFfWYaytQRnrXZr1jJmuYX8Ps7MzDOXSjOXSnPoxGTZHe25hsOBi2okyYU0h8bKu2apHGDNzEuN5JWqOomTLmUUZyjwe+taqhYybulRjGmIUMD5Andqco5DY1PMpzOMeBhdVcxwOMgzZ2aJzzqpBh8fmySVUU8jtrIGA75V27TV6f7+eeBLqnq2juVpKfOpDKen561py5gGyF0DZHFp3a2VB5LcTMDA4jolu8u4ZsjfzViicA6wZuYlkPydiDwB7AEeFJEBoPVCah2cmrKhv8Y0yqZAD+C0CkRicTb2d1f1f3HnlgAiLCZwjI7GCfl9ZX1RDAV6mE9liM+21pK7XnJtvR94IbBHVReAGeC19S5YK7AFrYxpnGzQOOnWSJZaWncp/b5OnjPQRyR2DnBqJl6G/RYqU6v1k3jpbH8DkFLVtIj8PvA3wOa6l6wFZGfVWiAxZvn5ezrwdbbx5PgUR0/PlP2hX8hIOEh0NMHZmXmePTNbdud9tt9m1QUS4P9V1SkReQnwKuALwKfrW6zWMObm+bGmLWOWX3a47YNPnAK8TRpcynA4yNmZef7+wJj7vLxZ8tlmsGLLADcrL4EkmxPgF4BPq+q3AFvgyoOTk0m6O9oI9HQufbAxpuYG/T7OzswjAjuHyk+Nki8bjO7+t2cQgV1ljgLb2O8EkrFVGEiOi8j/Bt4I3C8i3R7PQ0RuEZEnReSIiLy/wP7LRORBEdkvIg+LyFDOvq0i8l0ROSQij4vINnf75SLyIxH5qbty44oNauOTc4QCvqraZY0xlcs2K1850IffV/0XumtC/XR3tPHTU9NctdHJClyOro42NvR1t9wQYC8B4Y3AA8AtqhoH1uFhHomItAOfAl4NbAfuEJHteYd9ArhbVXcBdwIfzdl3N/BxVb0OuAE45W7/Y+CTqnoVcA54u4d7aIiTCVsZ0ZhGygaScuZ6lNLZ3saOLYGqrhkKdK++PhJVnQWeAl4lIu8ENqrqdz1c+wbgiKoeVdV54B4uHe21HXjQffxQdr8bcDpU9XtuGaZVdVacr/Y3A/e653wBuNVDWSry9OkZfnLsXMXn28qIxjRW9otcNfNH8mWbtyq9ZiumSfEyauu3gS8CG92fvxGRd3m49hYglvN81N2WKwrc7j6+DegXkfU4s+fjIvJ1EdknIh93azjrgbiqpkpcM1vud4jIXhHZOzEx4aG4l/rQtw7y+984WNG5i0vsWo3EmIa5cqAPEbhh27qaXfNnLl+HCDy/wmvm5wBrBV6att4O/IyqfkhVPwS8APh1D+cV6hjIn875HuBGEdkH3AgcB1JAB/BSd//zgSuAt3q8prNR9TOqukdV9wwMDHgo7qWGh4I8eXKK8/PppQ/Oc252gflUxma1G9NAL71qAw+/5yauCfXX7Jqv2D7Iw++5iasHK7tmyO/j3OzCJWubNDMvgUS4MHIL97GX3uNRIJzzfAg4kXuAqp5Q1dep6m7gg+62hHvuPrdZLIWzVvxzgdNAUEQ6il2zlkbCQdIZ5eCJRNnnZquu1rRlTOOICJet711R18x+JrRSrcRLIPlr4Eci8mF3hcR/Bz7r4bxHgKvcUVZdwJuA+3IPEJENIpItwweAz+Wcu9ZNxwJOv8jj6iSoeQh4vbv9LcC3PJSlIrvcMeL5axB4kf0jsRqJMSbX4gJXLdRP4qWz/U+BtwFncUZJvU1V/8zDeSngnTgjvg4BX1HVx0TkThHJrrR4E/CkiBwGBoG73HPTOM1aD4rIAZwa0F+657wP+F0ROYLTZ+IlqFVkY7+PLcEe9lUQSLKjMjZZjcQYk6MV06SUHATt1hb2q+oO4CflXlxV7wfuz9v2oZzH93JhBFb+ud8DdhXYfhRnRNiyGAkHK6qRjCeSiMBAv61FYoy5YHC1NW2pagaIisjWZSrPijMcDjB67jynp+fKOm88kWRDXzed7eUuQmmMaWX93R2s6WpfzMXXCrxMy9wEPCYiP8bJ/AuAqr6m+CmtIzvpKBqL87PXDXo+z4b+GmMKyeYAa6UaiZdA8gd1L8UKtnMoQJuUH0hOTiYZWrumjiUzxjSrUMC3mNS1FRQNJCLyHGBQVX+Qt/1lOPM9VoU1XR1cPdhfdof7+GSSPdvW1qlUxphmFvL7+NHTrbPYbKkG/D8Dpgpsn3X3rRq7tzod7l6Xx0wupInPLiyu0GaMMbmya7dnMq2x5G6pQLJNVffnb1TVvcC2upVoBRoeCjKZTPHMmVlPx9scEmNMKSG/j1RGOTMz3+ii1ESpQFLqU3BVfdXOroKWXWJzKdm1Bqyz3RhTyKC/tYYAlwokj4jIJTm1ROTtwKP1K9LKc/VgP2u62onGvKVKubBWu80hMcZcKjtRuVUWuCo1aut3gG+IyJu5EDj24KyOeFu9C7aStLcJO7YEiHjscM+mPrCmLWNMIYtpUlqkRlI0kKjqSeBFIvJyYIe7+e9V9fvLUrIVZiQc5PP/+gxzqTTdHe0ljx2fTNLX3UF/DVZkM8a0ng193bS3Scus3b7kPBJVfQgnUeKqNhIOMp/O8MTY1GKfSTEnJ5MM+q1ZyxhTWHubMNDXOislWv4Ojy50uC/dvDWWsJURjTGlZYcAtwILJB5tDvgY6O/2lMDxZCJp/SPGmJI2tdCSuxZIPBIRhoeCREZLB5JMRjk1NWdDf40xJYUCFkhWpZFwgKMTMyRmF4oec3pmjlRGrWnLGFPSoN/H1FyKmblUo4tSNQskZRgJO7mz9h8vXis56aaGthqJMaaU7DyzVuhwt0BShp1DSy+9Oz5pa7UbY5a2OLu9BZq3LJCUIdDTyRUDvSVHbo27qaGtRmKMKSWb1NVqJKvQSDhIJJYomgl4fDJJe5uwvs/mkRhjimultdstkJRpJBzk9PQcx+OFF6UZT8yxsd+ZtWqMMcX0dLXj93W0xMgtCyRlGglnl94tnMDx5KRNRjTGeNMqQ4AtkJTp2pCfrvY2okXmk9ha7cYYrwZbZO12CyRl6upoY/tmP5FjRQKJzWo3xni0KeCzPpLVaiQc5MDxBKl05qLt03MppudS1rRljPEk5PcxMTV3yWdJs7FAUoGRcJDzC2kOn5y+aPu4rYxojCnDYMBHRmFieq7RRamKBZIKZDMB5/eT2FrtxphyLA4BbvIOdwskFdi2fg2Bns5LZrhn/xg2WdOWMcaDVlm73QJJBUSE4XDwkhnulh7FGFOO7JdOq5GUICK3iMiTInJERN5fYP9lIvKgiOwXkYdFZChnX1pEIu7PfTnbPy8iT+fsG6nnPRQzEg5y+OTURZk7xxNJAj2d+DpLL8VrjDEA63q76GpvY3zS+kgKEpF24FPAq4HtwB0isj3vsE8Ad6vqLuBO4KM5+86r6oj785q8896bsy9Sr3soZSQcIKNw8PiFiYk2h8QYUw4RYaO/25q2SrgBOKKqR1V1HrgHeG3eMduBB93HDxXYv2IND1269O7JySSD1qxljClDyO9jLFE45VKzqGcg2QLEcp6PuttyRYHb3ce3Af0ist597hORvSLy7yJya955d7nNYZ8UkYZkR1zf1014Xc9FI7fGE0lCfkvWaIzxzlm73Zq2iimUtTA/Ze57gBtFZB9wI3AcyHY6bFXVPcB/AP5MRK50t38AuBZ4PrAOeF/BFxd5hxuI9k5MTFR3J0UMDwUXZ7gvpDNMTM8RclNDG2OMFyF37fZiGcWbQT0DySgQznk+BJzIPUBVT6jq61R1N/BBd1siu8/9fRR4GNjtPh9Txxzw1zhNaJdQ1c+o6h5V3TMwMFDTG8saCQc5kUhyajLJxNQcqjYZ0RhTnk0BH+cX0kwmm3fJ3XoGkkeAq0TkchHpAt4E3Jd7gIhsEJFsGT4AfM7dvjbbZCUiG4AXA4+7zze5vwW4FThYx3soaTET8GgiZ+ivNW0ZY7xrhbkkdQskqpoC3gk8ABwCvqKqj4nInSKSHYV1E/CkiBwGBoG73O3XAXtFJIrTCf9Hqvq4u++LInIAOABsAD5Sr3tYyvWbA7S3CZHYucXlMm1WuzGmHNl5Z2NNPJeko54XV9X7gfvztn0o5/G9wL0Fzvs3YGeRa95c42JWrKernWtD/URjCTa4KyJa05YxphyhFli73Wa2V2k4HCQ6Gmc8kaSrvY11vV2NLpIxpolsdEd6NnM6eQskVRoZCjKVTPFvT51hMNCN03VjjDHedHe0s763ywLJajay1elwP3A8Yc1axpiKDPp91rS1ml050Edvl5NbyzrajTGVCDX5SokWSKrU3ibsctOlWI3EGFOJQXdSYrOyQFID2YWuLH28MaYSIb+PMzPzzKXSjS5KReo6/He1GAkHAGvaMsZUJrsuyav/7J9pb6vtgJ3PvuX5bF2/pqbXzGeBpAZuvHojv/7Sy3nZ1fVJxWKMaW03XjPAbbu31KVG0tVR/4YnaeZEYV7t2bNH9+7d2+hiGGNMUxGRR93kuSVZH4kxxpiqWCAxxhhTFQskxhhjqmKBxBhjTFUskBhjjKmKBRJjjDFVsUBijDGmKhZIjDHGVGVVTEgUkQng2bzNG4DTDShOvbTa/UDr3ZPdz8rXavdU7f1cpqpLpuxYFYGkEBHZ62XGZrNotfuB1rsnu5+Vr9Xuabnux5q2jDHGVMUCiTHGmKqs5kDymUYXoMZa7X6g9e7J7mfla7V7Wpb7WbV9JMYYY2pjNddIjDHG1MCqCyQicouIPCkiR0Tk/Y0uTy2IyDMickBEIiLSdAuviMjnROSUiBzM2bZORL4nIj91f69tZBnLVeSePiwix933KSIiP9/IMpZDRMIi8pCIHBKRx0Tkt93tTfk+lbifZn6PfCLyYxGJuvf0B+72y0XkR+579GUR6ar5a6+mpi0RaQcOA68ARoFHgDtU9fGGFqxKIvIMsEdVm3L8u4i8DJgG7lbVHe62jwFnVfWP3IC/VlXf18hylqPIPX0YmFbVTzSybJUQkU3AJlX9iYj0A48CtwJvpQnfpxL380aa9z0SoFdVp0WkE/gX4LeB3wW+rqr3iMhfAFFV/XQtX3u11UhuAI6o6lFVnQfuAV7b4DKteqr6T8DZvM2vBb7gPv4Czn/yplHknpqWqo6p6k/cx1PAIWALTfo+lbifpqWOafdpp/ujwM3Ave72urxHqy2QbAFiOc9HafI/HpcC3xWRR0XkHY0uTI0MquoYOP/pgY0NLk+tvFNE9rtNX03RDJRPRLYBu4Ef0QLvU979QBO/RyLSLiIR4BTwPeApIK6qKfeQunzmrbZAIgW2tULb3otV9bnAq4HfcptVzMrzaeBKYAQYA/6kscUpn4j0AV8DfkdVJxtdnmoVuJ+mfo9UNa2qI8AQTgvMdYUOq/XrrrZAMgqEc54PAScaVJaaUdUT7u9TwDdw/oCa3Um3HTvbnn2qweWpmqqedP+jZ4C/pMneJ7fd/WvAF1X16+7mpn2fCt1Ps79HWaoaBx4GXgAERaTD3VWXz7zVFkgeAa5yRzF0AW8C7mtwmaoiIr1uZyEi0gu8EjhY+qymcB/wFvfxW4BvNbAsNZH9wHXdRhO9T25H7meBQ6r6pzm7mvJ9KnY/Tf4eDYhI0H3cA/wcTt/PQ8Dr3cPq8h6tqlFbAO5wvj8D2oHPqepdDS5SVUTkCpxaCEAH8LfNdk8i8iXgJpxMpSeB/w58E/gKsBU4BrxBVZum87rIPd2E02SiwDPAb2T7F1Y6EXkJ8M/AASDjbv49nH6FpnufStzPHTTve7QLpzO9HaeS8BVVvdP9jLgHWAfsA35FVedq+tqrLZAYY4yprdXWtGWMMabGLJAYY4ypigUSY4wxVbFAYowxpioWSIwxxlTFAolpCSLysIi8Km/b74jI/1rivOlS+2tQrgE38+o+EXlp3r6HRWSP+3ibm531VQWu8XE3m+vHKyzDTSLy7ZznHxGRB0Sk2y3D3px9e0Tk4ZzzVER+KWf/t0XkpkrKYVqXBRLTKr6EM8E015vc7Y30s8ATqrpbVf+50AEiMgQ8ALxbVR8ocMhvAM9V1fd6ecGcWcyF9n0QeDFwa85cgo0i8uoip4wCH/Tyumb1skBiWsW9wC+KSDcsJuLbDPyLiPSJyIMi8hNx1m25JONzgW/tfy4ib3UfP09EfuAmxXwgb/Zz9vjL3NfY7/7eKiIjwMeAnxdnbYueAuUOAd8Ffl9VL8myICL3Ab3Aj0Tklwu9jnvc50XkT0XkIeCPC/0Dici7gZ8HfklVz+fs+jjw+4XOAaJAQkReUWS/MRZITGtQ1TPAj4Fb3E1vAr6szozbJHCbm9jy5cCfuCkyluTmY/qfwOtV9XnA54BCmQP+HGftkV3AF4H/oaoR4ENuOUbyPryz7gb+XFW/WuS+XgOcd8//cqHXyTn8auDnVPXdBS71YuA/A6/OSTWe9UNgTkReXqgMwEcoHmiMsUBiWkpu81Zus5YA/5+I7Af+ESeN9qDHa14D7AC+56bn/n2cxHf5Xgj8rfv4/wAv8Xj9fwR+VUTWeDy+1Ot8VVXTRc47gvPv8Moi+4sGi2yTXH4fjzFZFkhMK/km8LMi8lygJ7twEfBmYAB4npti+yTgyzs3xcX/H7L7BXjMrRGMqOpOVS32YZzLa+6hj+Hkq/pqqb4Nj68zU+K4kzjNWp8sVPNQ1e/j3PMLipx/F9ZXYoqwQGJahttk8zBO81NuJ3sAOKWqC+6H6GUFTn8W2O6OZArgdJIDPAkMiMgLwWnqEpHrC5z/b1yoDb0ZZ5lTr/4rMAl81kOTW8Wvo6qHgdcBf+P23+S7C/hvRc79LrAWGPb6emb1sEBiWs2XcD7s7snZ9kVgjzvM9c3AE/knqWoMJ4vtfvf4fe72eZwU3H8sIlEgAryowOv+P8Db3OazX8VZK9sTtx/nLcAmnBpKKRW/jvtajwBvA+4TkSvz9t0PTJQ4/S4KN+uZVc6y/xpjjKmK1UiMMcZUxQKJMcaYqlggMcYYUxULJMYYY6pigcQYY0xVLJAYY4ypigUSY4wxVbFAYowxpir/F19pSpGIa+FNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the results\n",
    "plt.plot(k_range, grid_mean_scores)\n",
    "plt.xlabel('Value of K for KNN')\n",
    "plt.ylabel('Cross-Validated Accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.98\n",
      "{'n_neighbors': 13}\n",
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=13, p=2,\n",
      "           weights='uniform')\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(grid.best_score_)\n",
    "print(grid.best_params_)\n",
    "print(grid.best_estimator_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching multiple parameters simultaneously"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Example:** tuning `max_depth` and `min_samples_leaf` for a `DecisionTreeClassifier`\n",
    "- Could tune parameters **independently**: change `max_depth` while leaving `min_samples_leaf` at its default value, and vice versa\n",
    "- But, best performance might be achieved when **neither parameter** is at its default value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the parameter values that should be searched\n",
    "k_range = list(range(1, 31))\n",
    "weight_options = ['uniform', 'distance']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], 'weights': ['uniform', 'distance']}\n"
     ]
    }
   ],
   "source": [
    "# create a parameter grid: map the parameter names to the values that should be searched\n",
    "param_grid = dict(n_neighbors=k_range, weights=weight_options)\n",
    "print(param_grid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, error_score='raise',\n",
       "       estimator=KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=30, p=2,\n",
       "           weights='uniform'),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], 'weights': ['uniform', 'distance']},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "       scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate and fit the grid\n",
    "grid = GridSearchCV(knn, param_grid, cv=10, scoring='accuracy', return_train_score=False)\n",
    "grid.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.053333</td>\n",
       "      <td>{'n_neighbors': 1, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.053333</td>\n",
       "      <td>{'n_neighbors': 1, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.052068</td>\n",
       "      <td>{'n_neighbors': 2, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.053333</td>\n",
       "      <td>{'n_neighbors': 2, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 3, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 3, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 4, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 4, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 5, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 5, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 6, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 6, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 7, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 7, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 8, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 8, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 9, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 9, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 10, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 10, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 11, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 11, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 12, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 12, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 13, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 13, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 14, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 14, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 15, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 15, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 16, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 16, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 17, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 17, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 18, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 18, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 19, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 19, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 20, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 20, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 21, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 21, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 22, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 22, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 23, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 23, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 24, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 24, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 25, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 25, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>0.960000</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'n_neighbors': 26, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 26, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'n_neighbors': 27, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'n_neighbors': 27, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'n_neighbors': 28, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 28, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'n_neighbors': 29, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'n_neighbors': 29, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'n_neighbors': 30, 'weights': 'uniform'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'n_neighbors': 30, 'weights': 'distance'}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score  std_test_score  \\\n",
       "0          0.960000        0.053333   \n",
       "1          0.960000        0.053333   \n",
       "2          0.953333        0.052068   \n",
       "3          0.960000        0.053333   \n",
       "4          0.966667        0.044721   \n",
       "5          0.966667        0.044721   \n",
       "6          0.966667        0.044721   \n",
       "7          0.966667        0.044721   \n",
       "8          0.966667        0.044721   \n",
       "9          0.966667        0.044721   \n",
       "10         0.966667        0.044721   \n",
       "11         0.966667        0.044721   \n",
       "12         0.966667        0.044721   \n",
       "13         0.966667        0.044721   \n",
       "14         0.966667        0.044721   \n",
       "15         0.966667        0.044721   \n",
       "16         0.973333        0.032660   \n",
       "17         0.973333        0.032660   \n",
       "18         0.966667        0.044721   \n",
       "19         0.973333        0.032660   \n",
       "20         0.966667        0.044721   \n",
       "21         0.973333        0.032660   \n",
       "22         0.973333        0.032660   \n",
       "23         0.973333        0.044222   \n",
       "24         0.980000        0.030551   \n",
       "25         0.973333        0.032660   \n",
       "26         0.973333        0.044222   \n",
       "27         0.973333        0.032660   \n",
       "28         0.973333        0.032660   \n",
       "29         0.980000        0.030551   \n",
       "30         0.973333        0.032660   \n",
       "31         0.973333        0.032660   \n",
       "32         0.973333        0.032660   \n",
       "33         0.980000        0.030551   \n",
       "34         0.980000        0.030551   \n",
       "35         0.973333        0.032660   \n",
       "36         0.973333        0.032660   \n",
       "37         0.980000        0.030551   \n",
       "38         0.980000        0.030551   \n",
       "39         0.966667        0.044721   \n",
       "40         0.966667        0.033333   \n",
       "41         0.966667        0.044721   \n",
       "42         0.966667        0.033333   \n",
       "43         0.966667        0.044721   \n",
       "44         0.973333        0.032660   \n",
       "45         0.973333        0.032660   \n",
       "46         0.960000        0.044222   \n",
       "47         0.973333        0.032660   \n",
       "48         0.966667        0.033333   \n",
       "49         0.973333        0.032660   \n",
       "50         0.960000        0.044222   \n",
       "51         0.966667        0.044721   \n",
       "52         0.966667        0.044721   \n",
       "53         0.980000        0.030551   \n",
       "54         0.953333        0.042687   \n",
       "55         0.973333        0.032660   \n",
       "56         0.953333        0.042687   \n",
       "57         0.973333        0.032660   \n",
       "58         0.953333        0.042687   \n",
       "59         0.966667        0.033333   \n",
       "\n",
       "                                        params  \n",
       "0     {'n_neighbors': 1, 'weights': 'uniform'}  \n",
       "1    {'n_neighbors': 1, 'weights': 'distance'}  \n",
       "2     {'n_neighbors': 2, 'weights': 'uniform'}  \n",
       "3    {'n_neighbors': 2, 'weights': 'distance'}  \n",
       "4     {'n_neighbors': 3, 'weights': 'uniform'}  \n",
       "5    {'n_neighbors': 3, 'weights': 'distance'}  \n",
       "6     {'n_neighbors': 4, 'weights': 'uniform'}  \n",
       "7    {'n_neighbors': 4, 'weights': 'distance'}  \n",
       "8     {'n_neighbors': 5, 'weights': 'uniform'}  \n",
       "9    {'n_neighbors': 5, 'weights': 'distance'}  \n",
       "10    {'n_neighbors': 6, 'weights': 'uniform'}  \n",
       "11   {'n_neighbors': 6, 'weights': 'distance'}  \n",
       "12    {'n_neighbors': 7, 'weights': 'uniform'}  \n",
       "13   {'n_neighbors': 7, 'weights': 'distance'}  \n",
       "14    {'n_neighbors': 8, 'weights': 'uniform'}  \n",
       "15   {'n_neighbors': 8, 'weights': 'distance'}  \n",
       "16    {'n_neighbors': 9, 'weights': 'uniform'}  \n",
       "17   {'n_neighbors': 9, 'weights': 'distance'}  \n",
       "18   {'n_neighbors': 10, 'weights': 'uniform'}  \n",
       "19  {'n_neighbors': 10, 'weights': 'distance'}  \n",
       "20   {'n_neighbors': 11, 'weights': 'uniform'}  \n",
       "21  {'n_neighbors': 11, 'weights': 'distance'}  \n",
       "22   {'n_neighbors': 12, 'weights': 'uniform'}  \n",
       "23  {'n_neighbors': 12, 'weights': 'distance'}  \n",
       "24   {'n_neighbors': 13, 'weights': 'uniform'}  \n",
       "25  {'n_neighbors': 13, 'weights': 'distance'}  \n",
       "26   {'n_neighbors': 14, 'weights': 'uniform'}  \n",
       "27  {'n_neighbors': 14, 'weights': 'distance'}  \n",
       "28   {'n_neighbors': 15, 'weights': 'uniform'}  \n",
       "29  {'n_neighbors': 15, 'weights': 'distance'}  \n",
       "30   {'n_neighbors': 16, 'weights': 'uniform'}  \n",
       "31  {'n_neighbors': 16, 'weights': 'distance'}  \n",
       "32   {'n_neighbors': 17, 'weights': 'uniform'}  \n",
       "33  {'n_neighbors': 17, 'weights': 'distance'}  \n",
       "34   {'n_neighbors': 18, 'weights': 'uniform'}  \n",
       "35  {'n_neighbors': 18, 'weights': 'distance'}  \n",
       "36   {'n_neighbors': 19, 'weights': 'uniform'}  \n",
       "37  {'n_neighbors': 19, 'weights': 'distance'}  \n",
       "38   {'n_neighbors': 20, 'weights': 'uniform'}  \n",
       "39  {'n_neighbors': 20, 'weights': 'distance'}  \n",
       "40   {'n_neighbors': 21, 'weights': 'uniform'}  \n",
       "41  {'n_neighbors': 21, 'weights': 'distance'}  \n",
       "42   {'n_neighbors': 22, 'weights': 'uniform'}  \n",
       "43  {'n_neighbors': 22, 'weights': 'distance'}  \n",
       "44   {'n_neighbors': 23, 'weights': 'uniform'}  \n",
       "45  {'n_neighbors': 23, 'weights': 'distance'}  \n",
       "46   {'n_neighbors': 24, 'weights': 'uniform'}  \n",
       "47  {'n_neighbors': 24, 'weights': 'distance'}  \n",
       "48   {'n_neighbors': 25, 'weights': 'uniform'}  \n",
       "49  {'n_neighbors': 25, 'weights': 'distance'}  \n",
       "50   {'n_neighbors': 26, 'weights': 'uniform'}  \n",
       "51  {'n_neighbors': 26, 'weights': 'distance'}  \n",
       "52   {'n_neighbors': 27, 'weights': 'uniform'}  \n",
       "53  {'n_neighbors': 27, 'weights': 'distance'}  \n",
       "54   {'n_neighbors': 28, 'weights': 'uniform'}  \n",
       "55  {'n_neighbors': 28, 'weights': 'distance'}  \n",
       "56   {'n_neighbors': 29, 'weights': 'uniform'}  \n",
       "57  {'n_neighbors': 29, 'weights': 'distance'}  \n",
       "58   {'n_neighbors': 30, 'weights': 'uniform'}  \n",
       "59  {'n_neighbors': 30, 'weights': 'distance'}  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the results\n",
    "pd.DataFrame(grid.cv_results_)[['mean_test_score', 'std_test_score', 'params']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.98\n",
      "{'n_neighbors': 13, 'weights': 'uniform'}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the best parameters to make predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train your model using all data and the best known parameters\n",
    "knn = KNeighborsClassifier(n_neighbors=13, weights='uniform')\n",
    "knn.fit(X, y)\n",
    "\n",
    "# make a prediction on out-of-sample data\n",
    "knn.predict([[3, 5, 4, 2]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shortcut: GridSearchCV automatically refits the best model using all of the data\n",
    "grid.predict([[3, 5, 4, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reducing computational expense using `RandomizedSearchCV`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Searching many different parameters at once may be computationally infeasible\n",
    "- `RandomizedSearchCV` searches a subset of the parameters, and you control the computational \"budget\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify \"parameter distributions\" rather than a \"parameter grid\"\n",
    "param_dist = dict(n_neighbors=k_range, weights=weight_options)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Important:** Specify a continuous distribution (rather than a list of values) for any continous parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'weights': 'distance', 'n_neighbors': 16}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>{'weights': 'uniform', 'n_neighbors': 22}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.030551</td>\n",
       "      <td>{'weights': 'uniform', 'n_neighbors': 18}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'weights': 'uniform', 'n_neighbors': 27}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.953333</td>\n",
       "      <td>0.042687</td>\n",
       "      <td>{'weights': 'uniform', 'n_neighbors': 29}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'weights': 'distance', 'n_neighbors': 10}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.966667</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>{'weights': 'distance', 'n_neighbors': 22}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'weights': 'uniform', 'n_neighbors': 14}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.044222</td>\n",
       "      <td>{'weights': 'distance', 'n_neighbors': 12}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.973333</td>\n",
       "      <td>0.032660</td>\n",
       "      <td>{'weights': 'uniform', 'n_neighbors': 15}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_test_score  std_test_score                                      params\n",
       "0         0.973333        0.032660  {'weights': 'distance', 'n_neighbors': 16}\n",
       "1         0.966667        0.033333   {'weights': 'uniform', 'n_neighbors': 22}\n",
       "2         0.980000        0.030551   {'weights': 'uniform', 'n_neighbors': 18}\n",
       "3         0.966667        0.044721   {'weights': 'uniform', 'n_neighbors': 27}\n",
       "4         0.953333        0.042687   {'weights': 'uniform', 'n_neighbors': 29}\n",
       "5         0.973333        0.032660  {'weights': 'distance', 'n_neighbors': 10}\n",
       "6         0.966667        0.044721  {'weights': 'distance', 'n_neighbors': 22}\n",
       "7         0.973333        0.044222   {'weights': 'uniform', 'n_neighbors': 14}\n",
       "8         0.973333        0.044222  {'weights': 'distance', 'n_neighbors': 12}\n",
       "9         0.973333        0.032660   {'weights': 'uniform', 'n_neighbors': 15}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# n_iter controls the number of searches\n",
    "rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring='accuracy', n_iter=10, random_state=5, return_train_score=False)\n",
    "rand.fit(X, y)\n",
    "pd.DataFrame(rand.cv_results_)[['mean_test_score', 'std_test_score', 'params']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.98\n",
      "{'weights': 'uniform', 'n_neighbors': 18}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(rand.best_score_)\n",
    "print(rand.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.973, 0.98, 0.98, 0.98, 0.973, 0.98, 0.98, 0.973, 0.98, 0.973, 0.973, 0.98, 0.98, 0.98, 0.98, 0.973, 0.98, 0.98, 0.98, 0.973]\n"
     ]
    }
   ],
   "source": [
    "# run RandomizedSearchCV 20 times (with n_iter=10) and record the best score\n",
    "best_scores = []\n",
    "for _ in range(20):\n",
    "    rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring='accuracy', n_iter=10, return_train_score=False)\n",
    "    rand.fit(X, y)\n",
    "    best_scores.append(round(rand.best_score_, 3))\n",
    "print(best_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "- scikit-learn documentation: [Grid search](http://scikit-learn.org/stable/modules/grid_search.html), [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html), [RandomizedSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "- Timed example: [Comparing randomized search and grid search](http://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html)\n",
    "- scikit-learn workshop by Andreas Mueller: [Video segment on randomized search](https://youtu.be/0wUF_Ov8b0A?t=17m38s) (3 minutes), [related notebook](https://github.com/amueller/pydata-nyc-advanced-sklearn/blob/master/Chapter%203%20-%20Randomized%20Hyper%20Parameter%20Search.ipynb)\n",
    "- Paper by Yoshua Bengio: [Random Search for Hyper-Parameter Optimization](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comments or Questions?\n",
    "\n",
    "- Email: <kevin@dataschool.io>\n",
    "- Website: http://dataschool.io\n",
    "- Twitter: [@justmarkham](https://twitter.com/justmarkham)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    @font-face {\n",
       "        font-family: \"Computer Modern\";\n",
       "        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "    }\n",
       "    div.cell{\n",
       "        width: 90%;\n",
       "/*        margin-left:auto;*/\n",
       "/*        margin-right:auto;*/\n",
       "    }\n",
       "    ul {\n",
       "        line-height: 145%;\n",
       "        font-size: 90%;\n",
       "    }\n",
       "    li {\n",
       "        margin-bottom: 1em;\n",
       "    }\n",
       "    h1 {\n",
       "        font-family: Helvetica, serif;\n",
       "    }\n",
       "    h4{\n",
       "        margin-top: 12px;\n",
       "        margin-bottom: 3px;\n",
       "       }\n",
       "    div.text_cell_render{\n",
       "        font-family: Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
       "        line-height: 145%;\n",
       "        font-size: 130%;\n",
       "        width: 90%;\n",
       "        margin-left:auto;\n",
       "        margin-right:auto;\n",
       "    }\n",
       "    .CodeMirror{\n",
       "            font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
       "    }\n",
       "/*    .prompt{\n",
       "        display: None;\n",
       "    }*/\n",
       "    .text_cell_render h5 {\n",
       "        font-weight: 300;\n",
       "        font-size: 16pt;\n",
       "        color: #4057A1;\n",
       "        font-style: italic;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "    }\n",
       "\n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "        }\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"]\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}