Hyperparameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Beginning Machine Learning with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring hyperparameters\n",
    "\n",
    "In the introductory lesson and at various points in other earlier lessons, we utilized **hyperparameters** to models to tune their performance.  In this lesson we will look at hyperparamters a bit more systematically, and especially look at *grid search* which is a nice API to use in exploring hyperparametric space.\n",
    "\n",
    "While there is overlap in the hyperparameters used by different models, the same name often has a somewhat different meaning because the underlying mathematical process is different.  Moreover, different models usually have mostly different collections of hyperparameters that pertain to them.  Learning the available hyperparameters is a matter of learning about the individual model class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from src.setup import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Wisconsin breast cancer dataset\n",
    "\n",
    "For this lesson, we will look at another sample dataset included with scikit-learn.  The cancer dataset has 30 features and a binary target of \"malignant\" or \"benign.\"  This dataset is moderate sized with 569 samples.  \n",
    "\n",
    "Our goal in this lesson is not to identify the *optimal* classifier and hyperparameters, but simply to explore how to work with the parametric space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['malignant', 'benign'], dtype='<U9')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cancer = load_breast_cancer()\n",
    "cancer.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _breast_cancer_dataset:\n",
      "\n",
      "Breast cancer wisconsin (diagnostic) dataset\n",
      "--------------------------------------------\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    :Number of Instances: 569\n",
      "\n",
      "    :Number of Attributes: 30 numeric, predictive attributes and the class\n",
      "\n",
      "    :Attribute Information:\n",
      "        - radius (mean of distances from center to points on the perimeter)\n",
      "        - texture (standard deviation of gray-scale values)\n",
      "        - perimeter\n",
      "        - area\n",
      "        - smoothness (local variation in radius lengths)\n",
      "        - compactness (perimeter^2 / area - 1.0)\n",
      "        - concavity (severity of concave portions of the contour)\n",
      "        - concave points (number of concave portions of the contour)\n",
      "        - symmetry\n",
      "        - fractal dimension (\"coastline approximation\" - 1)\n",
      "\n",
      "        The mean, standard error, and \"worst\" or largest (mean of the three\n",
      "        worst/largest values) of these features were computed for each image,\n",
      "        resulting in 30 features.  For instance, field 0 is Mean Radius, field\n",
      "        10 is Radius SE, field 20 is Worst Radius.\n",
      "\n",
      "        - class:\n",
      "                - WDBC-Malignant\n",
      "                - WDBC-Benign\n",
      "\n",
      "    :Summary Statistics:\n",
      "\n",
      "    ===================================== ====== ======\n",
      "                                           Min    Max\n",
      "    ===================================== ====== ======\n",
      "    radius (mean):                        6.981  28.11\n",
      "    texture (mean):                       9.71   39.28\n",
      "    perimeter (mean):                     43.79  188.5\n",
      "    area (mean):                          143.5  2501.0\n",
      "    smoothness (mean):                    0.053  0.163\n",
      "    compactness (mean):                   0.019  0.345\n",
      "    concavity (mean):                     0.0    0.427\n",
      "    concave points (mean):                0.0    0.201\n",
      "    symmetry (mean):                      0.106  0.304\n",
      "    fractal dimension (mean):             0.05   0.097\n",
      "    radius (standard error):              0.112  2.873\n",
      "    texture (standard error):             0.36   4.885\n",
      "    perimeter (standard error):           0.757  21.98\n",
      "    area (standard error):                6.802  542.2\n",
      "    smoothness (standard error):          0.002  0.031\n",
      "    compactness (standard error):         0.002  0.135\n",
      "    concavity (standard error):           0.0    0.396\n",
      "    concave points (standard error):      0.0    0.053\n",
      "    symmetry (standard error):            0.008  0.079\n",
      "    fractal dimension (standard error):   0.001  0.03\n",
      "    radius (worst):                       7.93   36.04\n",
      "    texture (worst):                      12.02  49.54\n",
      "    perimeter (worst):                    50.41  251.2\n",
      "    area (worst):                         185.2  4254.0\n",
      "    smoothness (worst):                   0.071  0.223\n",
      "    compactness (worst):                  0.027  1.058\n",
      "    concavity (worst):                    0.0    1.252\n",
      "    concave points (worst):               0.0    0.291\n",
      "    symmetry (worst):                     0.156  0.664\n",
      "    fractal dimension (worst):            0.055  0.208\n",
      "    ===================================== ====== ======\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Class Distribution: 212 - Malignant, 357 - Benign\n",
      "\n",
      "    :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian\n",
      "\n",
      "    :Donor: Nick Street\n",
      "\n",
      "    :Date: November, 1995\n",
      "\n",
      "This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.\n",
      "https://goo.gl/U2Uwz2\n",
      "\n",
      "Features are computed from a digitized image of a fine needle\n",
      "aspirate (FNA) of a breast mass.  They describe\n",
      "characteristics of the cell nuclei present in the image.\n",
      "\n",
      "Separating plane described above was obtained using\n",
      "Multisurface Method-Tree (MSM-T) [K. P. Bennett, \"Decision Tree\n",
      "Construction Via Linear Programming.\" Proceedings of the 4th\n",
      "Midwest Artificial Intelligence and Cognitive Science Society,\n",
      "pp. 97-101, 1992], a classification method which uses linear\n",
      "programming to construct a decision tree.  Relevant features\n",
      "were selected using an exhaustive search in the space of 1-4\n",
      "features and 1-3 separating planes.\n",
      "\n",
      "The actual linear program used to obtain the separating plane\n",
      "in the 3-dimensional space is that described in:\n",
      "[K. P. Bennett and O. L. Mangasarian: \"Robust Linear\n",
      "Programming Discrimination of Two Linearly Inseparable Sets\",\n",
      "Optimization Methods and Software 1, 1992, 23-34].\n",
      "\n",
      "This database is also available through the UW CS ftp server:\n",
      "\n",
      "ftp ftp.cs.wisc.edu\n",
      "cd math-prog/cpo-dataset/machine-learn/WDBC/\n",
      "\n",
      ".. topic:: References\n",
      "\n",
      "   - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction \n",
      "     for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on \n",
      "     Electronic Imaging: Science and Technology, volume 1905, pages 861-870,\n",
      "     San Jose, CA, 1993.\n",
      "   - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and \n",
      "     prognosis via linear programming. Operations Research, 43(4), pages 570-577, \n",
      "     July-August 1995.\n",
      "   - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques\n",
      "     to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) \n",
      "     163-171.\n"
     ]
    }
   ],
   "source": [
    "print(cancer.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive classification\n",
    "\n",
    "For now, we use K Nearest Neighbors (KNN) classification, mostly because it is easy to understand.  The general idea of KNN is simply to identify the K points that are \"closest\" to a test point or newly observed point, and let the plurality win.$^1$  KNN does quite well for numerous classification and regression problems.\n",
    "\n",
    "<hr/>\n",
    "<small>$^1$<i>The winner may not be a majority.  For example with 8 nearest neighbors and four classes, we might have a predicted point whose closest neighbors are 2 points from each class.  The tie is broken arbitrarily by the order of the training data.  Even if we had 9 neighbors and the count of those nearby were <code>{A:3, B:2, C:2, D:2}</code>, letting A win would still be with only ⅓ of neighbors \"voting\" for A.</i></small> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9370629370629371"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train, y_train)\n",
    "knn.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A 93% score might strike you as quite good, especially having seen much worse ones in other examples.  However, the thing we are trying to predict is literally a life or death matter, which makes the number seem less impressive.  Moreover, we have not here teased out the differences between false positives and false negatives in that accuracy score.  Presumably, in this domain we would rather have more false positives than false negatives because unnecessarily treatment (or unnecessary additional testing) is less bad than a missed diagnosis.\n",
    "\n",
    "In a later lesson we look at metrics in more detail. For this lesson, we will only look at this model `.score()` method as our optimization goal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring one hyperparameter\n",
    "\n",
    "The most obvious hyperparameter for KNN classification is the number of neighbors used.  Many aspects of the data—from number of samples, to number of dimensions, to multi-modality in univariate distribution of features—can greatly affect the \"right\" answer.  Moreover, if we really want to arrive at the best classification, we should look at scaling issues that will be glossed over here but discussed in a later lesson on feature engineering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     0.923077\n",
       "2     0.909091\n",
       "3     0.923077\n",
       "4     0.902098\n",
       "5     0.937063\n",
       "6     0.937063\n",
       "7     0.930070\n",
       "8     0.937063\n",
       "9     0.923077\n",
       "10    0.923077\n",
       "11    0.923077\n",
       "12    0.923077\n",
       "13    0.923077\n",
       "14    0.923077\n",
       "15    0.923077\n",
       "16    0.923077\n",
       "17    0.916084\n",
       "18    0.923077\n",
       "19    0.916084\n",
       "20    0.916084\n",
       "21    0.916084\n",
       "22    0.916084\n",
       "23    0.909091\n",
       "24    0.916084\n",
       "25    0.909091\n",
       "26    0.909091\n",
       "27    0.909091\n",
       "28    0.909091\n",
       "29    0.909091\n",
       "30    0.909091\n",
       "31    0.909091\n",
       "32    0.909091\n",
       "33    0.909091\n",
       "34    0.909091\n",
       "35    0.909091\n",
       "36    0.902098\n",
       "37    0.895105\n",
       "38    0.902098\n",
       "39    0.902098\n",
       "Name: Score, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = []\n",
    "for k in range(1, 40):\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    score = knn.score(X_test, y_test)\n",
    "    scores.append(score)\n",
    "    \n",
    "scores = pd.Series(scores, index=range(1,40), name=\"Score\")\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is easier to see a pattern if we visualize the trend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAFOCAYAAADD+bpyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABK8ElEQVR4nO3dCZwUd5k//qfnYIaBGRiYi/u+ZshhgsRwBYhcGjXrHiau63pEIcasLnvo6u6q//3726y6m1V/MYfGK2ajMYpHNglgEgJESIAkJMxwzEBCuOaAAeZk7t/nqamaFJ05+qjqrq76vF+vh+qu7uqqLmq6n/6eoZ6eHiEiIiIib0hL9gEQERER0VuYnBERERF5CJMzIiIiIg9hckZERETkIUzOiIiIiDyEyRkRERGRhzA5I/KoUCg0FdGDyIjguR9D7EzEcZEz8P/1Y8T/n4zzif2qHyHOI150+LX/ErElwucOet3isW2I25w7OqLUwOSMyAH4AnkD0Y4oCFv/iplgTeWJdh/O81cRP+O5HtISxCrExJ6enoVOni+83sOI1fw/IIodkzMi57yOuNW6gyThCiyGe+0ER1ISR6kF/6fpUW4yBfEGkqhmN47Ha3jNU6phckbknIcQH7Xd/2vET8O+JEYhfoqoQxxH/DMizfqCRXwLcRZxDKve28+2DyLOIE5plVgkX8q26tFPIt7EqmfM9Z9AHDSrtjYjptiqvO5G1CIuIl5FzLdVxd2H2IpoRDxnbWc+vgixx9xOl4vCqqj+DfG8ue0Wq6QRy2wt8UKcQ1wwty2O5n1j3VosvoT4EG43Ifab68cjfoeoR1QhPjXIudL3dw/if81jfAExI+w8ZvRX7WZW0T1vnjt9D8fM86HrT5jnU68Ju4JBzuVc8zE97sOIvwg7znsRTyA0wVrRz3vp933rdYDFDxDXm+fpawNVN5rXo14fryPW2R4f8P/E2tb23NXm8es18T3zfV5WVTnQfkwzsO5Fc/vfIsbYtns/otw83/p/MS+sNPsLev3ibrP+v5n3T5nnW4/pxvD3TuQJOn0Tg+eA10B81wC8gXg34jBCvyD0i+oEQr9s9QlTzedpsvZbRK6uQxxBfNJ8bAPiEGISQr+AnjW3zTAf/w3ifsQIRBFC2wqtNx/7GGLnAMem++kx9z3CLM27GVFlHqsmG/+M+KP5/DWIfYjRiJD5nHHmYz9GNCKWIbIQ37b2ax7zecRfma95q3l/rPn4NsRRxGzzGPT+XeZj6xG/R+SY5+5aRN5Q77uf9/pVxM/C1j2H+B4iG3E1og5x4wDb6/urRyw038PDiJ+HnUfj/8P2nm6z/R90Ij5uvgdtT6bJ8D3muVptnruREZzLEeb183HzOK5BnEWU2ba9iFiM0OQ+u5/3MuD7Hux6sT3egfiU+V5uR5zW6yGaaxE0+W5AfNB8H58zX/e2CPej5/cUYr65r19Z/7/mddRsVs9mIv7RvKaH2f4mXzH/nvR6m2Oe0/G2/88Z/Ozj91+PB89B0g+AwXPgh2vAlpxpkvPvCC3F2Wp+IRnJmfnl04YotW2nSck287aWaG2wPbbaSgYQxea2w22Pa/LzbBTJ2XTbuietpNC8r1/wLWYyudJMGt+l68Ne68dWsmLeH4noMr8ANSl7Mez5u/TYbF+0/2x77DOIp8zbn0D8EXFl2PaDvu+hkjPzuPT4cm3r9P/nxwNsr+/vB7b770EciiI5q7Q9ptXaeqPYtu4c4uoIzuWHEDvCjk2Toa/Ytv3pINfjoO97sOvF9niV7b4mzXqjJJpr0SxJ3mV7XshMkG4baj+282sk8Ob9UkS7+bf0L4hHw65hTeSW2/4mP2F7fCai1vw7zUzG5wSD56AnwnPAtidEzldtbkdMC6/SNEsRhiGO29bp7Qnm7fHmF5f9McsUs3RAq5GsdWlhzx/KibDX+zZe6z9t6/SFJ+CD4Rms/79mic9k3N6E5d9jfUP462CdVovVm8c+PuyYw9+fqrbdbjETEuu8aULxc7yelthpo/4vO/C+9ZjqcZyNYce0YJBtBjrGSNTYbrfqP9h3+Dr76w10LvV9X6fVdbbnZpjn6W3bOvS+BzwPeJ0W8/yPNEtII/0/ueyaxutotfDJCPdjCf+byDT/li673rBtt1Yfh11v9n1r1e7nzQS+DLc3Y7kR67WkjshT2OaMyEH4oD9udgzQEpdfhz181qzC6WtXBJPNX/vqjJmg2B+zf8loaUUB9jHaDK32K4vm8MJeb73ttTS0JOSP5vv4DkKrFsvM6qN/sG3bd4z4grO+rPUL7nTYewt/fwMfWE9PB+JrCC0Z0XZqN5mlLtG+b/t7FPOYxuA4c6M9pn5Yjee1dMeiJUnxGOhc6vt+Luz/R6tDtdpvoPfq1vsOF83/iV7TE607od7Mq+9+hML/JjrMv6XLrjfztSeFvcfLzhGO8X8QS2zNDf4jymMhSggmZ0TO0wbXK8N7wuG+VjM9ivi6fmmajb83IqyhH/Sxv8H6iYh83P6ibVv9ktOxo/4Tj+Uh0rShOuKGGI/xPsQ/YfsyWwPvPzdvvxOhpTZaQqHv4RJCj93yHjy2BKGlgP+GeAHHp1/YTyBmY/2HzcbXWjWnydbjQx0MnrsCcYXZqLzB/ALuiuF9aymVNtw3PtvM49KE89/NTgdXmv8/2pYsKnitOvOL/yN6nAitijU6C8RhoHP5uHku/0r/H8x4p73B+xDH6tj77ue1o/k/+V+E/r/erNcEbt8RQ0Kr57sUoUnx/4d4zPa39F5t1G9eq39nJo3GD4xweM4cxEpElnlNt4Zd10SeweSMyGH44jiK2DvAw3eaCY/2xtQebf+D+KH52PcRWtWivQxf6qfkTUuS9Eu8wmxo/xhiXIzHuMksNdBqRE2GDiCsXnJ55rHoPo6b7aS+Zdtcj/krCK2C09K1vzRf85xZ4qVfkufMBto3Yb2WcgylxHw/eiwHzcbsP4vhff/SXGqvTz2HVnuoqWZJyyaz3Za2B4zFp8xSRH1/ZQMlAlEY6Fw2mm0ObzGPu9r8/9LEIlJOvu9wEf2fmP/3mvR/wzxnmqzvNZOoSD1ktrGrNjs3/I352tr55iOI7yJ0P+/TwHptk9YfPXd3mc+tNjsyaO9eIs+xesQQEQ39gREK6ZfkSXxuaMcHoui+cHpLNLXN2V/iGtLeyETUD5acERGRmwnZGu3kYVYnakmVtg3bzVNONDAmZ0RE5KbrzfHtrKrHm1FqZvRkJaL+sVqTiIiIyENYckZERETkIUzOiIiIiDzEVzMEFBQU9Eydqj3HiYiIiLxt3759Z9EGs9DXyZkmZnv3DjS8FBEREZF3hEKh8CnvDKzWJCIiIvIQJmdEREREHsLkjIiIiMhDmJwREREReQiTMyIiIiIPYXJGRERE5CFMzoiIiIg8xFfjnFF0urp7ZNvhWik/3SBl4/Nk+ZwiSU8LubZdtBK1HyIiIi9hchZQmvisv2+7VFdUydLynXJ32RJ5pHSm3L9h2aAJUKzbJer4iIiIUh2Ts4DSEilNfDbdu14yu7tk4/aH5H3r75P//kOJXDVx9IDb7T95QU4dqJTf3b+hb7ubb38ArzdXbpxX7OrxubEfIiIir2FyFlBaVaglUpr4KF3eUPG8fDdv3JDbrsfz7NstK98hFaeXOpo09Xd8buyHiIjIa5icBZS24dKqQi2R0sSnIy1dtuP+v7y3VBZOGzPgdi++Xi+/alwsf7/Dvt1S2YjXc//4nN8PERGR1zA5CyhtXK9tuN7z6ftk5cHnZef8pTKhdIZ8bPHUQdt0lSI52vXaLHkvtluB7XYggRqP7fT1nD6+n8yZLqs++T1Zc2SXPDtvsUx2YT9EREReE+rp6Un2MThmwYIFPXv37k32YaQMbXT/0QdfQDuyi/LtW66OqrfmY/tOyBd+9Zr82bUT5T/+9EpXGuk//MJx+fKmA8Zrv3Nqvjx827vYGYCIiHwjFArtQx62IHw9xzkLME16RmRlyITRw412XJEmWPq8D71zslw5cZRU1ja5ljBtLq+RKWNzZNGMsdLU1snEjIiIAoHJWcA1XOqQUcMzY9p27fwS2X/igpy+0OrwUYlcbOmQP1adNfYxqyhXjtY2SzdK7IiIiPyOyVnAXWztlLzhsTU9XDe/t2fnUweqnTwkwx8O1kgnkjHdx8yikdLa0SWnXEgCiYiIvIbJWcA1tHYgOYut5GxawQiZW5LrSnL2JF5z/KhsuQpVp7OKRxrrquqaHN8PERGR1zA5CzgjOcuOLTlTWu2453i91DZecuyYtH3Z9so6WYPXRmNJmVloJmc1TM6IiMj/mJwFmPa6bEQiFGubM6XVjtrhd0t5jWPH9eyhWmnv7O6rNs0fMUwKRg6TqlomZ0RE5H9MzgKs8VKHsYwnOZuNKsfpqN50smpTX6tgZJZcOyW/b90MlJ5V1jY6tg8iIiKvYnIWYBdRpalibXOmtNpRqzZ3HTsn55vb4z6mSx1d8uzhWllTdvnQHtruTEvO/DQuHxERUX+YnAWYlZzFU3KmtPpRq0i3Hoy/avO5I3XS0t7VV6Vp0XZnDZc6pa6xLe59EBEReRmTswBraO00lnnZ8c3iNX9CnkzMH+5I1aa+xuicTLlu+uXze84qzjWWbHdGRER+x+QswPpKzpAMxcOo2iwrkZ2VZ/vascVCOwHo+Gar5hVLZvrll6aOdaZ0RgIiIiI/Y3IW8NkBnKjWVOuuKJH2rm555lBtzK/x/FFN7jqN1wpXlJsluSjhY8kZERH5HZOzAOvrEBDHOGeWd0zKl+K8LHnytdirNp/CtrlZGbJ4ZsHbHjPGO0PpGXtsEhGR3zE5C3hylpEWkpxh6XG/VhpeZw2qNrcdqZWW9t62bNHoRKnblopqWTmvSLIy+j+eWUjOqmqb4z1UIiIiT2NyFmAN5tRNWirlBB1S41JHtzx3uC7qbV98vV7Ot3TIOrzGQLTk7GxTm1xoiX/IDiIiIq9ichbwkjMn2ptZFk4dI/k5mca8mNHSbYZnpssNs4sGfM6sIvbYJCIi/2NyFmA6blg8A9CGy0hPk9WlJUangLbOroi36+7ukc3l1bJ8TqEMH6SKlT02iYgoCJicBbzkLN4xzsKtvaLEmLhch9WI1EtvnpfaxjajWnQwE0YPl+zMNPbYJCIiX2NyFvA2Z05Wa6rFMwqMIS+iqdrU5w5DqdvKuQNXaVqdDnrn2ORYZ0RE5F9MzgLM6hDgpGEZafLuecWytaJGOrq6h3y+zpWpswIsnaVJ3dDHolWbR5mcERGRjzE5CyhNipzuEGDR6kl97d3Hzg353NdOXZRTF1qHrNK0D6ehz29G1SkREZEfMTkLqNaOLuns7nElObthdqExdlokVZv6HB1rbVVpcUSvbXUKOFrHqk0iIvInJmcB5eTsAOGyM9NlxZwi2VJeLV1IAIeq0rx+xlgZnTMsoteeaQ6nUVnD5IyIiPyJyVlANbT2Vgu6UXKmtJrybFO77H2jfsDnHK5plNfPNkdcpammjM0xStqqWHJGREQ+xeQs6CVnw50dSsOyYm6R0TlgsKpNnYdTJyfQsdEilZmeJtMKRrDkjIiIfIvJWcCTM7dKzkZmZciyWYXG4LI6yGx/tErznVPHSGFuVlSvbfTYZMkZERH5FJOzAA+j4WZypnSezDMXL8n+kxfe9tgxJFdarTnYXJqD9dg8fq5ZLnVEPgsBERFRqmByFlBudgiw6Hhn2j5MS8jCWdWd0bQ3s8xAcqaFcW8gQSMiIvIbJmcB1XCpNznT0fzdMionUxbNLDASMe2ZaacJ29WTRsu4UcOjfl1rAnT22CQiIj9ichbgkjNtF6aTlbtJqy3frG+RijMNfetO4L4OPhtLlaaaXjjC6EhQxZkCiIjIh5icBZRbswOEW11aLKjZvKxqUzsJqHXzx8U8jtrkMTkcToOIiHyJyVmAxzlzel7N/owdmSULp425bEgNvV06Lk8mj82J+XVnFo6UKg5ES0REPsTkLMiTnrvY3sxOS8i0CrKqtlFqGi7JvuPnY67StMwsHmkMYNsZweTqREREqYTJWYA7BCSiWlOtKSvpG3S2r0rzijiTM5SctSMx0/ZsREREfuJqchYKhdYiDiOqEF/s5/F8xCbEq4gXEfPN9dnm/f2IcsTX3DzOoLY5S0S1pioZlS3XTB5tVGdqgqaDyFpzZMZqVnHv9uwUQEREfuNacoaEKh2LexDrEKWIW7FOl3ZfQrzS09NzJZYfRXzbXN+GWIn1V2F5NUKTvHe5daxBlKgOAfbSM+2xuevYOZlbkjvohOiRmFE4wlhWsscmERH5jJslZwsRVUiwjiHacfvniA+EPUeTtaf1Bp5zCIupSMKKcVs1mc/RDEIjvm9z6tOB6sCW9q6EJWeaiO3Yf1ym1Z+U9bsfk2PPvyTr79seV4KWm50p41Aid5TJGRER+YybydkExAnb/ZPmOrv9iA/qDSRlmsxNQUw076cjXsHNWsRWJGsvuHisgZy6KVEdArYdrpULR16XLQ/eIf/03I/lt/dtkDMVR4318dDqUZacERGR37iZnIX6WRdeVHIXIt9Mwu5EvIzoNJ7Y09OFuNpM1hZa7dHetpNQ6NOIvRp1dXXOHb2PNVzq7BvBPxHKTzfI0vKdktndOxemLpeV75AKrI+HNQH6QBOrExERpSI3kzMtKZtku69J1mn7E5B8NSA+biZh2uasEPF62HN01uxtiLX97QSPP4BYoFFYqJuTF+bVtCsbnyc7ypZIR1p6b7UqltvLlkop1sebnGn17OmLrU4cJhERke+Tsz2IWSjRmoYYhtu3IH5nfwLWjzYfU7chtmvChnWF+pj5HJ188d0IbZNGDiZniWpztnxOkZSUzpSbb39A/mP5x4zluNIZxvp4WHNssscmERH5iWuNjpBkdSKx+ixubkZokckPsU6HxdhgPn4fFvMQP8U6re+qQHzS3Fzn9fmJ2eNTE8hH8fzH3TrWwLY5S1Bylp4Wkvs3LEMbs7moylwqG1FipomZro+35MxKzuJN9IiIiLzC1RbhSKiewOKJsHWalFm3d2Exq5/tXsXiHW4eW5AluuRMaSJ247xiI5wyZsQwGYtgyRkREfkJZwgI6OwAiU7O3DKDPTaJiMhnmJwFtORsWHqaZGWk/n//LCRnWnKG0tZkHwoREZEjUv/bmWKb9BylZmjTl/JnT9udabJZ16STShAREaU+JmcB1NDaieQsMQPQuo09NomIyG+YnAVQoufVdJO9xyYREZEfMDkLaIcAvyRnxXlZkpuVweSMiIh8g8lZQEvOEjU7gNu03ZzRY7OGJWdEROQPTM4CyE/Vmn09NuuYnBERkT8wOQsYHXKit7emPzoEWO3O6hrb5GJL7/htREREqYzJWcA0tXVKd48/BqC1zCo2OwXUNSb5SIiIiOLH5CxgGi51Gks/JWczC3snQGe7MyIi8gMmZwFjVf35pUOAmpA/XLIz09hjk4iIfIHJWcD4aV5N+6Tq0wtGSiXHOiMiIh9gchbAnppKp2/yE213xoFoiYjID5icBTQ581PJmZpZOFJOXWiV5rbeNnVERESpislZwOgwGn4tOVPH6pqTfCRERETxYXIWwOQsFBJjyiM/sebYrKzlcBpERJTamJwFcCgNTczS0pCh+ciUsSMkA++J7c6IiCjVMTkL4ryaPqvSVJnpaTK1YAR7bBIRUcpjchYwfptXM7xTwFEOp0FERCmOyVkA25z5NTnTTgFvnGuWts6uZB8KERFRzJicBbFa00ezA4R3CtB5Q98425LsQyEiIooZk7MAzhDg15Iz9tgkIiI/YHIWyA4B/hpGwzIDbc50mBD22CQiolTG5CxAtC3WpY5u35acZWemy6T8HPbYJCKilMbkLEAaWnunNvJrcmZVbbLHJhERpTImZwHi10nP7WYhOdMpnDq7upN9KERERDFhchawzgB+T85mIDlrR2J24nxrsg+FiIgoJkzOglhy5tOhNKySM8VOAURElKqYnAVsAFq/tznTkjPFCdCJiChVMTkLkCAkZ1oqWJKXzZIzIiJKWUzOAtkhwJ/jnNl7bLJak4iIUhWTswBpuNQp2ZlpkpWRnuxDSUhy1tPTk+xDISIiipq/i1Ac1NXdI9sO10r56QYpG58ny+cUSXpaKKX2dbHFv/Nq2k0vHCEt7V3y9ScOyvXTx0Z0/mI557H+PyVqX14/vlglcl9ERMkQ8lPpwoIFC3r27t3rypfB+vu2S3VFlSwt3yk7ypZISelMuX/DMse/FNzc14aH9snRuibZuvEGh47We/T83XL303L28FFZU7lbdkZw/mI557H+PyVqX14/vlglcl9ERG4LhUL7kIcteNt6JmdDe/pgjdx9z+Oy6d71ktndJR1p6XLz7Q/IxjveKzfOK3Z8X/+Fff3GhX19+Pu7pb2zWx67fZFDR+s9/Z2/1Z/8nlQXT5KM9P5r8XXA2nE1J2Tzg59xdZtE7ivZx+fm30ei/haJiJKVnLFaMwJafaK/0vXLQOlyWfkOqTi91PEvBDf3pR0CivOynThMz9Lztyzs/K2p3CUvXF0mV08a3e82r5y4INft3OX6NoncV7KPLxX/PoiIvILJWQS0XcvdqD7ZuP2hvl/r28uWykasd2Nf35i3WP7OhX3pDAGzi3MdOMrU+r/aoedvxcwBv7yN0piKJdLh8jaJ3Feyj8/Vv4+5ixLyt0hElCxMziKgDY4fQbuWm9bfJ8srnjfauYwvnWGsd5q+5tdLxssqVCWtRYmFfjGOc2hfvR0CMgLxf6VVXVqisj2C85eobYJwfEsO7JCtcxbJdBf/Pr6YXyRrbvuerDq8S57BD5kpLu2LiChZ2OYsiobIP9hxTP79yUPyNzfOlM/dONu1Bsgf+QEasledk5K8LPn6n1zhSG+0bhz/jC8/IXeiBGTj6jkOHak3Wb35KlAFVhplb0O3t/H78T38wpvyzKFa2fz5pTKnxPnSrNMXWmXRXc/IB6+ZIPuOnzd65e764spB29EREXkV25zFSb+cbnnnZCM5GzEsw9WeYcfqmo0lvu8ca0fT2NYp2jHXz5OeW/T/Rs9bNOcuUdv4/fjmTxgl1/2fp2VLeY0rydlTB6qN5Z0rZ8muo+fkS5tekyM1TUYSSUTkF/y5GYVROZlSmJvl6ujzTUiiTl+8JMMz06WuqU06urodnbopCMkZJY92OLl2Sr48aSZRbiRnc0tyZVrBCFldViz6G+mpA2dc2RcRUbIwOYvSzMKRUulicnbUfO13TR9jlHTVNbY5OnWTn+fVJG9YN79EKs40yJvnWhx93drGS7LneL2sxeurgpFZ8s6pY1xLBImIkoXJWZRmFY80Eii3xoezEr8lswqN5RmUojlachaAGQIoudaU9SZPTzpcoqVVpfpnt27+uMsSQf2b4VyqROQnTM5imLdR22/VNDhTohVOv2SGoXHzddPGGPdrGhxKzi6x5IwSY9KYHLkCbc+cLtHSKs3pqM6cjR9IlrVmosaqTSLyEyZnMSRnyq1f6lW1jUZ7mgmjhztactZXrYl2c0Ru06pHHZz2zMVWR17vfHO77Dp2znjdUOitzjglo7LlHZNHs2qTiHyFyVmMyVklkig3aNKn+xiNJCorI02qHfpys5Izv49zRt6g1Y323pXx2nqwxhiuw16lad+XzhzgdBs3IqJkYXIWpUI0QtZG9W6UnF3q6JI361uM5ExLB8ahVKDaoerThtZOo2fbyCwmZ+S+6YUjZU5xrmMlWprkTcwfLvMnvH3IDCthe6qcvTaJyB+YnEVJkyZNntzosfn62WZjbDOrdE6HJXCy5EyH0bBXCRG5aQ1KtPa8UR93j+NGtJfcWXlW1pZdXqVpb+Om0zqx1yYR+QWTsxjMQvJkDXnhJCvh0x6hSkvOHOutiS84DqNBiaTVjdq7cktFfKVnOuNAe1e3rLuiZNB9vfymc23ciIh8m5zhV+5axGFEFeKL/Tyej9iEeBXxImK+uX4S4lnEQUQ54nNuHme0tGTrHBoo1yOcpFWlWvWoHQJUyajhUotqTZ16yYmSMyZnlEg6WOzUsTlxtzt78rVqlCJnyTsm5Q/4HKvX5maOeUZEPuBacoaEKh2LexDrEKWIW7FOl3ZfQrzS09NzJZYfRXzbXN+J+Dusn4fluxB39LOt73psak/NKWNHSFZGem9yhi8kLTGob2l3plqTY5xRAmkVpCZNOs3ShRiv4Zb2Ttl2pNYYOy1tkCnT9G9SS7RZtUlEfuBmydlCRBUSrGMI/WT+OeIDYc/RhOtpvYHnHMJiKj7Qi3H7DOIlc712izyImODisXqix6YmezPQkNqiJWeq2oGqTR2EliVnlGha3diJkt+tFTUxbf/c4Tq51NHdNyvAUPvSNm5nm9wZg5CIyA/JmSZTJ2z3T5rr7PYjPqg3kJRpMjcFMdH+BKyfisU7EC+4dqRRGo+kKWdYuqMlZ50oIdMOAVZ7M6vNmVPJ2cXWTskbzp6alFhXThxljNkXa9WmloSNGTFMFk7tHZR5MFpKpy0AdCYBIqJU5mZy1l8dRHjjqbsQ2u7sFSzvRLxsVmn2vkAopJnKrxCfRwlaQ787CYU+jdirUVdX58yRD0GrV7SEy8nk7Hh9i3R09Rhzd9oH2FRnHJglQDsEcNJzSkbVplZJ7qg8a/S6jEZbZ5fRGWB1abFkpA/9UTVvXC6aBeQ4Pm0UEZGfkjMtKZtku68lYqftT9CEC/FxxNVmmzOdUPJ180M900zMHsbjvx5oJ3jsAcQCjcLC3vkoE0HbtziZnFXWXN5T05rYOR2JYLzDaej4ae2d3azWpKTQXpbadlITrWjo8BlNbZ3GkByRt3ErMdq4XWyJLhEkIkqp5AwfeDchYkni9iBmYdtpiGG4fQvid2GvPdp8TN2G2K4JG9ZpqduDiIO4/18x7Nt1M5Cc6TAX0ZYGDORoXW9yZm9zpolZcW4WkrM2h2YH4NRNlHjXTs6XQlzH0VZtapVmbnaGLJ5REPE2OiCt0cbtIKs2iSh1RZJ0aVJViXzpGwjtPRkRJFVaPflZxGazQf+jWKfDYmzQMJ+mr6frDpm9Oq0hMxYj/gqxUqs8zXhPpPtOVMmZOlrX7MjrVdY0Gm1zRoSN4F9szBLQGndnAMUOAZQM2gxgTVmxbDtcJ63tXRFt04GSNu1E8O55xTIsI/Lfhlehjdt4/M1wInQiSmVDthBHQvURJEY6Z8qtiB/htrYb+xHiEbMn5WDbPoHFE2Hr7rPd3oXFrH6224mFp4ey7+uxiaTq6kmj4369KpScaWlcOO0UcKi60ZmSs+EsOaPk0BKtn+1+U547Uts3Jtlgdh87Z1y3kfTSfFsbN2zz8AtvGlWinK6MiFJRRD9Jzcb4vzKHw9BP1j9BvIQPQm3EH0iTx+TIMDRS1qQqXjrIrLZfs0rj7Eryhhu9NfF/EFdnAMWSM0qW66aNkfyczIjHIdPnaY/oG2YXxpQIahvLaNu4ERGlUpuz9+ko/rj5DEKLXhYiUdAqyKsQf+/y8XmW9h7TkfyrzIb88Th1odUYy8kqjbMrGZUlLagKakQpQLwlZ0zOKJl/L6tKi+WZg7VGL8zBdOHHypbyalkxp0iyM3sHZI7GtVPyjc40rNokIj+XnP054m4dxR/xTYTxcxTLFiw+4erRedzM4pGOlJxZvT77LTlzYCBaq+daHhpXEyWLlmjpj4znq84O+ry9xkCy7VFXado70mgbt2cPRd7GjYgo1ZKzryBetO6gFG24OTCsJmjG6P5BpWOSvVnfYgxV4URy1l/JmRMD0TZc6i11Y5szSqZFM8dKblaGMVfmUFWa2glgxdyiuBLBVvxdPnckMWMfEhElOjn7JaLbdr/LXBd4OiaZNgU7FmePTZ0GSqthRudYo4q8pSQv/uRMqzW1/U5mBAN5ErlF54y9cV6RMcyF9sYcqP3lZlRpLptVGFdj/uumj8HfUyarNokoJUXybZ1hzo1pMG+/PYsIIKfm2NSSs5lFI/p9rCgvy1jqmGqx4rya5BXaU/MCqtlfOFbf7+P7T14wrnWdJzMe+kNk1bxieTqCNm5ERKmYnNWhGvP91h3c1snLB280EhDaIQDNW+RoHDMFaC/MSqOnZu6ApQ0FI4dJdUN8JWfsDEBeoL0vh2emDzjFkg5Um4E/Kh3fzImZCbSN2x+rzsX9WkREXkvOdMDYLyEpexOhE5l/AbHe3cNKDZo4TRk7wkiuYlXX2CaNaBPWX3szSzGqNuOZwkmTM84OQF4wHNXrK+YWouqyxuiVGf5DRdubLZpZIKNQJRmvxXgdo40b59okIr8lZ/jAPIp4F26WauD2IkSV+4eWGmbGOcemldgNlpxpp4C4qjWR/LEzAHmpavNsU5vsO37+svUVZxqMDjbxVmnafzyt1DZuFTXSOUAbNyIiL4qohThKzN6LxWcQf4vb/6rh7mGlDk2qXj/bPGAD53iG0bCUIDmraYivzVnecA6jQd6wcm6R0RszvERLqzS1mcDq0virNC2a6J3XNm6v99/GjYgoVQeh1emWPoS405xSScc9m+LycaUMTap0ouXj53TYt+hpZwKd3Fknhh6I9tjUL5hYh+xghwDyEu2FuWxWgWxGMmaf+UKrNBdOGyNjRw78txCtG2YXDdrGjYgoVUvOtBrzo1iex/JrWF6PmOTuYaUOqzqyKsYem9a0TTon4EDiGYhW2/Voo2h2CCCvVW2exvW8/+TFvr8f/VvQ8cmcbuO2fE5vGzcdpoOIyC/JmZURtCCBGI+lDjc/zb1DSi0zCq3krCmOYTQGrtK8bCDaGKo2G815NdkhgLxEh7nQXplWiZY1MO2aMmfam9npTAPa8ealNy9v40ZElMrJ2e+RlI3G8puIlxBvIB5x9ahSyAhU0UwYPTymHpvnm9uNaWoGGkbD3lsz1pIzzqtJXqS9Ma+fMdZoZ2b10rxm8mijfaUrbdzS0yKedJ2IyNPJGZIyffxpfHheQPzKbGs2F7fZIcCBHpvWvJxDlZyVxFFyZiVn7K1JXqNVmNpWU6sctaem01WaltzsTFmKNm5WIkhElNLJGT7ItAvif9rutyF6G4lQH02ujiLRirZNy2BzaoY3oNZOA7GUnDW09s6ryTZn5DWry4qNHkZf+NV+475e4+Fjnzm5r1MXWuXLvzkgTx98+xhrA9Hn6fO/83RlxNslaptYJXJfRBSbSMZX2IIStD/F8tdIzPhX3A9t0H+po9v48J80Jifik19Z02T0JNNq0aFoj80zMQxEy2pN8qr8nGFSlN4lOadOyi1HdsvPmqrlD3tnyv0blkm6jqnhEE0+nnjhqEyrPyl533lM7i5bIo+UDr0f3W79fduluqJKlpbvjGi7RG0Tq0Tui4jcTc42InTix04kaVp0o3/Bmqflxb5b/86xGU1yptWaM4pGSFoEH4patVnd0Bb1sTVYHQI4zhl5zLbDtTK6vkYef/AOyezuko4dD8nNtz+A9XPlRgemb7Lv59yhY7LF2s/2h2Ttp+6Vtf/dmyAO5HxLu/QcPixP/uAzxnYbI9jOqW3cOA/WudDEbNO9613fFxG5O0NALiINMUwTMvM+E7N+h9OIrt1ZVU2jzDR7ew5Fe2zGMoUTS87Iq8pPN8jKQ380kgSly2XlO6QC653ej5YS2ffzbuxXxw3U0qKBQh+/8fCuqLZzahs3zsNA58KtfRGRiyVnKC1b1t96JGjbY9+tv4zGL+KCkVlGNWWkmts6jXGeZhUP3lPTXq2pwwHoNDQZ6RFN7NCXnOmQBVp9SuQlZePzjGo1Lb0xSrTS0mV72VLZiPVu72fn/KXy1feXDVpapO2x7j6xxCjRi3Q7p7Zx4zxY5+Kb8xa7fs6JyP1qzX+w3dZugwsR+xAr49u1/9qdWb0vI6EdCOzjpA1FB6LVdrt1TW0oRRu6jVr47ACDDXJLlAzL5xQZ7Z20Wk1LbzRJGFc6w1jvhf3Esl282yzBNltnXy/TXDgP1r6+Vlgiqz/5PVl9ZJc8M2+RTHFpX0TkYnKGErL32e/jS15nB/hG7Lv0b9Xmb14+ZXTVjyQRskrZZhVHXq2pdAL0aJIzLTljT03yIq3S04bo2t6p4nRv6Y0mCU43TI91P7FsF+82j+65Uo5V1Mh/3XSVKw30tUNAPT72r3znlfL0nNnGBPRPfHopOwMQeUwss2GfRMx3+kBSnSZZOk1SLaoerUFjB6OlbJnpIZkSYQeCWAeibbjUKbkoOSPyIk1AtLrP7cbose4nlu3i2eaayfnyh0N/kC0V1XL1ZB3721l/PHpWmtq65BOLp+HzqkP+9hf75dVTF+XaKfmO74uI3G1z9l0srCE0tLHT1YjegYmoj9WwX0vEIknO9HnTCkZE3H6sbwqnKJMzlpwRpY78EcPkXdPHGAPm/sOaOY43R9DXHTEsXZbMKpC2zm7jB+JTB84wOSPymEgyg71mGzONXYgvoOruI64eVQqaWRzdBOja5myowWftRudkSlZGWtSzBGibs7zsWApIiShZk8IfO9ssR6LoYBQJ7Uy0BVWmK1FCl52ZbjR3WDyzwJjWikNYEqVecvYY4mf44/0J4mHc3o1fc5EP5hUQhSOzjCQokjk2tev88XPNSM4i66mp9Be0jnWmbc6iYXUIIKLUsEZnTkCBmTUpvFNefKNe6pvbZd38tyaX19snz7caQ2wQUWolZ08j7C3Q9fYf3Dmc1KXJkw6LEclYZ28gMdOel9GUnFnDadREkZzpr2FWaxKllqLcbFmANmBaBekkfb3szDRZPqewb92q0hKjvZvTiSARuZ+cZeNLvi/jMG+z5GyAdmeRJGd9PTWjTM603dmZhsgHom1FCV0nskBOek6UelWbh6ob5XVUbzpB5/3V5OyG2YWSM+ytZg5j0MbtumljWLVJlILJWTNKha6x7uD2tVhEP1R9AGhJ2DlUG2jVwWA0gdNe8tohIBrFSM5qLrZF3D6EswMQpaa1ZtWjUyVaL584b/QkX4ekL5xWbR6ra46oSQYReSc5+zzil0jKdmjg9i8Qn3X3sFK9U8DgH3L6+OQxOUaj3GiMQ7VmOxr1DpX8hSdnedlsc0aUSiaMHi5XTRzlWNXmk69VGz0zV857+2Cza8pKetu44TlElDpza+7BYi7idsRnEPOwTntu0gDDaUSSnEXb3syaJUBF2imgobXTWLJDAFFqVm2+evKinDzfEtfraEm79shcMrOg3x9qRfjRd+3kfLY7I0ql5AylZXdgMQJ/4AcQr+H2SKzTJI36+bWrc1hWDjKchnZnP3a2KaqemhbtrRnNWGes1iRKXVavynhLzw6capBTF1r7rdK0V6NqG7c3HGrjRkTuV2t+CknZBesObp/XdfHt1p/S0JBsRtGIQUvO3qxvkY6unphKzvoGoo1wrDMdRkPlDec4Z0SpZirapM4tyY07OdN2a9ojc1VpcQRt3Fi1SZQqyVlayDZMNW5qQ6lh7h1SapuFErHBkjOr0W20PTVVwcgs40OWJWdEwaClXfvePC+1UQ4+ba/S1OROZx3Q2QcGMjE/R6402rhxSA2iVEnONiMeRVJ2I2Ilbj+CeNLdw0pdWiKmbcIaL/WWWoWzErcZMSRnmpgV5WZF3ObMqtbMZYcAopS07ooSJFj4EC6PrURLZxnQ2Qa0/dpQtPRsP9q4aRUoEXk/OfuCORCtdgjQ9mevhg1KSzZWdeXRuuYBk7PxqJ4cmRVbVaO2O6uJtFoTCWIu9qNJHRGlHi1hn144IubqRq3S1HoPnXVgKFabNKcHvyUid3prdmOxG3EMsQBxI+Jg9LsKVnI2UNWmro+l1OyygWgvtkZccsYBaIlSl7Yo0Y4BL7zeO/VStDTR0tkGdNaBoUzra+PGqk0izyZn+FCYjfhXhCZi/xdxwkzWViD0PvVjypgcYzyh/nps6ijdmpxpu7RYFedlR9zmTIfSYHJGlNq0RKsLnx1bK6Ir0dLZBbQHZiRVmvaqzb3HdcDa2Nq4EZH7JWeHzFKy9yEZW4L4Lm53ObNb/8pITzN+gR7tp+TsNEq8dEqlWHpq2kvOmtu7BmzT9vZJz9lTkyiVlY3Pk4n5w6Ou2rRmF7B6YkaaCPa2cauJal9ElLjk7E8R+mnwLErPvq8dAnCbjZcioCVj/U2F0tdT05xJIBbWQLSRlJ5pmzPODkDkj6rN56vOGn/T0VRp6iwDOv5ipGbjs2k6flyyapPIo8kZSso2IT5kzg6wDfG3iGJ8UNyLWJ2oA0xF2qbsRH2LXEIpmZ1VmmbNJBCLElRrqkh6bGqbM84OQJT6tGpSx0d85mBtRM/XWQV0doFoqjStRFBL2nYfq5fzMbRxI6LEdQhoRjyMuAl3JyJeQXzRmd37t4cVmogYkwnbVdY0ScHIYYOON+TkQLTsEEDkD++YNBrtTbMinmLJ6nFpzTIQWxs3Vm0SeXkojT5I0OoR9yN0vDMaqsdm3eVVm3p/RhylZqoIH9CRVGt2dHVLC9qmseSMyB+zj6wtK5HnjtTh77p3ztyhkjPteamzDERr/gSrjRt7bRKlRHJGkdEOATq0WFVN42UjdVfifjztzVRWRrqMRcnbUNWa1tRNTM6I/EGrKC91dMu2w3WDPk9nE9BZBQabS3PIqk0kgjujbONGRM5hcuaC7Mx0mTwm57KSs7qmNnzQdcbV3iyagWh1X4rzahL5w8JpY4wfZkP12tTZBLTHpc4uECvdNpo2bkTkLCZnLpmpPTbRxsxSZd6eVRz7GGf2TgFDlZxZUzex5IzIH3Smj9VlxUiYat7W2chOkzedVSCW+Xst75iUH1UbNyJyFpMzF9udvXGu2Wj7paxStHjGOLOXnFUPMUuAlZxxKA0if1Vt6jiHOyvP9vu4ziKgswloRwCtnoynjduaKNq4EZGzmJy5RH+1arXA8XMtxn0tRcvNzjAmLo+X9tg839Ix6K9ntjkj8p/rp4/FD66MAas2dRYB7WkZa3szOx1SQ9u4PTdEGzcich6TswTNsalLXRfPr1n7FE5qsHZnrNYk8p9hGWny7tJi+QOqNq1SeTtN2rSnpc4qEK+FU8fImAjauBGR85icucSa3LzKnGNTZweIpw2I3ThzloDB2p1Zvaw4tyaRv2ipmP742nX03GXrdZ3OIhBvlaZ9KrrVSASfOVQrbZ2cuY/IN8kZPiDWIg4jqhBvG7gW6/IRmxCvIl5EzLc99kNELeKAm8folpFZGTIe1Y9aYnahpV3ONrU50t7ManMWScmZ/srWnqNE5B9LZxXIiGHpbyvReuaQlqb1RD0rwFBVm01tnQO2cSOiFEvOkFRpVnAPYh2iFHEr1unS7kuIV3p6eq7E8qOIb9se+zFirVvHlwgzi3vn2LSqNnXOTSeTs0FLzpCcsTMAkf/oD64Vc4v62pdZnnyt2uhhqbMJOGXRjAKjrSyrNon8U3K2EFGFxOsYQidp+zniA2HP0WTtab2B5xzCYioSuGLz/nYs6l08PtfpmGZH65rkiDmMhlMlZ1oql4sYbJaAhtZOGTU8w5H9EZH3qjbPNrXLnjd6PyKbUbqlPSt18FjtaekULX1fNa/YmMqpvzZuRJR6ydkExAnb/ZPmOrv9iA/qDSRlmsxNMefv9AWdDcDo7XSkFr9202TC6N62Yk7oHU5j8GpNjnFG5E/L5xRKFhInaw5NnTWgrbPb0SpNe9Wmfp7sPnZ5GzciSs3krL+fb2+Vwfe6C6HtznQy9TsRLyOiGlQH234asVejrs5bXb6tkjL9Ratzajr5i1aTszMNg3cIYGcAIn8agZLzG2YXGslZN6o2dbBYnT1AZxFw2jLsJ6efNm5ElJrJmZaUTbLd1xKx0/YnoOqyAfFxxNVmm7NCxOvR7ATbPoBYoFFYqJt7hzVVk5aeOdVT0z5LwGAD0bLkjMjfdIqlavxA00Fnnz1Ua8weoLMIuNXGbUv55W3ciMg9bjZK2oOYhRKtaVieQtyC+LD9CXhMW662mG3SbkNs14TNxWNKKC250gEjdZ5L7dmuH2xOfXjqQLR1jW3SiXYg2uW9v+SMHQKI/Gvl3GJBzab8/S/3G7MG6A82Jz9j7NaUFsv/vnpG/unXrxozByyfUxTRfvR4th2ulfLTDcbYa5Fsl6hteHzJOX+U5OQMSVYnkq/P4uZmhPbc/CHWlWPdBvPx+7CYh/gp1ukgOhWIT1rbY90jWCxHFOC2lsJ9Bds86NbxOk0v2vX3bZeiM8fl1sO75LmGxbL+XIPcv2GZIxdvMZIz/RGrE6pb455ZcJ6M3ppsc0bkX9oxqDC9W4Ydq5T1R3bL1qYl8tqRM459xtg/yzbtrJRp9SdlzD2Pyd1lS+SR0plD7sf6DKyuqJKl5Tsj2i5R2/D4knP+KHIh/SL3iwULFvTs3bs32YdhePpgjdx9z+Oy6d71ktndJR1p6XLz7Q/IxjveKzfOMzqkxkXHNPrEj/fKrz+zSK6ZnH/ZY41ob3bFV7fIl98zTz61bHrc+yIi79HPmG9+9/fy+/s3uPIZM9hn2Xs+fa9Muu4qmZSfYz3tbU6cb5GTu/fL/37/9oi3S9Q2PD53zp8b15/fofBpnzbLCl/PsRZcosW8+mtCL1qly2XlO6Ti9FJHLtySvN7Sspp+emxqNarK41AaRL7+jFlx8HnXPmMG+yxbefCP8pOCiWiPdmHA7XTu378+9MeotkvUNjw+d86fG9dfUDE5c4nWv2sx78btD/X9qthetlQ2OjDn3VAD0V5s6Z26idWaRP7l9mfMYPvZOX+p3PPhawb9EjZK3M6flo4dkW+XqG14fO6cPzeuv6BicuYSbRip9e9azKu/JvSiHVc6w1jvhPycTGOASO2tNdCk5+wQQORfbn/GxLufWLZL1DY8PufO3/s33I/tdsq20sUyyYXrL6jY5sxFVk+WClQLlLrQk+WGbz4rV00cLd+59R2Xrd9cXi3rH9onj9+5ROZPGOXY/ojIW9z+jIl3P7Fsl6hteHzOnD9t//z5X7wis4tz5bENi9gZwKE2Z0zOUthf3L/LGNb30Q3XX7b+0b0n5B8fe1V2/OMKmTRm4AadRERE8frKbw/IL/C989K/rJKcYayQcyI5c3MQWnKZjnV2puHtA9HqMBqKMwQQEZHbdNowHWxdpxEjZzA5S2HaKaDmYpsxrll4cqaD3urk6ERERG7SacN0+jBO8eUcJmcpTEcEb+/qlvpmnWDh8g4Bmpg5OZcnERFRf7R92qrSYnnmYI0xxAbFj8lZildr9jecho5zNionMxmHREREAbR2fokxjdjOyrPJPhRfYHKWwkrMaZtqwobT4KTnRESUSItmFEhudgarNh3C5CzFqzX7LTnjpOdERJRAOu7mqnnF8gdUbXaguQ3Fh8lZCivMzTLq+qvDkjOWnBERUTKqNvX7Z9fRczz5cWJylsI0MStCghY+S0DDpQ7ODkBERAm1bHah5AxLZ9WmA5icpbhiVG32W3LGDgFERJRA2ZnpsmJukWytqDZmD6DYMTnzw0C0F98aiLats8sYDJCTnhMRUaKtQ9Xm2aZ22fNGPU9+HJic+WEg2oa2vvsNrZ3GMi+bA9ASEVFirZhTJFkZafLUgWqe+jgwOfNBj82mtk5pRDszq0pTceomIiJKtBFZGUbbM03Oulm1GTMmZz4oOVNWuzPtDKCYnBERUbKqNrWj2isnL/A/IEZMzlLcOHMgWqvHplVyxjZnRESUDDfOK5bM9BCrNuPA5MxnA9HqALSKyRkRESWDfv/ojAFPHjgjPT3stRkLJmcprigv6/JqTavNWTbn1iQiouRVbZ6ob5Xy0w38L4gBkzMfjCszdsSwt1Vr5g1nb00iIkqOVaXFkhYSVm3GiMmZzwaibbjUiYQtTbIy0pN8VEREFFRjR2bJddPGGlWbFD0mZ74ZiNYsOWvpYHszIiJKunVXlMjRumaprGlM9qGkHCZnvhmI9q1qTXYGICKiZFtTVmIsn+SAtFFjcuaTHpv1ze1yqaOLk54TEZFnmtxcOyWfyVkMmJz5aCBaLT1jyRkREXmp1+bBMw1y/Fxzsg8lpTA589NAtGh3pjMEcHYAIiLyAlZtxobJmQ+UjDLHOtOSM3YIICIij5g0JkeumDCKVZtRYnLmAyVmydmpC63S2NbJkjMiIvKMtaja3H/igpzGdxRFhsmZD4zMypBcRFVtk+hMGXnZHICWiIi80+5MPcVemxFjcuYTxaOy5XB171gyHEqDiIi8YnrhSJlTnMvkLApMznw0EK2WnCl2CCAiIq9Vbe45Xi+1jb1jctLgmJz5aKyzts5u4zZLzoiIyGuzBWizmy3lNck+lJTA5MxnY50pJmdEROQlWq05rWAEqzYjxOTMh8kZqzWJiMhLQqGQUbW569g5Od/cnuzD8TwmZz5qc2ZhyRkREXmx12ZXd49sPciqzaEwOfPRHGYqPS0kI4alJ/loiIiILqeD0U4YPZxVmxFgcuYTRbm9yVlmekieOVRr/DohIiLyUtXm6rJi2X6kTr65+ZA8jRI0N7+ruvDauo/vPF0Z8b5i2cYNHK3UB/Ti+eJDu2Va/UlZfWS33H3+tDxSOlPu37DMKEkjIiLywndVeeUZmXT2hHTd9Uu5u2yJa99VXdjX+vu2S3VFlSwt3xnRvmLZxi1Mznxg2+FaqcHFtOXBOySzu0s6djwkN9/+ANbPlRvnFSf78IiIiIzvqpZjx9/6rtru3nfVNuxLk6xN96439rUR+7pp/X3ytd+PlXnj8vrd5uCZBjl5oFJ+f/+Gvm2S9V3K5MwHyk83GFm+XkxKl8vKd0jF6aVMzoiIKHDfVeX97Gt5xfNyf964Qbdbj+d44buUyZkPlI3PM4pfNcs3fo2kpcv2sqWyEeuJiIiC9l1Vhtf81rzFl+1rx/wl8q0/u0qWzCrod5udlWflR43VRu1Tsr9LmZz5wPI5RUa9uBa/apavF9O40hnGeiIiIq99Vy05sEOenrtIprr0XbUcr/m1onGy+pPfk7WVu4zvxfHY159cM2HA9mP62FMveuO7NNSj8yn4xIIFC3r27t2b7MNICm3IqHXsFSjKLUWWrxcTOwMQEZEXv6vu3nrEmA/6pX9dJTnDnC8n6ujqlmv/bauUTciTRdMLIv5eTPR3aSgU2oc8bEH4epac+YRePFonzg4ARETk9e+qNCw//qM9suvoOVe+t3bhdRsudconFk+XVaXFKfddynHOiIiIKKEWzyiQ3OwMefJAtSuv/yReVwdkXzpA+zKvY3JGRERECTUsI03ejdKprRU1RhWkk7p0iqiKalkxt0iyM1NzxhwmZ0RERJRwOhH6xdYO2X3snKOvu+eNejnb1C7r5g8+bIaXMTkjIiKihLthdqHkoOpRqyCd9BReLwslc8vnFDr6ur5JzkKh0FrEYUQV4ov9PJ6P2IR4FfEiYn6k2xIREVHq0irHFXOKZEt5tWNzWHbjdTQ508RvRFbq9nl0LTlDQqUVvfcg1iFKEbdinS7tvoR4paen50osP4r4dhTbEhERUYpXbWoV5F5URTrhlZMXpLrhkqy7osSR1/NjydlCRBUSr2OIdtz+OeIDYc/RhOtpvYHnHMJiKpKw4gi3JSIiohS2Ym6R0TnAqarNp/A6mekhWTk3teeVdjM5m4A4Ybt/0lxntx/xQb2BpEwTsimIiRFuS0RERClsJKoel80qlM2o2tQqyXj09PQgyTsji2YUyKjhmQ4dof+Ss/6G1A0/83chtN3ZK1jeiXgZ0Rnhtr07CYU+jdirUVdXF8/xEhERUYKtQ9XmmYuXZD+qJOOd7PxEfavxeqnOzeRMS7sm2e5ridjpsCy3AfFxxNVmmzPtWvF6JNvaXuMBxAKNwsLU7ZlBREQURDreWUZayKiSjMdT2F5nWopmRoAgJmd7ELNQojUNMQy3b0H8zv4ErB9tPqZuQ2zXhC2SbYmIiCj1jcrJlEUzC4x2Z1o1GSut0rxu2lgZOzIr1pfwf3KGE6zVk59FbEYcRDyKdeVItjZomE+bh9B1h8yemZ8bbFu3jpWIiIiSR6si36xvkYozWj4TvcqaRjla15zyvTQtrg4CgoTqCSyeCFt3n+32LixmRbotERER+c9qVEV+edNrshmlZ2XjR0W9vdXbc02ZP5IzzhBARERESaVVkQunjYl5SI0nsd21U/KlOC/b4SNLDiZnRERElHQ6F2ZlbZNUIaJx/FyzHER1qB96aVqYnBEREVHSWVWSTx04E3WpmX17P2ByRkRERElXMipbrpk8OuqqTX3+FRNGyaQxOS4dWeIxOSMiIiLPVG2Wn26QN8+1RPT80xdaZf+JC8YcnX7C5IyIiIg8wUqynio/E/HAs8pP7c0UkzMiIiLyBK2anD8hL+KqTU3O5hTnyvTCkS4fWWIxOSMiIiJPVW2+/OYFOXOxddDn1TZekj3H631XpamYnBEREZFnrDWTLR2QdjBbymtEZ3vyy6wAdkzOiIiIyDNmoIpydvHIIas2tUpzWsEIo1rTb5icERERkaesRdXmnjfq5WxTW7+Pn29ul13HzhmlbKFQKMFH5z4mZ0REROQp65B0dff0Vl32Z+vBGunCE/R5fsTkjIiIiDxlbkmuTB2bg6rNMwNWaU4YPdwYfNaPmJwRERGRp4RQValVm7uOnpOLLR2XPdZ4qUN2Vp71bZWmYnJGREREnrMOyVcnqi61CtPumUO10t7V7dsqTcXkjIiIiDznyomjjKrL8InQn3ytWopys+SayflJOjL3MTkjIiIizwmhynJNWYlsRxVmU1unsa6lvVO2Hak11qel+bNKUzE5IyIiIk9ad0WJtHd2G1WZ6rnDdXKpw99VmorJGREREXnStai6LEQVplW1qQPT5udkysJpY5J8ZO5ickZERESelIaqyzVlxfLsoTqj16aWoK0uLZGMdH+nL/5+d0RERJTyE6G3dnTJ/3nioNH2bK0P59IMx+SMiIiIPOs6VGGOHp4hv9h7QoZlpEk72pzp7AB+lpHsAyAiIiIarNfmqJ5Oya8/KWsqd8t3L5yWX5bOlPs3LJN0n/bYZHJGREREnrXtcK3k1J6R3z14h2R2d0nH9ofk5tsfwPq5cuO84mQfnitYrUlERESeVX66QZYffN5IzJQul5XvkAqs9ysmZ0RERORZZePzZEfZEulISzfu63J72VIpxXq/YrUmERERedbyOUXyCNqYaVWmlphpYjaudIax3q+YnBEREZFnpaeFjMb/2sas4vRS2YgSM03MdL1fMTkjIiIiT0tHIqaN//3aASAc25wREREReQiTMyIiIiIPYXJGRERE5CFMzoiIiIg8hMkZERERkYcwOSMiIiLyECZnRERERB4S6unpSfYxODlzfR0Wx6PYpABx1qXDSSU8DzwXvC74N8LPCn5u8jsk8d+nU5CHFfo6OYshmduL978g2ceRbDwPPBe8Lvg3ws8Kfm7yO8Q736es1iQiIiLyECZnRERERB4S9OTsgWQfgEfwPPBc8Lrg3wg/K/i5ye8Qj3yfBrrNGREREZHXBL3kjIiIiMhT0gLam2It4jCiCvHFZB9PMuH9v4F4DfGK9jJJ9vEkEt7vDxG1iAO2dWMQWxGV5jI/mceY5HPxVcQp89rQeE8yjzER8B4nIZ5FHESUIz4X1OtikHMRqOsC7y8b8SJiv3kevhbga2KgcxGoa8IO7zUd8TLicSevi8BVa+qJxOIIYhXiJGIP4lach4qkHlgSkzMsFuD9nw3ge1+GRRPip3j/881138CiHvfvMhP3fNz+QjKPM4nn4qu6Dve/ldSDSyC853FYjMN7fgm3c3F7H+JmxMeCdl0Mci7+IkjXBd57CIsReL9NuJmJ2zsRmqh+MIDXxEDnYm2Qrgk7nIeNWOjQGXl4/zc59R0SxJKzhYgqnKxjiHbc/jniA0k+JkoC/P9vx6I+bLVeCz8xb//E/DIK6rkIHJyHM5qMmLcbsTiImBDE62KQcxEoeO9Kf7goTUg0egJ6TQx0LgIpFApNxOK9iB/YVjtyXQQxOdMPlxO2+yeD+IFjo39YW3CR7UN8OtkH4wHF+qWkN8xlUZKPJ9k+i+viVbPa0/fVNnZ4v1OxeAfihaBfF2HnInDXhVl19Qpu1iK24hoI7DUxwLkI3DVh+m/EPyK6zfvi1HURxORMi2XDBTbzh8W4gK7Bch3iDrN6i0jdi5iBuBqhHzL/GZTTgr+DkVj8CvF5/H00JPt4PHYuAndd4H13IfT9aknJQpwTo+o/iAY4F4G7JkKh0E1Y1OJcaHW/44KYnGlJ2STbfb3ATifpWJIOF5bx3rHUX0GbzGrfIKsx29pYbW70vAQSroka84NYfxV+PyjXBv7fM81k5GG8918H+bro71wE9bpQeM8XsNhmtrEK5DXR37kI6DWxGPF+s922No9aids/c+q6CGJyph0AZuGkTUMMw+1bEL9L8jElBd7/CLOhr3Ebi9WIvt56AaXXwl+bt3X52yQeS1JZHzCmPwnCtYH3rCXrDyIO4ovmv4J8XQx0LoJ2XeD9FiJGm7eHY/FuxKGAXhP9nougXRMKfxP/hJiImGrmEc/g9kecui4yYtkoleHkdeJC+ixubkZoz80fYl15kg8rWYoRm3o/g41r4X9wLp5K7iElDt73I1gsRxTgtpaofgVxF+JR3P8klm8i/jyJh5jsc7Ect682q/311+H6JB5iIn8N/xXCGF7GXPelgF4XA52LWwN2XWji8RO853SzQONRfE4+jvu7AnhNDHQuHgrYNTEYRz4rAjeUBhEREZGXBbFak4iIiMizmJwREREReQiTMyIiIiIPYXJGRERE5CFMzoiIiIg8hMkZERERkYcwOSMiglAoNB7x2FAnA8+xJn4OX/9jxJ/xZBIRkzMiIoemMkMkJblCUhe4AcGJaGAsOSOilBEKhaYiDiK+jyhHbDGnkenvudsQ/4F4EXEEsdRcn474JmIP4lXEettrG9POYJmDeNR8/BeIFxALbK/9dcR+xG6EzrRheTfu7zD3d5P53GzEjxA60v7LiBXm+o8hfon4Pe7q+xiH2I54RY/DOl4iCh4mZ0SUamYh7kEpVxmWOvnynw7y3Aw8Tydh/rw5JZXSaVUuYv07sdT4FBKhaWHbfQZxHs+5Est/Q1xre0znod2Nx67Ccrtub3tM59m7AfFexH2amGF5hz6A51+Bxa3m9De6Xl2P+Gs8thLLDyM247ZOg6OvbU2ZREQBw6J0Iko1ryOBsRKXfWZCNJBf9/O81YgrkSBZVZijzITviG27JYhv6w3sS0uxXrU91o543Pa6q2yP6VyD3VhWYptjWM41X+u75mvpJNHHcXO2+fytWFdv3t6D+CEez8TyN7b3SEQBw5IzIko1bbbbXUP8yGzr53khxJ1aQmXGNMSWsO30OQPp6HlrUuLw/YdPVtwzxGs19z2xp0dL4ZYhTiF0IumPDrIdEfkYkzMiCprNiNvNEipt+zUboVWVdjsRf2E+XoqFVklG4s/x/DTEDNyejjiM0KTrL619YTHZXH8ZPDYFi1okad/H8kHENdG+MSLyB1ZrElHQ/MCs4nwJCZGWatUhbg57zvfMtmFanfkyQpcXI3htTbqeQ2gngQ1ItC7hNb5ntj97DctOxMewvq1315dZjvgHrO/AUofrYMkZUUCF3iqdJyIiq0cnFplmcqWlYE8jZuO+tjcjInIVS86IiN4uB/GsWfWpRVy3MzEjokRhyRkRpTQkUPdgsThs9beRTP0oGcdDRBQvJmdEREREHsLemkREREQewuSMiIiIyEOYnBERERF5CJMzIiIiIg9hckZERETkIf8P4fxsh1Vd/vMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(range(1, 40), scores,marker='o', markerfacecolor='red', markersize=5)\n",
    "plt.xlabel('n_neighbors')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.title('Model response to number of neighbors');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring many hyperparameters\n",
    "\n",
    "While `n_neighbors` is the most obvious hyperparameter, others can significantly affect the accuracy (or other metrics) as well.  All scikit-learn models have default values for their hyperparameters, but how good those choices are is very domain specific.\n",
    "\n",
    "Two additional hyperparameters of interest to KNN are `weight` and `metric`.  There are a few other hyperparameters, but those are used for performance considerations not for fundamental behavior of trained models.\n",
    "\n",
    "By default, `weight` is `\"uniform\"` meaning that it simply gives one \"vote\" to each of the closest neighbors.  But `\"distance\"` is quite plausible; it weights each such neighbor by the inverse of distance (with cutoffs for only K neighbors considered nonetheless).\n",
    "\n",
    "By default, `metric` is `\"minkowski\"` which is a generalization of Pythagorean distance to higher dimensions.  But `\"manhattan\"` distance is also often useful; it measure the \"city blocks\" to get from point to point (i.e. the sum of the distance in each direction).  Other are available and occasionally better choices.\n",
    "\n",
    "| identifier | distance function\n",
    "|------------|----------------------\n",
    "| euclidean  | $$ \\sqrt{\\vphantom{\\int}}{\\sum (x-y)^2} $$ \n",
    "| manhattan  | $$ \\sum{|x-y|} $$\n",
    "| chebyshev  | $$ \\max {\\Big\\{x-y\\Big\\}} $$\n",
    "| minkowski  | $$ \\sum{\\big(|x-y|^p\\big)^{1/p}} $$\n",
    "| wminkowski | $$ \\Big({\\sum |w \\cdot (x-y)|^p}\\Big)^{1/p} $$\n",
    "| seuclidean | $$ \\sqrt{\\vphantom{\\int}}{\\sum \\frac{(x-y)^2}{V}} $$\n",
    "| mahalanobis| $$ \\sqrt{(x-y)^{'} \\cdot V^{-1} \\cdot (x-y)} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us try combining a couple of these hyperparameters in the same search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "metrics = ['minkowski', 'manhattan', 'euclidean', 'chebyshev']\n",
    "K = range(1, 18, 2)\n",
    "scores = np.empty((len(metrics),len(K)))\n",
    "\n",
    "for x, k in enumerate(K):\n",
    "    for y, metric in enumerate(metrics):\n",
    "        knn = KNeighborsClassifier(n_neighbors=k, metric=metric)\n",
    "        knn.fit(X_train, y_train)\n",
    "        score = knn.score(X_test, y_test)\n",
    "        scores[y, x] = score "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgoAAAD1CAYAAAAxmt2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABAxElEQVR4nO2dB3hVVdb+100PqYQ0QkkoAwhEwAELg9IEVCw4ilLUsY2jYpnvr4KOOuCo3+iMzFhGhk8HC5ZPRpoIoygqWNAPUBBCh4SQQHohDUi7/3ed7CT3xgS4IeQm1/fH8z6n7H3O2fvsc89ee+11gs1utwshhBBCSGN4NbaTEEIIIYSGAiGEEEJOCD0KhBBCCKGhQAghhBDXoUeBEEIIITQUCCGEEOI69CgQ0gQ2my0BskM+J7tJyHMz9DVvZttsH0JI86GhQDwCdBYHoHIossH+LaYzSXBX2X5O4D7Pgd4+jeMfgpKgYihFt1uyfKTRe/4G9BTvDWkKGgrEk0iBptZu4OWXiEWg+4rTOBwBn/j2QDdBHaFLoHtwv6ac+VZpO+3Z3p6P9lZe4jo0FIgn8ZbpZGr5DbSwwUstDFoI5UCp0GOQ9TvA0ht6DsqFkrFrYiPHLoAyoEM6CtNjXHCR3wYdxK7Pzf5boZ1QAbQaijf7lb9D2dARaCs00GH0Nx/61Iy619UeZ9KHQxvNcboc7pC2FnoS+sYc+0mtBwbLAPUEQHlQoTk2xpV6Y5927H+Arsd6CfSj2R8HrYDyoX3Qb5u6V3a7/S/QD1AltBu7PoB+dZJbPF3vq2m3R801Y6EyqJND+X5p2t3XTBXpfXjJ3Ktd0FiHvE3W2eFYbaN87JpzCue7xbS13vdk6HcOaaOgdGgWlIldr2PZEVppyltg1rs2aEst03pzrz/UukLvQEWm/eq8aFjvZ54ZbYPd0HVm/x16/6CZtedxaLMl5vrq2bnP4Vxa38XmeSnCLq37udAmc+0s6G8naTPSntA/4UzxHrT3ZwAcgC6GtHM5C9KXehqknahmSDD5FprOJ0T3QXug20zandAuqBsUAX1hjvUx6cuh/4GCoGhoA/Q7k3Yz9HUTZdPr2M21g4yXYxK0z5RVR2SPQetN/gnQ91C4GWFrns4m7Q2oGLoI8odeqL2uKXMBdKM551Sz3cmkr4X2Q31MGXT7GZOmHZd2Eh3MvfslFHqyejdS1znQ2w32rYPmQQHQYCgHGnsKbap136ztcpL7+qqpzyDouN4vk/4f6C6H/H+HXnJor0rovyBf6HroiN7DU2xrPfZec58DT+F8anT2MnUaCZVB55i0UebYZ02b6vnUwLnGtIc+q+9rmRzqstY8P3rOMGiHeZYvNmXSZ+11kzfI/BZuMWnnQLnQAIdn6imHc3uZ5++PkB/UE1LDeYJDG1eYZ9jLlPdb6EaTHgyd7+53AmVvufcrb2bL3UzKfffAwVDQDvfPkI5uPzUvRstQMB2gdiT9HY7TDnKtWf/csVMC482xeo4Yc2ygQ7p2xF+4YCj0dNj3Ua2B4vByLjOGzRjz0j9f9zc4l77U33PY1pdylTFu1EDY0CC/vsBvduhcHnNIuxv62KzfCq2Hzm5w/AnrfTJDwZRLyxfisE/b541TaNMnIPVK+J/kvnZ12Kcd+hSzrp31N2Zd215H6+c6tNdhyNbg2BtPsa0PNihLk+drouxqiNzvYCiUQwEnuBdqYBU4bGtbPuqwPVefKYftK6AtDvfhqwbnUyNodhOGwnmN1O8RB8ND2/jLBulfmvaKbI3fO2Vv1XvAuSXiidMP+tLq0XDaAUSaEVKqwz5d72LW48zIyzGtlngzUlRXdO0+rwb5T0Zag/O9gHPpC74WPXEX/DA/x/5/YP1lqDvWl2H5IPYXNTwP9pUY97eWPa5BmRvWT0xnWUuZMTRq75t26u/hfOrJ0IDER1ug3lqmfJSzuEGZhp7oIFzrHjONdCGO1U77RDRVJ/Uc6TRNT+NFOYJzaeddyyG78/+Kl2rKeyp1bqz+TZ1P63MpFrNNObyMp2CbQ94cHHqsdgP5OxgPyCUmXkMJ0ekP5FPDS8lyOP5oI9u190Hrc55OKTmk+5g2bwzNH9cgvxpaX52g/rdBf4J0ykVjhZ5AOVc2cX7SzqChQDwKvJxSzYvqMvPyciTXuEzjjatW6Q4dMusZprMUhzTHF+NxM2KqbG7xGpzvaZzrnUYz2u0vYvEi6qJu739DGv3/uEmuKyPSg82Ug45mVXXxCg51+PikBbPbK8yI8Akzt/0fM43zHxfr3fC/o9UyReCcIQ7GguM9/wnIq96Nh6GLcEz6KVyz8YKg48W5/m3m4Ps10jF2Qbp6AOwO5Vpxim3d2H+72+j5sEunE5YYw+cDvdfYpx4F2wnO9wDUFzoP+TORf7CZhnE85lTR+qzDecadYl00fwry/+IE53Q6Bnn3YjEV5VQj6NeQxjDolFdpM8pL2hgMZiSeiBoIYxq+pMxITDuOp7XjgrRT/X9Q7ed8mnafBo1pMJnprGqPVSPiE2gu0kL1hQj1gnS+uTnMhx7B8QMcgucmm/VhkI4AdVSrddCRZu0oUrkMaSMg9Y48Cf0fypdmOvU+2D8N8oHU5dwfOunIDnlHQ4kmYE89F2o4VDWj3jqq1eBN691iyqVTGn82AZNnm/Zp1EBCunbq/w2Nw7E6L366LDTTAldCDT/bjDbt7WvuvcaC/Oc02rrR80HaTv4mNqPSeBd0WutEhBivgAaWRhhvRHNZaZ6LG03ZfM0zpuWrbTP1utSiXpciE1wZqM8ENFCPaeoCSLsBisK9q9Yym92Ozyxpx9BQIB4HXlb7oU1NJN9rOl/thPQPJL0LvWbSNChutZkX/wFa2uDYm8xLf4cJElwMdW5mGZeZ4DV19WvHnARpB6KEmrIUGPd1HvScw+Hvmo4j3wQdTjfn1HyXm9Gors/UbexXT8rJiDX10bLsNAGIbzej3hp0p+jXE3oPa+f3E4x3YZmZG9f4kcZ4ygTyadS+Tquo1KhqFrjON1ho56VfUmgciyP/B+moWe/P09C15h42t60bPZ/xpNxnDFE91zTjuTgRz5sgQT3Xd6fiFWoKc301TKaYNsh0CJxUFkD9dapBPR3GoNYYB/VipJgy/MsETTaFTpFs1/YyAbZTHKdSSPvGCrwhhLQP8CLWwLN0/G41aJOc2j3TINV3cc/+5bBPvQy3Y9+IFmqXFj0fIW0JxigQQjwW4y7XzwGvcndZCGmvcOqBEOKpRsKbWKyBft/gqwtCiCu/JU49EEIIIaQp6FEghBBCSJMwRsFg/ua5SrwCfH8Z0K3uT8QTN2Iv8IxHtEr/eLEH4BPY3D8hQYjn0smv/f+5iPxDx6SkoLzRv9PBqYdGCOrT2T7gRQ1iJu6mYnmUu4vQIhT284yvi6L6ncqXloT8vJgev9HdRTht5k7eIAeTiho1FDj1QAghhJAmoaFACCGEEBoKhBBCCHEdehQIIYQQQkOBEEIIIa5DjwIhhBBCaCgQQgghxHXoUSCEEEJIk9BQIIQQQggNBUIIIYS4Dj0KhBBCCKGhQAghhBDXoUeBEEIIITQUCCGEEOI69CgQQgghhIYCIYQQQlyHHgVCCCGE0FAghBBCiOvQo0AIIYSQJvFpMoU0i/Mj+8gDZ10OC8xLPkjfKAtT1jmlh/gEyOOJ10qXDhFSXlUpTyYtkeSSLIkOCJM5iZOlk3+I2O12WZa+QRalrndbK3hKPYb3T5CHrhslXl5esvybbfL66o1O6SEd/GXOTROka2SYlFdWyZyFq2X/4by6dC+bTd55ZLpkF5bI/fOWt3bxLS6KT5DZI0ejDjZZlJQk8zdtcEoP9feXv4ybIPFh4XIcbTHz09WyJy9P/Ly95d+Tr7eW3qj/R3v3yvPf8Zk6HTzld8F6tJ32iA8aJiNj7habzUu2F34km/Lec0r39wqWi+MelHDfOKm0l8uajOck7/gBCfaJkvFxsyTIp6PY8S+pYJVsKVjWPgwFm812JRb9cfOfOUGeUVg8iDyXt/T1Ha4xB4sSXOO5Bvv/hMWX2L+mpa/pJTaZ2f9KuWfjAsk+ViRvXjBDvsreKSml2XV5bu41WvYUZcjMzW/jAYmy8s9A/ip7tbyw+z+yu+iwdPD2k4XD75UNufucjm0tPKYe6OQfnjpG7nphiWQVFFsd/rqt+yU5I78uz22XnCe707LlgfkrJCGmI/KPlTufX1yXPm3MEEnJzJegAL9WL39tHf40eqzcuHSxZJYUywdTp8ua5H2yL7++DjOGnSc7cnLkzpUrpGfHCOQfIzcgf3lVlUxb8r6UVVSIDwyF96+bImsPpMiWzIzWr4cHPFOeUAfWo221hw1P1ajYe2XZwVlSUpEjU3q8LMnF6yW//GBdnmGR0yT32H5ZlT5HOvp1k9HIv/TgTKmWKjx/8yXn2D7x9QqUqQn/lIOl3zsd22anHtABrziRkeBuULY/ngkjQRkQ3k3Sy/Lk8NECWH5V8knmj3JRzFlOeXoERcvGvH3WemppjnQO7CgRfsGwEIuth1YpqyqXlJJsiQoIPRPFPCmeUo+BCbGSll0oh3KPSGVVtazeuEtGnd3LKU/PzhGyYVfND+tAVoHEdQqViJAO1nZ0eLCMSOwpy+CJcBeDYmMl9UihpBUdkYrqavlwz24Z16u3U57enTrJ+rSaOiQX5EvX0DCJ7FBTBzUSFDUUVPgFtGr5PemZ8oQ6KKxH22mPmMC+cqT8sBRVZKDjr4SRuVZ6hvzKKU+Ef7yklW621gvK0yTENxbGTbiUVeZbRoJSUX3UMhCCfSPPSDldMhQwGk+AdkH/gpKgd6CLoW+gvdC50M3QP0z+N6AXofVQMnRtI+ccBm2GekJjzfo26DXI35xzqcl7FXQU8oMC9Jxm/33QDmgr5Oy3qUn/LfQRFGjK9JNytARR/qGSdfRI3baOOqL8w5zy7C3OgEU40FrvH9ZVYgPC4ZZ0fkA7B4ZL39A4uKHSzkQxT4qn1CO6Y7DlSaglC9MHUR1DnPLsSc+RsUN+Ya0PgGHROSJUYnCcolMWLyz9UqrhmnQXsUHBklFcX4dMrOs+R3bCmzChd43xMCgmVrqEhkpscHCdR2LV9Btl0x13ydcHU+FNyGy9wnvYM+UJdVBYj7bTHsE+kVJcWe/FUK9CsE8npzy58Cb0ChlhrccE9JVQ3xgYBFFOeUKwLzqgt2Qe3XVGytkcj4K+kV6Azob6QdMgrcWD0B8ayd/ZpOs0g5OnAR32cCzmQ1dBat69AV2PEX+imRa5C/oBGmIOuRBKgoZB50H/Z/Y/rHlwnJbpzgbXuAeLK6BJSD/aVKWQ7w5ok6rySNnJ70Jj52h0r3MnszB5nTWP+TZcXdd1Hy578GJRN1gtgXCDPTP4BvnbrpVSWnW8WeU4XTylHo3SoNPXmAWNU3jv0RtkyqjB1jREFbwPFyb2kPziMtl5sPVdw47gefzJvoZmi8YshPkHWAbBbwYPke3ZqEN1TS41cia+85ZcsOAVy4joA++DO/CEZ8oT6qCwHm2pPWwn/X1rzEKAd7BM6zFfBkVMsrwI1fBo1eJrC5CJXWbLuqx5Ul7dvL7rTMQopKDD3WZeYtux+AzbdvUCYD2hkfzLkawtoyP+GIf96rN7BRqP9MNIG2TOvcekvwnNwPbzSNsHaf5zob9BF0He0Fcm71ZIvRsabeYYcXYjlG6MhBofbBMgXcuikqA+nZs1hMw+XgRXUv0IQ0cSOdjniD6MGuBUy/KRM+VwWYG17o1glmeHTJfVGVtkbZbeWvfgMfUoKIF3oN6DEIOphBx4FRwpPVaOAMZP6rZXPX2bHMorkgnD+slITFOMGNhD/Hx8JCjQT5665VJ57PWPWq38SgbiEjqH1NchFutZpc51KCkvtwIYa/nq1tutqQpHio8fl+/S02VkfA8r0LG18YRnyhPqoLAebac9SipzYFhG122rp6C00vn3qZ3/pwhgrOWWXm9jqqLGM+iFbnBi1zmYRvlM9hd/fcbK2RyPgqPZVe2wXd2E4XG8CfNJI6qOOXgLfmpa1aMGwaWQdvZrjIdC9aVJnwi9DP0S+h4GQ205kozx0vUE524xdhxJl24dIiUO85I+Nm8ZHzvICnZyJBijDU1Truo6TLbkp9RZso8PvAZzZTny7oEz1+CngqfUY3tqpnSPDrfiDny8vazOf+1Wa7aqjuBAfytNuXpEovyw95BlPLy0/Gu55JFXZeKjC+ThBatk4660VjcSlK2YKkgID0fcQSgClrzkij59Zc3+/U55Qvz9rTRlysBE2QCDQI2HiMBAK03x9/aREd27y37EMLgDT3imPKEOCuvRdtoj6+huCffrgumEWHTGPtIndJQVzOiIn1eQlaYMCL9MDpVtq/McXNz5QcQmpMrm/Hrj1NM+jyyEboM+QcdeiuV3kMZA9Mbofp/xBtR+e6QGwUIV0nKQR/2nsdB2rOsbshv2f4H1r81USO0krkaA/BNagbQJ6rk4kxVSF+Nfd6yQF4feas0Nf5i+SZIRJPPrbuoIEVmatkF6BEfL7MTJcB1VWwE0T5nRx6DweLmsyznWHKe6LZV5ez6R9bm7z2SRPbsecL8/u+gLmXffNdanhR+sT5LkjDy59kKdoRJZ/NVW6RkbIU/ecomVV9OeeKveu9AWqMLUwewvPpeFV6MOeNTf354ke/PzZFpiTR3e3bZVekdEyNzxl1rTDJo2y3gXooOC5Dns90Yb6hTGqr275fOUZDfVo/0/U55QB9ajbbWHHePrtZkvyaRuz1ifR+4o/Njq+BPDaz4I3Fa4EsGM3WVC3Czrmco/niprMuZaaXGBA+Ws8HGIYUi2piWU9dmvyYFS58+nWwKbfkPqSjAjFitxjBWto4GBZntxbRqkPpKh2HePY7rJr58rBjt+Hon17ljXodqtpoN/zhgw+sH7XchzXIMQjWFxBbbVsNApglisX4l1X6x/AYUZr8Tb+tWF4+eRaiSY+Ihx5vx1ZWoMnXoY8OLNp3xfyJmjYrlz0E57pbCf+wIiW5KofrnuLgIhbY7p8c5/n6U9MnfyBjmYVGQ7bUPh5wINhbYDDYW2BQ0FQn5+hgL/hDMhhBBCmoSGAiGEEEJoKBBCCCHEdehRIIQQQggNBUIIIYS4Dj0KhBBCCKGhQAghhBDXoUeBEEIIITQUCCGEEOI69CgQQgghpEloKBBCCCGEhgIhhBBCXIceBUIIIYTQUCCEEEKI69CjQAghhBAaCoQQQghxHXoUCCGEEEJDgRBCCCGuQ48CIYQQQprEp8mUnxk2m+0OLFTSvYuPfDd4sZtLdPr0eu9OdxfhtOn9yrfuLkKLEHn+2e4uQouQe3aUu4tw2hT2s7u7CC1CVL9cdxeB/EygR8Fgt9tfgYaqojp5u7NNCCGEkDYDDQVCCCGE0FAghBBCiOvQo0AIIYQQGgqEEEIIcR16FAghhBDSJDQUCCGEEEJDgRBCCCGuQ48CIYQQQmgoEEIIIcR16FEghBBCCA0FQgghhLgOPQqEEEIIoaFACCGEENehR4EQQgghNBQIIYQQ4jr0KBBCCCGEhgIhhBBCXIceBUIIIYTQUCCEEEKI6/i4fgg5IX4Xii30Max4i/3ov0VKX3FOt4WKLezPSO4uYj8u9qJHRCr3miTs9x8tUp0n9ryJbr3RF8UnyOyRo8XLyyaLkpJk/qYNTumh/v7yl3ETJD4sXI5XVcrMT1fLnrw88fP2ln9Pvt5aent5yUd798rz3613Uy1Ehk4YLHc/f4t4eaMsCz6TRc8ud0oPDg+SBxbcLXG9YqT8WIXMvW2eHNieZqW9lfyyHC0+JtVV1VJVWSUzzn3YHVWQoef1krt/PwF1sMlHH26WRW8538/gkAB54A9XSFyXjlJeXilz//tDOZCcU5eubfjya7dLbk6RPP7QotYufh3D+yfIQ9eNQnm8ZPk32+T11Rud0kM6+MucmyZI18gwKcf9nrNwtew/nFeX7mWzyTuPTJfswhK5f55zO7YWnvK7OD+yjzxw1uVwKXvJB+kbZWHKOqf0EJ8AeTzxWunSIULKUY8nk5ZIckmWRAeEyZzEydLJP0TsdrssS98gi1JZj9MhPmiYjIy5W2w2L9le+JFsynvPKd3fK1gujntQwn3jpNJeLmsynpO84wck2CdKxsfNkiCfjmLHv6SCVbKlYNnpFKXtGQo2m60ED1qwC/lHYVGOY9ab7UlY7MH2jjNVRtfxQmc/R+wFN4tUZYqt0xKxH/sc6/vqctiC7xJ7xU6RwhkwFnoi/2zk/42VZj+6VKTsLRgSf3VXBepeyH8aPVZuXLpYMkuK5YOp02VN8j7Zl59fl2fGsPNkR06O3LlyhfTsGIH8Y+QG5C+vqpJpS96XsooK8cEL8f3rpsjaAymyJTOj9euB69/7j9tk1vgnJTc9X/6x4c/y7YpNcnBnel2eqX/4tez/MUWeuOav0q1vHPLfLjPH/aku/cExc6Qor7jVy16Ldkj3PniJzLr/HcnNLpJ/LLhdvv1qjxw8kFuXZ+pNv5L9e7PkiUfel27xneTeBy6Vmfe9XZd+9XXnWvk7BPm5owp1z9TDU8fIXS8skayCYqvDX7d1vyRn1D9Tt11ynuxOy5YH5q+QhJiOyD9W7nx+cV36tDFDJCUzX4IC3FMPj/ldiE1m9r9S7tm4QLKPFcmbF8yQr7J3Skppdl2em3uNlj1FGTJz89voyKKs/DOQv8peLS/s/o/sLjosHbz9ZOHwe2VD7j6nY1mPU8eG1hgVe68sOzhLSipyZEqPlyW5eL3klx+syzMscprkHtsvq9LnSEe/bjIa+ZcenCkYvqDd5kvOsX3i6xUoUxP+KQdLv3c69ucYo6CGwnCHbTUU+rupLI3jezaMglRIR6QVMBJWiQSMdc7j3Rvmzrc161XJ2O6KVuhUs12BEZb9SKsWuTEGxcZK6pFCSSs6IhXV1fLhnt0yrhfK7UDvTp1kfVrNA5lckC9dQ8MkskMHa1tfhoq+EFWoVKuWv5a+5/aWw/syJTMlWyorKmXtom9k+FVDnfLEn9VVNn+WZK2n7T4sMQlREh4d5o7iNkrf/nFyOL1AMg8XSmVltaxds12GX9jXKU98jyjZvCnFWk9LzZOYzmES3jHI2o6MCpHzhv/C8kS4k4EJsZKWXSiHco9IJTw0qzfuklFn93LK07NzhGzYVfNMHcgqkLhOoRIRUvNMRYcHy4jEnrIMngh34Sm/iwHh3SS9LE8OHy3ACLVKPsn8US6KOcspT4+gaNmYVzPASS3Nkc6BHSXCLxgj2WLLSFDKqsolpSRbogJCW70OiifUIyawrxwpPyxFFRno+CthnK2VniG/csoT4R8vaaU1v9+C8jQJ8Y2FkRYuZZX5lpGgVFQftQyEYN/IM1LOkxoKGLknQLugf0FJ0DvQxdA30F7oXKP10GaztN5kWN4MLYU+Nnn/0uDcT0M/Qt9BMWbfFdD/mXOt0f1aBiTdCf0X1rdAI7F+JfRXs90L+i200ZxvCWT9OrF8A3rRlCsZuralb2IdXrHo/B1GCOpV8LKqVU/lTrEFjK83LLzjao5rQ8QGBUtGcf0oOhPrus+RnRg1Tehd85IcFBMrXUJDJTY4uG7ktWr6jbLpjrvk64OpGDVltl7hHYjsEiE56fWua/UqRHYxRpkheesBGfHr86z1vsN6S0x8lER1rckDz6o8s/oxeXnjs3LZby9uvYI7EBkVKjlZRXXbOn2gnb8jyfAmjBjVz1rve1acxMSES1R0TZ67MGXx6strpLraPZ1SLdEdgy1PQi1ZmD6I6uhcjz3pOTJ2yC+s9QEwLDpHhEoMjlN0yuKFpV9KtTaKm/CU30WUf6hkHa0fkKhXIcrf2TjeW5yBketAa71/WFeJDQjHtINzR9o5MFz6hsbBXV4zVdfaeEI9gn0ipbiy3hujXoVgH+d3lHoTeoWMsNZjAvpKqG8MDIIopzwh2Bcd0Fsyj+46I+U8VY+CPvkvQOjZRN9I0yAt+YPQHyAt3UWYBhiC5R+h/3Y4djB0PZSoS3TU3cx+HfJ8h2MGYfkl9Fuz/2vofHMunayZifUDWM6H/o71wZBOqK2AHjLb+7G+FMth5nzw7cttDmXobMp7OfRMYxVEue6ANqly8qpO8bacCs4vNrvGLNjCMC2xQmwdboThoDMnLXm90wf34Cf7Gr6edW42zD/AevH9ZvAQ2Z6dLVWmM9KX+cR33pILFrxivSz7YJTlDhqphjWv6sh7zyyXEMQpzP/hrzLpnktl3+YUKx5B+a8Rj8ndQ2fJo5c9LVfePUESL3QerbQGjVThp3V46xsJQZzC/Dd+K5MmD5N9ezOlqspueRIKC0pl7273dEgnpUE9NGZB4xTee/QGmTJqsDUNUQXvw4WJPSS/uEx2Hmx997ZH/i4a3etck4XJ66w4hbcxtXBd9+GyBx2uTjvUEohph2cG3yB/27VSSquOn9HyNoVn1MN20mdKYxYCvINlWo/5MihikuVFqIYHpRZfW4BM7DJb1mXNk/LqMrfGKKTg5bTN/Fi2Y/EZtu1Y13062lcz7k1s/8LU09fhWM1rmX1I114xHlLTrRxaafJ8D40z6/DFyyLk1c5dJyNrfKonZyCOeQrLcEhN+NUOactRBn06dtR6LhqCdI06tCIPhw4KaN6wpRovZG8ttsE7VuzVDV5u9hIEMNYHxdmivoCdUD9n3hbIwPxr55D60V4s1rNKS5zylJSXW4FatXx16+2WS9aR4uPH5bv0dBkZ38MK6GptcuBBqPUOKJFdIyTvcP18slJWfFSeQwBjLRrAqFMVSl5GgbUsxCj+m+UbrKmMbV+pDdp65ODaUTGhTh6GvFzntigrK5fnnv6wbvutJfdiqqJARl08QC4Y0UfOvaC3+Pn5IEbBX2bNniTPPtH6gYDZBSXwDtQ/UzGYSsiBV8GR0mPlCGD8pG571dO3yaG8IpkwrJ+MxDTFiIE9xM/HR4IC/eSpWy6Vx17/qNXK70m/i+zjRXB514+8dYSdg32OaKepAYy1LB85Uw6X1fwevBF09+yQ6bI6Y4uszdLuwD14Qj1KKnNgyETXbaunoLTS+ZnQzv9TBDDWckuvtzFVUWP8e4m3TOw6B9Mon8n+Yh1ji1s9Co6mVrXDdrUxNp6EvkBnqz6eK6CAJo6tcjBOKtTYaGT/S9A/kKQeiN81ONeJeAO6xxz3xAnK8FMTrqWogN3knVATdwBbyRYwEVf+zDmPLaTejgq8Dk+BxiU4v2zczVa4RBPCwzG/GoogGS+5ok9fWbNfnTb1hCC6W9OUKQMTZQNefPqSjAgMtNIUf28fGdG9u+zHXK072L1xn3T5RWeJTYgWH18fGXX9r6xgRkeCwjpYacqlt4+VbV/utIyHAIxsA4NrHiFd/+W4QXIgqfVdk7t3HpYuMHBiO4eLj4+X1fl/+/Ue5zoE+1tpVh2uHCLbthy0jIfX5n8u0ya9IDde85I8/celsuX7FLcYCcr21EzpHh1uxR34eHtZnf/arYjRcSA40N9KU64ekSg/7D1kGQ8vLf9aLnnkVZn46AJ5eMEq2bgrrdWNBE/6Xew4ki7dOkRKHObrfWzeMj52kBXM6EgwRuGaplzVdZhsyU+pG3E/PvAazOnnyLsHzlzHdCp4Qj2yju6WcL8umE6IRWfsI31CR1nBjI74eQVZacqA8MvkUNm2Os/BxZ0fRGxCqmzOrzeG2vJXD2rWHTLrN7fguWo+B6hBJwcdJ5d023GSU9cz4DHQXni6wzlakSp4C54QW8fXzOeRiNiuRLBJ4NSa5KP/izvey3zVANuocj9iFx+pO9oW9nc8FefCfOsIT8NXsB8w26PnaO1awH6b/cXnsvDqazCvigjt7UmyNz9PpiXqzJPIu9u2Su+ICJk7/lLLnapps8woKjooSJ7Dfm+4adVVu2rvbvk8xblDaC30s8Z/3LtA/vzxo9bnkatf/0JSd6TL5b+rcV6t/J9PpTuCGWe9eY/l4j6ItLm3/9NKC48JkzlLH7LWvX285Yv//Vo2rd7ihjrY5R9/+1j+/Pdp1ueRq1f+KKkpOXL5pHNq6rD8B+meECmzHr/KcnEfTMmVuX+u9y60FbRszy76Qubdd431JccH65MkOSNPrr2w5pla/NVW6RkbIU/ecomVV9OeeKveu9AW8JTfhbre/7pjhbw49FYrbuLD9E2SjGC+X3fDuwcsTdsgPYKjZXbiZNSj2gr0e8qMygeFx8tlXc6x5v7Vna/M2/OJrM/dzXo0AzvG2mszX5JJ3Z6xPo/cUfix1fEnhussuci2wpUIZuwuE+JmWW2RfzxV1mTMtdLiAgfKWeHjEMOQbE1LKOuzX5MDpc6f7LYEtobznT/JUBNIuNJ4C6zgQLO9uDbNxBe8CenH259DNyJdgyDVaBiK9XvMsZr3OWyvdfw80gQYXo5tDX68Cut/Nx39d5DGHYzC/j5YX2y8GPqE6vJVSM1DPV4jBGdCqZBOiYSY89WV91Q/y9Sphw2ra0Mp2i+93tP4z/ZN7/+nj4AHcH5NZ9LeyT275muK9kxhP/cGdrYUUf3qP5El7mV6vPPfBGmPzJ28QQ4mFdma5VEwgYQDHbZvbiJNO/JaHjfp2km/4ZC/xkyqWa/rrE0nvtisf4DFB42UQ/2tDd+2jp9H6lDwn40c5+ThcOVvNxBCCCE/d9rT31EghBBCSCtDQ4EQQgghNBQIIYQQ4jr0KBBCCCGEhgIhhBBCXIceBUIIIYTQUCCEEEKI69CjQAghhBAaCoQQQghxHXoUCCGEEEJDgRBCCCGuQ48CIYQQQmgoEEIIIcR16FEghBBCCA0FQgghhLgOPQqEEEIIoaFACCGEENehR4EQQgghTeLTZMrPmJ1HO8r5W651dzFOm6h+ue4uwmmTe8cF7i5Ci1DYz+7uIrQIUf1y3F2E0ybK3QUgpJ1Bj4LBZrPdAW1SVR4pc2ebEEIIIW0GGgoGu93+CjRU5RPWwZ1tQgghhLQZaCgQQgghhIYCIYQQQlyHHgVCCCGE0FAghBBCiOvQo0AIIYQQGgqEEEIIcR16FAghhBBCQ4EQQgghrkOPAiGEEEJoKBBCCCHEdehRIIQQQggNBUIIIYS4Dj0KhBBCCKGhQAghhBDXoUeBEEIIITQUCCGEEOI69CgQQgghhIYCIYQQQlzHx/VDyIk4P7KPPHDW5XDVeMkH6RtlYco6p/QQnwB5PPFa6dIhQsqrKuXJpCWSXJIl0QFhMidxsnTyDxG73S7L0jfIotT1brvZnlKP4f0T5KHrRomXl5cs/2abvL56o1N6SAd/mXPTBOkaGSbllVUyZ+Fq2X84ry7dy2aTdx6ZLtmFJXL/vOWtXXyLi+ITZPbI0aiDTRYlJcn8TRuc0kP9/eUv4yZIfFi4HEdbzPx0tezJyxM/b2/59+TrraU36v/R3r3y/Hd8pk4HT/ldsB5tpz3ig4bJyJi7xWbzku2FH8mmvPec0v29guXiuAcl3DdOKu3lsibjOck7fkCCfaJkfNwsCfLpKHb8SypYJVsKlnmmoWCz2UrQUMFYxmHzRaxf20ietVg8iLRNrV/CU8dLbDKz/5Vyz8YFkn2sSN68YIZ8lb1TUkqz6/Lc3Gu07CnKkJmb38YDEmXln4H8VfZqeWH3f2R30WHp4O0nC4ffKxty9zkdy3q42B7o5B+eOkbuemGJZBUUWx3+uq37JTkjvy7PbZecJ7vTsuWB+SskIaYj8o+VO59fXJc+bcwQScnMl6AAv1Zvh9o6/Gn0WLlx6WLJLCmWD6ZOlzXJ+2Rffn0dZgw7T3bk5MidK1dIz44RyD9GbkD+8qoqmbbkfSmrqBAfGArvXzdF1h5IkS2ZGa1fDw/4bXhCHViPttUeNjxVo2LvlWUHZ0lJRY5M6fGyJBevl/zyg3V5hkVOk9xj+2VV+hzp6NdNRiP/0oMzpVqq8PzNl5xj+8TXK1CmJvxTDpZ+73Ssx8UowAg43JiR0J4YEN5N0svy5PDRAlh+VfJJ5o9yUcxZTnl6BEXLxrx91npqaY50DuwoEX7BsBCLrYdWKasql5SSbIkKCG31OiieUo+BCbGSll0oh3KPSGVVtazeuEtGnd3LKU/PzhGyYVfND+tAVoHEdQqViJAO1nZ0eLCMSOwpy+CJcBeDYmMl9UihpBUdkYrqavlwz24Z16u3U57enTrJ+rSaOiQX5EvX0DCJ7FBTBzUSFDUUVPiltWr5PemZ8oQ6KKxH22mPmMC+cqT8sBRVZKDjr4SRuVZ6hvzKKU+Ef7yklW621gvK0yTENxbGTbiUVeZbRoJSUX3UMhCCfSPPSDlPy1DASP8GaAO0BfofyFs9BA7p10JvmPUYaBn0o9HwBudKgJLMeiD0HrQVWoRdgQ75xkPfQj9A70PBZv8foY16DugVyGb2r4WeNeXcA114OnU+EVH+oZJ19Ejdto46ovzDnPLsLc6ARTjQWu8f1lViA8LhlnR+QDsHhkvf0Di4odLOVFFPiKfUI7pjsOVJqCUL0wdRHUOc8uxJz5GxQ35hrQ+AYdE5IlRicJyiUxYvLP1SquGadBexQcGSUVxfh0ys6z5HdsKbMKF3jfEwKCZWuoSGSmxwcJ1HYtX0G2XTHXfJ1wdT4U3IbL3Ce9gz5Ql1UFiPttMewT6RUlxZ78VQr0KwTyenPOpN6BUywlqPCegrob4xMAiinPKEYF90QG/JPLqrbRkK6HDVlL4e+hU8AYOxrIKmn+CQF6F1yDsIy3Og7SfIexdUhrxnY/k09EtzTTWXHoMuRpqeQ6ci/p855h/YNwwaaAyLyx3O54P952L5e2h2E/W5A9qkqjxSdqKqN4llmfwE505mYfI6ax7zbbi6rus+XPbgxaJusFoC4QZ7ZvAN8rddK6W06nizynG6eEo9GqVBp68xCxqn8N6jN8iUUYOtaYgqeB8uTOwh+cVlsvNg67uGHTH2rhMNzRaNWQjzD7AMgt8MHiLbs1GH6ppcauRMfOctuWDBK5YR0QfeB3fgCc+UJ9RBYT3aUnvYTvr71piFAO9gmdZjvgyKmGR5Earh0arF1xYgE7vMlnVZ86S8unl915mMURhrOnAdxYvpnE/0Vh0D3aQr6LS1lvWm+U+5yBgWmle9ClvN/vOh/tA35po6cfytSRuNfTOxVJ9rhDFEPjRpS83yeyihsQviOq9goZKgPp2bNYTMPl4EV1L9CENHEjnY54g+jBrgVMvykTPlcFmBte6NYJZnh0yX1RlbZG3WieyoM4vH1KOgBN6Beg9CDKYScuBVcKT0WDkCGD+p21719G1yKK9IJgzrJyMxTTFiYA/x8/GRoEA/eeqWS+Wx1z9qtfIrGYhL6BxSX4dYrGeVOtehpLzcCmCs5atbb7emKhwpPn5cvktPl5HxPaxAx9bGE54pT6iDwnq0nfYoqcyBYRldt62egtJK59+ndv6fIoCxllt6vY2pihrPoJd4y8SuczCN8pnsL/76jJXzdKYetKd+U70JRn2hOdjn2MkGnMb57U1c81OHa/aHboOBoNeZB12L7UQsX21w7VpTsepMBnDuOJIu3TpEShzmJX1s3jI+dpAV7ORIMEYbmqZc1XWYbMlPqbNkHx94DebKcuTdA2euwU8FT6nH9tRM6R4dbsUd+Hh7WZ3/2q3JTnmCA/2tNOXqEYnyw95DlvHw0vKv5ZJHXpWJjy6Qhxesko270lrdSFC2YqogITwccQehCFjykiv69JU1+/c75Qnx97fSlCkDE2UDDAI1HiICA600xd/bR0Z07y77EcPgDjzhmfKEOiisR9tpj6yjuyXcrwumE2LRGftIn9BRVjCjI35eQVaaMiD8MjlUtq3Oc3Bx5wcRm5Aqm/PrjdMzwel0mp9BH6CT/js652wsdRSvQ58sMy2xG7oaKnbIr1MKz2ssA5ZBOM7ZHK/nSzON8QXy6lSCTkEo30EvY19vHLsPS/UedHXwZOSamAUNiqwPXW8l1MX41x0r5MWht1pzwx+mb5JkBMn8utu5NW6NtA3SIzhaZidOhuuo2gqgecqMPgaFx8tlXc6x5jjVbanM2/OJrM/V28h6NKs94H5/dtEXMu++a6xPCz9YnyTJGXly7YU1j9Pir7ZKz9gIefKWS6y8mvbEW/XehbZAFaYOZn/xuSy8GnXACOj97UmyNz9PpiXW1OHdbVuld0SEzB1/qTXNoGmzjHchOihInsN+BA5ZUxir9u6Wz1OS3VSP9v/b8IQ6sB5tqz3sCGFcm/mSTOr2jPV55I7Cj62OPzG8ZuZ8W+FKBDN2lwlxs6xnKv94qqzJmGulxQUOlLPCxyGGIdmallDWZ78mB0qdP59uCWz6DWmzD7bZNEbhEUiHMxpePcN03M9CGhmiwYnBuMbNGsxoXPs9zcj+Luz/1uHzSJ0SWKkxBhrMiPXXzTTDFkgjte7TzyORNsacv2aohJgF7F+B/U9hfQp0wFw7VT0cjp9WmhiHTVhvdPqhFp16GPDizc2+L6TlqFjuHLTTXins576AyJYkql+uu4tASJtjerzz32dpj8ydvEEOJhX9NGjidN3w6HD1iwRVQxY3kjcLi6sa2W+FZ2OpHfxAs37UdPqNXfNzLIY1sl+DHB9rZP8oh3V9y53QSCCEEEJIG/w7CoQQQghpe9BQIIQQQggNBUIIIYS4Dj0KhBBCCKGhQAghhBDXoUeBEEIIITQUCCGEEOI69CgQQgghhIYCIYQQQlyHHgVCCCGE0FAghBBCiOvQo0AIIYQQGgqEEEIIcR16FAghhBBCQ4EQQgghrkOPAiGEEEJoKBBCCCHEdehRIIQQQggNBUIIIYS4jo/rh3gmNpvtDixU4hvcUSqWR7m5RKeP76QcdxeBkDZHzq5IdxehRYjql+vuIrQI0+M3ursI5CRw6sFgt9tfgYaqfAKCTnLbCCGEkJ8HNBQIIYQQQkOBEEIIIa5DjwIhhBBCaCgQQgghxHXoUSCEEEIIDQVCCCGEuA49CoQQQgihoUAIIYQQ16FHgRBCCCE0FAghhBDiOvQoEEIIIYSGAiGEEEJchx4FQgghhNBQIIQQQojr0KNACCGEEBoKhBBCCHEdehQIIYQQQkOBEEIIIa5DjwIhhBBCmsSnyRTSLIb3T5CHrhslXl5esvybbfL66o1O6SEd/GXOTROka2SYlFdWyZyFq2X/4by6dC+bTd55ZLpkF5bI/fOWu60Vzo/sIw+cdTksSS/5IH2jLExZ55Qe4hMgjydeK106REh5VaU8mbREkkuyJDogTOYkTpZO/iFit9tlWfoGWZS63k218Iz2uCg+QWaPHI062GRRUpLM37TBKT3U31/+Mm6CxIeFy3G0xcxPV8uevDzx8/aWf0++3lp6o/4f7d0rz3/nvrbwhGeKbdF22kKJDxomI2PuFpvNS7YXfiSb8t5zSvf3CpaL4x6UcN84qbSXy5qM5yTv+AEJ9omS8XGzJMino9jxL6lglWwpWMY6tLShYLPZ3sBiJR6WxaeYP8HkH9jca5rzrMXiQZxn0+mc50ygncrDU8fIXS8skayCYquDWbd1vyRn5Nflue2S82R3WrY8MH+FJMR0RP6xcufz9bdw2pghkpKZL0EBfu6ogoWX2GRm/yvlno0LJPtYkbx5wQz5KnunpJRm1+W5uddo2VOUITM3v40fa5SVfwbyV9mr5YXd/5HdRYelg7efLBx+r2zI3ed0bKvVwwPaQ+vwp9Fj5caliyWzpFg+mDpd1iTvk3359XWYMew82ZGTI3euXCE9O0Yg/xi5AfnLq6pk2pL3payiQnxgKLx/3RRZeyBFtmRmtH49POCZYlu0nbZQbHiqRsXeK8sOzpKSihyZ0uNlSS5eL/nlB+vyDIucJrnH9suq9DnS0a+bjEb+pQdnSrVU4fmbLznH9omvV6BMTfinHCz93ulY1qEeTj20IAMTYiUtu1AO5R6RyqpqWb1xl4w6u5dTnp6dI2TDrpqH8UBWgcR1CpWIkA7WdnR4sIxI7CnLMPJ1JwPCu0l6WZ4cPloAK7xKPsn8US6KOcspT4+gaNmYt89aTy3Nkc6BHSXCLxjWerH1ElHKqsolpSRbogJCW70OntIeg2JjJfVIoaQVHZGK6mr5cM9uGdert1Oe3p06yfq0mjokF+RL19AwiexQUwc1EhQ1FFSC0ZM78IRnim3RdtpCiQnsK0fKD0tRRQY6/koYmWulZ8ivnPJE+MdLWulma72gPE1CfGNh4IRLWWW+ZSQoFdVHLQMh2DeSdThdQwEj+ZugrdCP0Ftm90VYXw8lQ9c65H0I2mjyP+FwGh9sv2n2L4Y6QGOhOp8P1sdBSyFv9VpASdA26L8czjMZ2xugPdCF5jjN/1eH6/7O7F8EXeZwfj3nNadab1eI7hhsjVxryYK7OqpjiFOePek5MnbIL6z1AejIOkeESgyOU9RF/sLSL6UaLj13EuUfKllHj9Rt6wgwyj/MKc/e4gxY5zXOof5hXSU2IBwuYucXRufAcOkbGgeXYNqZL7SHtkdsULBkFNfXIRPrus+RnfAmTOhdYzwMiomVLqGhEhscXDcKXjX9Rtl0x13y9cFUeBMyW6/wHvZMsS3aTlsowT6RUlxZ78lQr0KwTyenPOpN6BUywlqPCegrob4xMAiinPKEYF90QG/JPLrrzBe6ndbhlAwFdKwDsHgUGgOX/yAs7zdJnSGtweXQMybveCz0zXsuNBj6JfZdZPL3hV7BOc7Gsgi6G/ocOgt5amt+C/S6ObaLTlVAiWZfLT7Yp+f/PTTb7LsNOoL9w7BU/Rbn7IGlTlpdb8qm/uOx0H8aqeMd0CZV5bHSU7ktp0aDTkbnyHVe/L1Hb5ApowZbbu8qjHYvTOwh+cVlsvNg67vwGmJrdK9zPRYmr7PmlN+G6/G67sNlD17y6pasJRBuyWcG3yB/27VSSquOn9HyenJ74Hn8yb6GZovGLIT5B1gGwW8GD5Ht2ahDdU0uNXImvvOWXLDgFcuI6APvgzvwhGeKbdF22sK0yEl/GxqzEOAdLNN6zJdBEZMsL0I1PFq1+NoCZGKX2bIua56UV5ed4fK23zqcaozCGGgxOuFc3cAy3/xolmNdn54d2I4xedVQUG2uNZqM4aC+0TTk/8bsfxu6D9vPGQ/FDViqMXABdBOkQ7+e2PcSlqugTxzKs9Qsv4cSHK57toNnI8xc9yPoRez3x/IS6Etc82jDCmLfK1iopENUt2YNIbMLSjAarR+xxsB1nYNRrCOlx8oRMFdflVVP3yaH8opkwrB+MhJu8REDe4ifj48EBfrJU7dcKo+9rsVvXbKPF8GtVz/a01FdDvY5oi8HDTarZfnImXK4rMBa90Zg0bNDpsvqjC2yNmt76xTaQ9sjA3EJnUPq6xCL9axS5zqUlJdbAYy1fHXr7dZUhSPFx4/Ld+npMjK+hxXo2Np4wjPFtmg7baGUVObAsIyu29ZRdmml87OtHeenCGCs5ZZeb2Oqosar5iXeMrHrHEylfCb7i79unUK30zqc6tSDWgWNdZ7HGzGNdPlndLyDjXpDC0xaw3PUbquBcAM0FXof+SshfSrVe6HBizOgfzVy3SoHY0eve6/DdXtAn0DHzDkmGM+Cc1hsC7I9NVO6R4db89w+3l5WZ7N2a7JTnuBAfytNuXpEovyw95DVWb20/Gu55JFXZeKjC+ThBatk4640txgJyo4j6dKtQ6TEYY7Yx+Yt42MHWYFnjgRj5KdpylVdh8mW/JS6kcXjA6/B3GWOvHvAPT8+T2qPrZgqSAgPR9xBKIKuvOSKPn1lzf79TnlC/P2tNGXKwETZAINAjYeIwEArTfH39pER3bvLfsQwuANPeKbYFm2nLZSso7sl3K8LXPGx6Mh8pE/oKCuY0RE/ryArTRkQfpkcKttWN+q+uPODiE1Ilc359cZpa5PVTupwqh6Fz6BlGJX/HR1vHpYRJ8irQ5snkecd5C3Bsgu2ayKqRLpj+wLs/9YYBdaThu3D2K8RMo9B43QftjWypBxpS7Cub0b9ykJOct27kPdzHFOBZR9sH8J6qTEOboeGQjefYp1dRt29zy76Qubdd431KdsH65MkOSNPrr1QZ1rgkvlqq/SMjZAnb7nEyqtpT7zl6ChpG6i79687VsiLQ2+15rg/TN8kyQha+nW3c2vcOWkbpEdwtMxOnAwXWLUV0PSUGQkOCo+Xy7qcY803qwtZmbfnE1mfu7v16+EB7VGFqYPZX3wuC69GHTCSe397kuzNz5NpiTV1eHfbVukdESFzx19qTTNo2izjXYgOCpLnsB/BO5bbfNXe3fJ5SrKb6tH+nym2RdtpC8WOEMa1mS/JpG7PWJ9H7ij82Oo0E8N1JlxkW+FKBDN2lwlxs6xnKv94qqzJmGulxQUOlLPCx2H+P9ly6Svrs1+TA6XOnx6zDjXY9FvYUwEvmt9g8ZAZxddOK9R9Hon0EqwHm/X7TceslBhvQZWJDfgSGg7thW7EMWXmmClY/B7b55vtQcbTUOv1eARpHzl+HmmMiU1YT8C65nsKukIPh3KgSUg7gjRfrKuvZgW2NQbihOjUQ99rHGMn2ye+k/QWtG8qljsH7bRXCvu5N0C1pYjqZ80+tmtydrV+dPuZwBPaQpke7/y3TYh7mDt5gxxMKrKd1t9RQAf7JhZvniC9LhQb6y9goWpI/xNcQoMiX3U4x49YnNPIdUY5rOfWxiiYWIk/GDU8Rj0a7oniIoQQQtoxbeIvM2LEr0GJOkXwgLvLQgghhJA2ZihgxP9Ld5eBEEIIIT+Ff5mREEIIIU1CQ4EQQgghNBQIIYQQ4jr0KBBCCCGEhgIhhBBCXIceBUIIIYTQUCCEEEKI69CjQAghhBAaCoQQQghxHXoUCCGEEEJDgRBCCCGuQ48CIYQQQmgoEEIIIcR16FEghBBCCA0FQgghhLgOPQqEEEIIaRIaCoQQQghpEpvdbm8y8eeEzWa7AwuV0hfafYYvGQnlnuFrnGk8oQ4K69F2YFu0HdgWP6/2iIc9ENVYAg0F9xkmm9AoQ911/ZbAE+qgsB5tB7ZF24Ft0bawufF9y6kHQgghhNBQIIQQQojr0KPgPl5x47VbCk+og8J6tB3YFm0HtkXb4hV3XZgxCoQQQghpEnoUCCGEEEJDgRBCCCGuQ49C63/i8hqUDSW19rVbCpQ9ANoA/Qhth55wd5maC8p+ANoGbdHPj9xdHldBmfuasteqCPq9u8vVHFDu+/V3YZ6p37fn3zTWJ5t6VEND22kdnoS2mufqEyjOnWU8jXrMgQ45/EYuc2cZm1mHRQ7l13fWlpOdpyVhjEIrgwa+CIsSaKHdbh/Y2tdvoTrYsAhC+Uuw6ov1r6H7sf2dm4vmMvqjw2Ioyt7u/3AU6uKNxSHoPNQn1d3lcbHs+lt4DzoXKoc+hu5CPfa6tWDN/E1j31lYVEP/Az2I/ZvaYR1CsV5k1u/Doj+273RjMZtbjzm6D9vPubVwLdRHIH0uFkeQ9qdTPOVpQ49CK4PG/RKL/Na+bgvXQdEHWVFDQcU/8el+xkL725uRYNCO9TuUvQyqxPo66Go3l6nZv2ns2wmd6b/ueqbrYBkJhiDd1aqF+vm+X79sqg5mkHYd9L+tWSYaCqTZo1fj/sqGPsXD/X/t9Fbqy0/dqt9DtX/Cu70ypbVfIC2IulkvQht0gjpgXd3D3dxcpp89aIunoTTciOnQH9vxDbnHTKOoW7+juwtzGlwIZbW2p42GAmkWeFCroMFY7Qqda1zH7ZFfoR7nYHkpNMO4/dodKLcfFldC77u7LM1BR+BYPAt9aqYdfoTUs0Dc2y6PQmqwvQPd004b459QL0jfVxmQuu7bK1PdMRigoUBOC7xECrFYC13STst/2CzVM7LMzJG3R9TQ+QH1yHJ3QZoLyr5AjTboIuN6bfPxCT8j3oWucXchmoP+JszARmNGXm2vv3GbzeaDxa+hRa19bRoKpDkPbBQUbtYDsbgY2uXyidwMyh4EhdSuYzEeaq9fo7hlpNGSoA2izbK7eSG26/q0d9AOv3DYvLI9/sZNPTo7bF7djn/jF0O7YPCkt/aF1UIhrfvQ6stvFBSJdW3w2TqSameNoD+8N02UvRqb/0YdVrq5TM0hBlqGetT+Ft5FPdTt3a4wc/rjoN+5uyynyRLUpROWFdAMtEWBuwvU3N+08Yi8BOl/27sK+7egPhPcWMzm1OEyrPfFUkfiGiDbpr94OEE9RmF9sIlHOtDWfye2pvsIt8Ug8fNIQgghhDQJpx4IIYQQQkOBEEIIIa5DjwIhhBBCaCgQQgghxHXoUSCEEEIIDQVCCCGEuA49CoQQQghpkv8P1/xyZtOleD0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "heatmap(scores, \n",
    "        title='Model response to 2 hyperparameters',\n",
    "        xticklabels=list(K),\n",
    "        yticklabels=metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GridSearchCV\n",
    "\n",
    "So far, so good.  It would not *too* hard to keep track of the best model discovered within the inner loop.  And any Python programmer could construct more nested loops to search over 3, or 4, or 5, different hyperparameters.  \n",
    "\n",
    "We could store all the scores in a parameter grid of N dimensions.  \n",
    "\n",
    "Maybe while we are at it, it would be nice to remember the training and scoring times different hyperparameters take. \n",
    "\n",
    "It could be useful to allow for different scoring metrics to be performed within the nested search of hyperparameters.  Or actually perform multiple different scoring functions that may inform the quality of hyperparameter sets differently.\n",
    "\n",
    "We might also want our code to perform more robust and configurable train/test split strategies.\n",
    "\n",
    "But really, it is much easier to take the `GridSearchCV` function from scikit-learn that does all of this for us and is well-tested to avoid any pitfalls, bugs, or edge cases we might overlook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=KNeighborsClassifier(),\n",
       "             param_grid={'metric': ['minkowski', 'manhattan', 'euclidean',\n",
       "                                    'chebyshev'],\n",
       "                         'n_neighbors': range(1, 18, 2),\n",
       "                         'weights': ['uniform', 'distance']})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "parameters = {'n_neighbors': range(1, 18, 2),\n",
    "              'weights': ['uniform', 'distance'],\n",
    "              'metric': ['minkowski', 'manhattan', 'euclidean', 'chebyshev']\n",
    "             }\n",
    "\n",
    "grid = GridSearchCV(KNeighborsClassifier(), parameters)\n",
    "# Best fit over cross-product of parameter space, cross-validated\n",
    "model = grid.fit(cancer.data, cancer.target)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identifying the best hyperparameters\n",
    "\n",
    "One additional nice detail of `GridSearchCV` is that by default once a best set of hyperparameters is identified, the model is refit against the entire dataset rather than only the training split.  This can improve accuracy while avoiding overfitting in the initial hyperparameter choice.\n",
    "\n",
    "The object delivered in an attribute—but also usable directly as the grid search model object itself—reflects this improved refitting (if the argument `refit` is kept as the default `True` value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'metric': 'manhattan', 'n_neighbors': 9, 'weights': 'uniform'} \n",
      "\n",
      "KNeighborsClassifier(metric='manhattan', n_neighbors=9) \n",
      "\n",
      "0.9385188635305077\n"
     ]
    }
   ],
   "source": [
    "print(model.best_params_,'\\n')\n",
    "print(model.best_estimator_,'\\n')\n",
    "print(model.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0,\n",
       "       0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0,\n",
       "       1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0,\n",
       "       1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0,\n",
       "       0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1,\n",
       "       1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0,\n",
       "       0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0,\n",
       "       1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1,\n",
       "       1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0,\n",
       "       0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0,\n",
       "       0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1,\n",
       "       1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1,\n",
       "       1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,\n",
       "       1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1,\n",
       "       1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1,\n",
       "       1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(cancer.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examining the search space\n",
    "\n",
    "Some the information collected about the search is times taken for steps.  Given a search across a large, multi-dimensional, hyperparameter space can require many combinations, the fitting can take a considerable time. `KNeighborsClassifier` was chosen for this lesson in part because it is a very fast model.\n",
    "\n",
    "Moreover, while KNN performs pretty much equally quickly across a range of hyperparameters, that is definitely **not true** of many other models.  In some cases, a hyperparameter choses among different algorithms with very different performance characteristics.  In others, a hyperparameter chooses among threshhold type values that can greatly affect convergence rates or other computational details.  Being able to know not only that this combination of hyperparameters has better *accuracy*, but also what the relative *performance* of each is, can be imporant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_metric</th>\n",
       "      <th>param_n_neighbors</th>\n",
       "      <th>param_weights</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>split4_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rank_test_score</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001082</td>\n",
       "      <td>0.000186</td>\n",
       "      <td>0.003229</td>\n",
       "      <td>0.000230</td>\n",
       "      <td>manhattan</td>\n",
       "      <td>11</td>\n",
       "      <td>distance</td>\n",
       "      <td>{'metric': 'manhattan', 'n_neighbors': 11, 'we...</td>\n",
       "      <td>0.877193</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.956140</td>\n",
       "      <td>0.973684</td>\n",
       "      <td>0.955752</td>\n",
       "      <td>0.938519</td>\n",
       "      <td>0.033706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001867</td>\n",
       "      <td>0.000276</td>\n",
       "      <td>0.007957</td>\n",
       "      <td>0.001490</td>\n",
       "      <td>manhattan</td>\n",
       "      <td>9</td>\n",
       "      <td>uniform</td>\n",
       "      <td>{'metric': 'manhattan', 'n_neighbors': 9, 'wei...</td>\n",
       "      <td>0.877193</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.956140</td>\n",
       "      <td>0.973684</td>\n",
       "      <td>0.955752</td>\n",
       "      <td>0.938519</td>\n",
       "      <td>0.033706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.001146</td>\n",
       "      <td>0.000100</td>\n",
       "      <td>0.006247</td>\n",
       "      <td>0.000341</td>\n",
       "      <td>manhattan</td>\n",
       "      <td>11</td>\n",
       "      <td>uniform</td>\n",
       "      <td>{'metric': 'manhattan', 'n_neighbors': 11, 'we...</td>\n",
       "      <td>0.877193</td>\n",
       "      <td>0.938596</td>\n",
       "      <td>0.956140</td>\n",
       "      <td>0.964912</td>\n",
       "      <td>0.946903</td>\n",
       "      <td>0.936749</td>\n",
       "      <td>0.031057</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.001382</td>\n",
       "      <td>0.000189</td>\n",
       "      <td>0.004027</td>\n",
       "      <td>0.000132</td>\n",
       "      <td>manhattan</td>\n",
       "      <td>9</td>\n",
       "      <td>distance</td>\n",
       "      <td>{'metric': 'manhattan', 'n_neighbors': 9, 'wei...</td>\n",
       "      <td>0.877193</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.956140</td>\n",
       "      <td>0.964912</td>\n",
       "      <td>0.955752</td>\n",
       "      <td>0.935010</td>\n",
       "      <td>0.032584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.001216</td>\n",
       "      <td>0.000178</td>\n",
       "      <td>0.003713</td>\n",
       "      <td>0.000305</td>\n",
       "      <td>manhattan</td>\n",
       "      <td>15</td>\n",
       "      <td>distance</td>\n",
       "      <td>{'metric': 'manhattan', 'n_neighbors': 15, 'we...</td>\n",
       "      <td>0.859649</td>\n",
       "      <td>0.938596</td>\n",
       "      <td>0.947368</td>\n",
       "      <td>0.964912</td>\n",
       "      <td>0.955752</td>\n",
       "      <td>0.933256</td>\n",
       "      <td>0.037826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>0.000853</td>\n",
       "      <td>0.000117</td>\n",
       "      <td>0.001040</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>euclidean</td>\n",
       "      <td>1</td>\n",
       "      <td>distance</td>\n",
       "      <td>{'metric': 'euclidean', 'n_neighbors': 1, 'wei...</td>\n",
       "      <td>0.859649</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.911504</td>\n",
       "      <td>0.905108</td>\n",
       "      <td>0.023754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>0.001427</td>\n",
       "      <td>0.000329</td>\n",
       "      <td>0.001507</td>\n",
       "      <td>0.000364</td>\n",
       "      <td>minkowski</td>\n",
       "      <td>1</td>\n",
       "      <td>distance</td>\n",
       "      <td>{'metric': 'minkowski', 'n_neighbors': 1, 'wei...</td>\n",
       "      <td>0.859649</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.911504</td>\n",
       "      <td>0.905108</td>\n",
       "      <td>0.023754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>0.001537</td>\n",
       "      <td>0.000607</td>\n",
       "      <td>0.001471</td>\n",
       "      <td>0.000480</td>\n",
       "      <td>minkowski</td>\n",
       "      <td>1</td>\n",
       "      <td>uniform</td>\n",
       "      <td>{'metric': 'minkowski', 'n_neighbors': 1, 'wei...</td>\n",
       "      <td>0.859649</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.911504</td>\n",
       "      <td>0.905108</td>\n",
       "      <td>0.023754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>0.001172</td>\n",
       "      <td>0.000123</td>\n",
       "      <td>0.002419</td>\n",
       "      <td>0.000209</td>\n",
       "      <td>chebyshev</td>\n",
       "      <td>1</td>\n",
       "      <td>distance</td>\n",
       "      <td>{'metric': 'chebyshev', 'n_neighbors': 1, 'wei...</td>\n",
       "      <td>0.868421</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.893805</td>\n",
       "      <td>0.905077</td>\n",
       "      <td>0.021864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>0.001291</td>\n",
       "      <td>0.000153</td>\n",
       "      <td>0.006822</td>\n",
       "      <td>0.001321</td>\n",
       "      <td>chebyshev</td>\n",
       "      <td>1</td>\n",
       "      <td>uniform</td>\n",
       "      <td>{'metric': 'chebyshev', 'n_neighbors': 1, 'wei...</td>\n",
       "      <td>0.868421</td>\n",
       "      <td>0.929825</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.912281</td>\n",
       "      <td>0.893805</td>\n",
       "      <td>0.905077</td>\n",
       "      <td>0.021864</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>72 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "rank_test_score                                                                 \n",
       "1                     0.001082      0.000186         0.003229        0.000230   \n",
       "1                     0.001867      0.000276         0.007957        0.001490   \n",
       "3                     0.001146      0.000100         0.006247        0.000341   \n",
       "4                     0.001382      0.000189         0.004027        0.000132   \n",
       "5                     0.001216      0.000178         0.003713        0.000305   \n",
       "...                        ...           ...              ...             ...   \n",
       "67                    0.000853      0.000117         0.001040        0.000175   \n",
       "67                    0.001427      0.000329         0.001507        0.000364   \n",
       "67                    0.001537      0.000607         0.001471        0.000480   \n",
       "71                    0.001172      0.000123         0.002419        0.000209   \n",
       "71                    0.001291      0.000153         0.006822        0.001321   \n",
       "\n",
       "                param_metric param_n_neighbors param_weights  \\\n",
       "rank_test_score                                                \n",
       "1                  manhattan                11      distance   \n",
       "1                  manhattan                 9       uniform   \n",
       "3                  manhattan                11       uniform   \n",
       "4                  manhattan                 9      distance   \n",
       "5                  manhattan                15      distance   \n",
       "...                      ...               ...           ...   \n",
       "67                 euclidean                 1      distance   \n",
       "67                 minkowski                 1      distance   \n",
       "67                 minkowski                 1       uniform   \n",
       "71                 chebyshev                 1      distance   \n",
       "71                 chebyshev                 1       uniform   \n",
       "\n",
       "                                                            params  \\\n",
       "rank_test_score                                                      \n",
       "1                {'metric': 'manhattan', 'n_neighbors': 11, 'we...   \n",
       "1                {'metric': 'manhattan', 'n_neighbors': 9, 'wei...   \n",
       "3                {'metric': 'manhattan', 'n_neighbors': 11, 'we...   \n",
       "4                {'metric': 'manhattan', 'n_neighbors': 9, 'wei...   \n",
       "5                {'metric': 'manhattan', 'n_neighbors': 15, 'we...   \n",
       "...                                                            ...   \n",
       "67               {'metric': 'euclidean', 'n_neighbors': 1, 'wei...   \n",
       "67               {'metric': 'minkowski', 'n_neighbors': 1, 'wei...   \n",
       "67               {'metric': 'minkowski', 'n_neighbors': 1, 'wei...   \n",
       "71               {'metric': 'chebyshev', 'n_neighbors': 1, 'wei...   \n",
       "71               {'metric': 'chebyshev', 'n_neighbors': 1, 'wei...   \n",
       "\n",
       "                 split0_test_score  split1_test_score  split2_test_score  \\\n",
       "rank_test_score                                                            \n",
       "1                         0.877193           0.929825           0.956140   \n",
       "1                         0.877193           0.929825           0.956140   \n",
       "3                         0.877193           0.938596           0.956140   \n",
       "4                         0.877193           0.921053           0.956140   \n",
       "5                         0.859649           0.938596           0.947368   \n",
       "...                            ...                ...                ...   \n",
       "67                        0.859649           0.929825           0.912281   \n",
       "67                        0.859649           0.929825           0.912281   \n",
       "67                        0.859649           0.929825           0.912281   \n",
       "71                        0.868421           0.929825           0.921053   \n",
       "71                        0.868421           0.929825           0.921053   \n",
       "\n",
       "                 split3_test_score  split4_test_score  mean_test_score  \\\n",
       "rank_test_score                                                          \n",
       "1                         0.973684           0.955752         0.938519   \n",
       "1                         0.973684           0.955752         0.938519   \n",
       "3                         0.964912           0.946903         0.936749   \n",
       "4                         0.964912           0.955752         0.935010   \n",
       "5                         0.964912           0.955752         0.933256   \n",
       "...                            ...                ...              ...   \n",
       "67                        0.912281           0.911504         0.905108   \n",
       "67                        0.912281           0.911504         0.905108   \n",
       "67                        0.912281           0.911504         0.905108   \n",
       "71                        0.912281           0.893805         0.905077   \n",
       "71                        0.912281           0.893805         0.905077   \n",
       "\n",
       "                 std_test_score  \n",
       "rank_test_score                  \n",
       "1                      0.033706  \n",
       "1                      0.033706  \n",
       "3                      0.031057  \n",
       "4                      0.032584  \n",
       "5                      0.037826  \n",
       "...                         ...  \n",
       "67                     0.023754  \n",
       "67                     0.023754  \n",
       "67                     0.023754  \n",
       "71                     0.021864  \n",
       "71                     0.021864  \n",
       "\n",
       "[72 rows x 15 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(pd.DataFrame(grid.cv_results_)\n",
    "   .set_index('rank_test_score')\n",
    "   .sort_index()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next lesson\n",
    "\n",
    "In the next several lessons, we turned first to Clustering, and a discussion of *unsupervised learning* techniques.  The few lessons past that will look broadly at *feature engineering*, which includes decomposition, an unsupervised technique of a different sort.\n",
    "\n",
    "<a href=\"Clustering.ipynb\"><img src=\"img/open-notebook.png\" align=\"left\"/></a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}