practicals/week_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Under- and overfitting, model selection\n",
    "\n",
    "## Preliminaries\n",
    "\n",
    "In the first set of exercises you had to implement the training and evaluation of the linear regression and $k$-NN methods from scratch in order to practice your `numpy` skills. From this set of exercises onward, you can use the implementations provided in `scikit-learn` or other higher-level libraries. We start this set of exercises by demonstrating some of the features of `scikit-learn`.\n",
    "\n",
    "For example, implementation of linear regression model fitting with an analytical solution for the parameters is provided by the class `sklearn.linar_model.LinearRegression`. You can train a linear regression model in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets, linear_model\n",
    "\n",
    "# load the diabetes dataset\n",
    "diabetes = datasets.load_diabetes()\n",
    "\n",
    "# use only one feature\n",
    "X = diabetes.data[:, np.newaxis, 2]\n",
    "y = diabetes.target\n",
    "\n",
    "# split the data into training/testing sets\n",
    "X_train = X[:-20]\n",
    "X_test = X[-20:]\n",
    "\n",
    "# split the targets into training/testing sets\n",
    "y_train = y[:-20]\n",
    "y_test = y[-20:]\n",
    "\n",
    "# create linear regression object\n",
    "model = linear_model.LinearRegression()\n",
    "\n",
    "# train the model using the training dataset\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the training dataset and the learned regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEWCAYAAACNJFuYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2deZgU1bm43zPDDPsiIyoRBzDq9SJgFLd4JVFJXCOSqFciKpoY4hiXGHdNoiZyjcvPaEwMmkhcZuJG4hJj4m6MxqhAEBERAUFAVERBEGSZ+X5/VDXU9NTeVV3VM9/7PPV0dy2nzqnu/r7zLeccIyIoiqIoipOqrCugKIqi5A9VDoqiKEobVDkoiqIobVDloCiKorRBlYOiKIrSBlUOiqIoShtUOSgVgzFmpDHmrazrUakYY8YZY57Iuh5KZWB0nIOSN4wxC4HTROSprOuiKB0VtRwUJQBjTKckzkn6noqSJqoclIrBGHOgMWaJ4/NCY8z5xpiZxphVxpj7jDFdHMe/YYyZYYxZaYz5lzFmuOPYxcaY+caY1caY2caYbzqOnWKMedEY80tjzArgCpe6XGGMmWKMaTTGfAqcYoypcpS7whhzvzGmr+Oak40xi+xjP7Hr/7U45RljutjnrrDb96oxZltH/RfYbXvHGDPOsf8FR332t69bZb/u7zj2nDHm5/ZzWG2MecIYs3Vp36BSSahyUCqd/wUOAwYDw4FTAIwxewCTge8DdcCtwCPGmM72dfOBkUBv4Eqg0RjT31HuvsACYFtgose9jwamAH2AJuAsYAzwVeALwCfAb+z6DAFuAcYB/e37bh+3PGC8XcYOdvtOB9YZY7oDvwIOF5GewP7AjOKK20rmr/a5dcANwF+NMXWO004ATgW2AWqB8z2eg9IOUeWgVDq/EpH3RORj4C/Al+z9E4BbReRlEWkWkTuB9cB+ACLygH1di4jcB7wN7OMo9z0RuVlENonIOo97vyQiD9llrMMS0JeJyBIRWY9lcRxru4iOBf4iIi+IyAbgp0BxwC9KeRuxhPpOdvumicindjktwFBjTFcRWSYib7jU/UjgbRG5227jPcAc4CjHOX8Qkbl2Xe53PFulA6DKQal03ne8Xwv0sN8PBM6zXS4rjTErsXrZX4DNLp4ZjmNDAafbZHGIexefMxB40FHmm0AzlvXxBef5IrIWWFFCeXcDjwP3GmPeM8Zca4ypEZHPgOOxFMsyY8xfjTG7utT9C8Cion2LaG3NeD1bpQOgykFprywGJopIH8fWTUTuMcYMBH4HnAnUiUgfYBZgHNeHSeMrPmcxljvHec8uIrIUWAYMKJxojOmK1fOPVZ6IbBSRK0VkCJbr6BvAyQAi8riIfB3LfTXHbmsx72EpHyf1wNIQ7VY6AKoclLxSYwddC1vU7J3fAacbY/Y1Ft2NMUcaY3oC3bEE8XIAY8ypWJZDqUwCJtrKB2NMP2PM0faxKcBRdhC4FstFZNyLCS7PGHOQMWaYMaYa+BTLzdRijNnWGHO0HXtYD6zBcjMV8xiwizHmBGNMJ2PM8cAQ4NH4zVfaE6oclLzyGLDOsV0R5WIRmQp8D/g1ViB3HnawWkRmA/8PeAn4ABgGvJhAnW8CHgGeMMasBv6NFdjG9vufBdyLZUWsAT7EEuCRywO2w1I4n2K5m/6B5WqqAn6EZRl8jBXMbiguWERWYFkb52G5ty4EviEiH8VrutLe0EFwipIBxpgewEpgZxF5J+v6KEoxajkoSpkwxhxljOlmu3yuB14HFmZbK0VxR5WDopSPo7HcPe8BOwNjRU13JaeoW0lRFEVpg1oOiqIoShvaxeReW2+9tQwaNCjraiiKolQU06ZN+0hE+rkdaxfKYdCgQUydOjXraiiKolQUxpjiUfKbUbeSoiiK0gZVDoqiKEobVDkoiqIobUhdOdjz4rxijHnNGPOGMeZKe/8d9kIkM+ztS/Z+Y4z5lTFmnrEWcdkz7ToqiqIorSlHQHo9cLCIrDHG1AAvGGP+Zh+7QESmFJ1/ONYAoZ2x5pH5LVvmk1EURVHKQOqWg1issT/W2JvfyLujgbvs6/4N9ClaoUtRFKX90dQEgwZBVZX12tSUaXXKEnMwxlQbY2ZgzUL5pIi8bB+aaLuOfulYvnF7Wi96soS2yylijJlgjJlqjJm6fPnyVOuvKIqSKk1NMGECLFoEItbrhAmZKoiyKAd7GcMvYS12so8xZihwCbArsDfQF7goYpm3icheIrJXv36uYzgURVEqg8sug7VrW+9bu9banxFlzVYSkZXAs8Bh9tq2Yq+N+we2rN+7FGs5xwID0NWpFEUphZy5bNrw7rvR9peBcmQr9TPG9LHfdwW+DswpxBGMMQYYg7VMI1iLm5xsZy3tB6wSkWVp11NRlHZKDl02baivj7a/DJTDcugPPGuMmQm8ihVzeBRoMsa8jjWn/dbAVfb5jwELsFbu+h1wRhnqqChKeyWHLps2TJwI3bq13tetm7U/I1JPZRWRmcAeLvsP9jhfgB+kXS9FUSqIpiZLmL/7rtWbnjgRxo0Ld20OXTZtKLQlbhtToF1MvKcoSjum4BYq9P4LbiEIJzzr661r3PbniXHjMlUGxej0GYqi5JtS3UI5dNlUAqocFKWj094zecaNg9tug4EDwRjr9bbbctVLzyPqVlKUjkypLptykIRbKGcum0pALQdF6choJo/igSoHRenIVEomj7qFyo66lRSlI6OZPIoHajkoSkdGXTaKB6ocFKUjoy4bxQNVDorS0Rk3DhYuhJYW6zULxZD3dNoOiMYcFEXJlkpIp+2AqOWgKFmhvWWLSkin7YCo5aAoWaC95S1UQjptB0QtB0XJAu0tbyGHaxkoqhwUJRu0t7wFTafNJaoclMokz/76MHWL0lvOc1uTQNNpY7NoEXz4YUqFi0jFbyNGjBClA9HYKNKtm4i16KO1detm7c+asHVL+jylQ7Fggchpp4l06iRyzjnxywGmiodczVywJ7GpcuhgDBzYWlgWtoEDs65ZtLo1Nlr7jbFe3QR+nttaiYR55jlm3jyR73zHUgqdO4uceabI4sXxy1PloLQvjHEXmMZkXbPk65antla4YK1kK2zuXJHx40Wqq0W6dBE5+2yRJUtKL9dPOWjMQak88pzdknTd8tLWQurtokWWWC2k3lZS/KMCM8TeegtOPhl23RXuvx/OPhsWLICbboLtt0/33qoclMqj3NktUQLCSddt4kSorW29r7a2/Jk8FShY21BBGWJvvmnF44cMgSlT4NxzLaVwww3Qv3+ZKuFlUiS1AV2AV4DXgDeAK+39g4GXgXnAfUCtvb+z/XmefXxQ0D3UrdQBKZeLI44rIsm6NTaK1NS0vn9NTfldIXlyb8WlAuI3b7whMnas9Vi7dRO54AKRDz5I735kGXMADNDDfl9jC/z9gPuBsfb+SUCD/f4MYJL9fixwX9A9VDkoqZG1QMn6/nmrRynkOOYwc6bIccdZSqFHD5GLLxb58MP07+unHFJ3K9l1WGN/rLE3AQ4Gptj77wTG2O+Ptj9jHx9ljDFp11Npp5Q6RiBrV0TW9y/QHgaq5XA8xcyZcOyxMHw4/P3vcMkl1sS4V18N/fplVi0LL62R5AZUAzOANcA1wNbAPMfxHYBZ9vtZwADHsfnA1n7lq+WguJJET9Grx1xdXZ6snTz12Cs9WylHTJ8uMmaM9VX26iXyk5+IrFhR/nqQl1RWoA/wLHBAqcoBmABMBabW19en9eyUSiYJweqmYIq3NF0TSbpCVLhnztSpIqNHW19j794il18u8vHH2dUnN8rBqgs/BS4APgI62fu+DDxuv38c+LL9vpN9nvErUy2HnBNXKJUqzJIKojrrUV1dvp584b4FS6Vwn7iKIaf+9o7AK6+IHHmk9di32krkZz8TWbky61plrByAfkAf+31X4J/AN4AHaB2QPsN+/wNaB6TvD7qHKoccE1copekSKkWQlytrJ2lhnif3VAfi3/8WOfxw61H37Sty1VUiq1ZlXastZK0chgP/AWbaLqOf2vt3xEpxnWcris72/i7253n28R2D7qHKIcfEFUppuYRK7S2XS8gmfZ/2kIpaQbz4osghh1iPuK5O5OqrRT79NOtatSVXbqU0NlUOOSauUErDJZSEn71c7pmkhXldXXmUWgfnn/8U+drXrEfbr5/INdeIrF6dda288VMOOkJaSZe40z8kNW3EuHFWbmBLi/VaatpiudIhk5w2o6kJPv207f4sRlq3U/7xDzj4YBg50kpPvf56eOcduPBC6NEj69rFxEtrVNKmlkOOyTLmUMkk2X4vF1VdXeLV7ki0tIg884zIV79qPc7tthO54QaRzz7LumbhQd1KSqYEuXa8jmeZepmHtM+k6qDxhkRpaRF58kmRkSOtx9i/v8hNN4msXZt1zaKjykFJhySEV9YWglsbsq5TFHRNCG8SVvAtLSKPPy6y//7W49t+e5GbbxZZty6R2maCKgcleZISoFkKLq82ZBG8jSPIdDU5bxJsc0uLyGOPiey3n1XMDjuI3HKLyOefp1DvMqPKQUmepIR6li4PrzZ4bWnVKa4gS3rVuTj1ztr15kUCv8+WFpFHHxXZZx/r0vp6kUmT2odSKKDKQUmepIR6qX9iL7dQGKHl1QavLS3LIe4zCPsdpKUY8myNlPD7bGkRefhhkREjrEsGDRL53e9E1q8vQ73LjCoHJXmSshxKETJu19bWtl3/wKs8vyyecgq+KIIs6lQeXs+3oaG1+6yuLr3R51lYGDF+ny0tIg8+KLLHHtapO+4ocvvtIhs2pF/drFDloCRPHiaEi+IWKhaYhWuLBXOhDeUUaGEFWZxJAKM8oyiLCIVVaA0N3s84TSL8PpubRaZMEdl9d+u0nXYSueOO9q0UCqhyUNIha59zFLdQQWi5CY1COeVqQ/Fza2jwF2ROZea2+U0fnpbrLIxCa2z0vn+5Eg58fp/NzSL33y8ybJhVpV12EbnrLpGNG9OvWl5Q5aC0T+JYDklmRyWZYdTQ4D3WI8ha8POjpxV0D9Mz97t3OeIiHmzaJHLPPSJDhlhV2XVXkaYma39HQ5WD0j6JGnNobCxdKPrdO+kMI7/znZvfSGc/S6lUJRkk0MPep0zB7U2brCJ33dW6xZAhlpLoiEqhgCoHpf0SNlspqAce1XJIO8Mo6PxihegnSN3cWLW1bcuJEnMIg9czMiachZGQ62njRstdtMsuVrFDh1rupObmRIqvaFQ5KIpfD7y2dotbB8ItrBM3VTINyyGOIG1sLC1bKew93KyWhobW56U01mXjRiuwvNNOVnHDh1uBZ1UKW1DloCh+PfDq6rauqCD3RtzeblQXSpiYQwKCNBZh5sSqq7O2Mk7vsWGDlYK6445WMXvsYaWoqlJoiyoHJd+UIxgZNTAbJKRKHZ8Rpb3O872UXLlnWPULrEd9LgnFHNavtwarDR5sFTFihDWYraWlxLa2Y1Q5KPmlXCNtw/bAo/TKs0jl9Zr3qRzKIcwAvLhrbJfwLNevt6a1KOj/vfe2pr1QpRCMKgclvwS5FJL0jTc2eguvqJZDEsQRiHFHU5eqvEpRrqW4vXza8Pnn1gR4O+xg3WLffa0J8kIphazH6OQEVQ5KvnD+Mf2ESWOjeywgKDsn6N5RhFxao3nTToVN2u0V1i0X13KI8IzWTf6j3HyzNWU2WFNoP/54BEsh7/NClRFVDkp+CCucBw70F0il9OaDLIgw2UqlknZAO+nywyrSODGHkM9oLV3kJs6SL1QvExA54ABr0Z3I7qOUU2crCVUOSn4I0wMtCJMgy6IUsu49lpK+GcYlknSqrV+MIa0V/Ow2fEZXuYEfyna8JyDyVZ6VZ54pIaagK+NtRpVDJZGmL7SUshP+w3v+OZ1lp2U5JN2mOKTde/UT8nGUipsFkbIyXbPDrnI9P5JtsSyFg3hanuMrpT8jtRw2k6lyAHYAngVmA28A59j7rwCWAjPs7QjHNZcA84C3gEOD7tFulEOavdlSfdBpL3bvlS6adMwhL6RtuYRx30V1R5VJma5eLXLNNSL9eq0TEPkaT8jzHJDcM8raaswRWSuH/sCe9vuewFxgiK0cznc5fwjwGtAZGAzMB6r97tFulEOaPZpSyk56srqog8DcspXaQ7ZJ2m0Ik3qaZCC7RD79VOTqq0W23tq67SGHiLz407+n84zaw+8nAXLlVgIeBr7uoxwuAS5xfH4c+LJfme1GOaTpCy2l7KTr5cx+iRP8LVWApSEY8i5sskqBDcGqVSJXXSXSt69VpcMPF3nppVRvqdjkRjkAg4B3gV62clgIzAQmA1vZ5/waONFxze3AsS5lTQCmAlPr6+vTe3rlpCNYDgXiCPiglMow9fGa7yeqgkq7zDD3jCLAc+hnX7lS5Gc/E9lqK6sqRx4p8vLLmVWnQ5IL5QD0AKYB37I/bwtUA1XARGCyRFAOzq3dWA4dIeZQIKqwCuNDD2PJBGVLBbm43ARyKWXGIa5izYmf/ZNPRC6/XKR3b6sao0eLTJ1a9mookgPlANTY7qEfeRwfBMyy33dct5JI+89WKhDVVRUmBdavFxxlIFfUuZTCTKudZA/dLxMpjOWVketrxaT75Se9b5RerBQQGTPiXZk+vaxVaF8k8H1mHZA2wF3AjUX7+zvenwvca7/frSggvaDDBKQrkbg/0KiWQ5AATmJmUz8F5VffMEonyRz6clopCfDRRyKXjn5derJKQOQYHpAZDM9lXSuGhCzBrJXDAYDYsYXNaavA3cDr9v5HipTFZXaW0lvA4UH3UOWQEeV0VQXFGvzuGdZi8FNQfpZOGOWTpOUQND9UTvL1ly8XufhikR49RAzNchz3yUyG5rKuFUdCMaTM3Uppb6ocMqLUH2gUq6MURRTG7RNUZlBbnW6r4vsl3UMup5USgw8+ELngApHu3a2qjB0rMovdkqtr3jPDykFCGYSqHJR0KPc0BEm7sIrr77ZKmfPeXusXFNcpbeEVZAll1Bt//32R886zHktVlcgJJ4jMnh1Q56h1zVFgPVPUclDlkGvylh7pFMrOFcjq6tqumeyl2MIEtZ1rMWchqPzcWBkIyvfeEzn3XJGuXS2lcNJJInPmhKhznLrm7TeXFe0h5lCOTZVDRrhNb5H0IvVR6uLn96+pab1cZVDPO0wb4gqqJCyLUgcSJsDSpSJnny3SpYtVhfHjRebO9bkgiXZXirVaDio9W6kcmyqHjGhsbNsjz2reo6iprkmMTYgjqNqBW2TxYpEf/ECkc2dLKXznOyLz5pXp5uW0HNrBdxWEKgclHZKec8mrFxSmhxQm6OwU2klkGMVpf9jAdg57qosWWZ602lqRTp1ETjtNZMGCMleinAK7A7iwVDko6ZCUie/3hw8rDOIMkgsaGBfUjjiCKmpKbA56qu+8IzJhguWZq6kR+f73rX2ZUS4F2gHWfVDloKRDUj2rOAPM3AS9nyUQZwxF2LmakprjKGc91fnzLeugUyfLWmhosKyHDkPOvo80UOWgpENSPV2/Hlrc2USd2UpBKaZBlkuSvdQ403CUuac6b57Iqada8YTOnUXOPNOKM3Q4cmrJJYkqByU9khCe5ehNB/3R3dqRlnCIOoFfmXqqc+daGUfV1VYG0tlniyxZUpZb55ccx4CSQJWDkm+SiDkEkWTwuHBd0pMiOhc1KmNPdc4ckRNPtMYodO1qjVl4771Ub6nkBFUOSrqU2rsqFoyF1d6SKL+UoHMpk/1FwSteUvwcSinfZUzE7GsekRNO2KIUzjtPZNmy0m+nVA6qHJT0KLVnn6Zft9R01TgZUHFI053k8gxmMUSO5x4xNEu3zhvlggus+ZCUjocqByU9Ss3bT9N1U+pAtzDKJeslXJ11DYhjzGSoHMd9YmiWHnwqF/N/8uGAPUqvv1KxqHJQ0qPUvP00XTd+ZYdVPEFuqawsh+LMrOJpTBwZUDMYLt9iioBIT1bJpVwlH9E3OeWmVCyqHPJCXjMfSqlXqZlGabpukh7BnZclXMNYNCDT+x8hY7r+XUCkFyvlJ1wpK9gqeeWmVCyqHPJAXnOmS61XQ4O7cGpoCOcuSdN141Z2oU5xA+dpKfcoZQco1KnsKUfxsIBI727r5fJOV8nH9Gl7bh5+f0qmqHLIA3kdbVlqvZIa3ZyE68ZrrEKh7LQX4SkXHkr3ZfaWI/mLgMhWrJCf9b5eVq4Uz2ylimy7kiiqHPJATka/RqpXKRPexZkrqBQrJujapJVzli7Cora8xL5yGI8JiPTlI5nIJbKq67Yq/JVAVDnkgTxYDm4CzatedXWlTXhXnK0UttcaV+gG1SNJ5Zy1i9C+/4t8WQ7BiinUsVyu7nalfEpPtQqU0KhyyAM5ESht7u+1mpnbaF0vl1BQu8rR9iDhn6RyzljRP/+8yKjd3hMQ6ccHcm2fibL69/eW5d5K+0KVQ17IkSuilUBzq1fcCe+ijGVIUpiGsWCSUlAZuQife07koIOsW22zjcj114usWZPqLZV2jioHJbpAS1Kgu5Xjdu9Sp8kIY8GUOs1HWoFzD1paRJ5+WuQrX7Fusd12Ir/8pchnn0WruqK4kalyAHYAngVmA28A59j7+wJPAm/br1vZ+w3wK2AeMBPYM+geHV45hBE2XkKtutr9fDdhW1vbdhrsMHXzUkxOYRqlZ+/VXr/nkIRiiLteRFA5Lte2tIg8+aTIAQdYp3zhCyI33SSydm20apedvI7lUVzJWjn0Lwh4oCcwFxgCXAtcbO+/GLjGfn8E8DdbSewHvBx0jw6tHMIKVT/hFkYI9+gRTyD69bQbGoLPixPjiPuM4rYDvCfJKxaWAbGclhaRv/9dZP/9rd3bby/y61+LrFsXvqqZkXVcTYlMrtxKwMPA14G3gP6yRYG8Zb+/Ffi24/zN53ltHVo5RHH/NDZuyRiK4hIJ2/t3I2h6jELv0u+cuO0Nc03Ynm6YNarDBOI9thaMPPaYyL77Wrt22EHklltEPv/c//Hmijxk5CmRyI1yAAYB7wK9gJWO/abwGXgUOMBx7GlgL5eyJgBTgan19fVpPbv8EzWWECeY6tdrDgrChpkeo1s3a95ot2MFt1dBgMeph991YVeA8+rx+wnCEG1vAfkLR8retf/ZfPmtt1aYUiiQ17E8iie5UA5AD2Aa8C3788qi459IBOXg3NRyCBBSpZwvEjyBnR9+VkfYLUzvO47l4GVFuY3xqKmxYi5B9XAKQp92t4A8zFEyglcFRAb1Wy2/+53I+vX+jzPXqOVQcfgphyrKgDGmBvgT0CQif7Z3f2CM6W8f7w98aO9fihXELjDA3qe4MXEidOvWel+3btb+JM4HqK9332+M/3UA48bB6adb58ahuhrWrvU/J6j+Xm1ubnY/f8WKtvfcuBF69oSBA622VFe7X+t8Vi7PrQXDg4xhT6ZzNI+wstPWTP7eS8xd2oPTToPaWsfJTU0waBBUVVmvTU3ebcwDcX5bSn7x0hpJbVguo7uAG4v2X0frgPS19vsjaR2QfiXoHh3achCJniFSPN1zUAaS1wR2zoByEA0N3j11r956GIuhutq/Hn4jtMO4vLysgoiD/5ox8gDHyHBmCIjsxFy5g/GycaNPvSsxuKvZShUFGWcrHQAIVlrqDHs7AqjDchm9DTwF9JUtyuQ3wHzgdQJcStIRlUNSf8Ak0kfj3sftnlGm9whbZ7/2eR2PMjo84Jk039Uo95njZSgzBUR2YY7czTjZSHX8ILqiJESmyqEcW4dSDkn2KMslgIIGjkVNQw1b5zDtcxPwCTzjTZtE/vhHkSFDrMt3NW9KE9+WTVSFK0+Du0oZUOXQnoiauurXsw3jPkmCOILOy/UVpc6lCNiYltKmTdapu+5q3WrIEJF77hHZdFdTtPK8rJe6upLqpyhOVDm0J8IKvDAulbhjF6IS1ULxq3uUssromtl4Z5PcVfdD2YU5AiJDB3wi998v0twcs0A/5VCp8Qgld5SkHLCmttg96Lwstw6lHMIKvKDzvI4X1mFIEr8ZYd16v0GD1sKu7hZXiEbolW/cKPKHCf+SnczbAiLDmSFT+JY0d+1e2nP06wRoPEJJiFKVw55YcyP9gYCRylltHUI5OHvNYVY0C7Iw/Fw0aeDMVqquFhk1yltwB9U9yrOIk8kVQqFs2CBy++0iO+5onbIH0+RBjpZmHPWpq4vv+vFTAOWIR6jbqkOQiFsJOAZ4Dbgc6Br2unJs7V45ROktF4hrORSOJykcIkwjIQMHJmcdxSGgzPXrRW67TWTQIGv3iBEiDzNaWsK0LYrrJynXWhzUbdVhKFk52OmlQ4HTgY+AJcBJYa4tx9bulUOcrJuGhrYjemtrg9M4E8rWCVV/r95v2Psn1YMOMT3H53SWSZNE6uutXXvvLfLoo9ZEeZHaF0WAeynohoZ018NWt1WHoVS30ovAe3bs4efAN4CdgJuB24KuL8fW7pVDGDeL23QPxYPOamrCuVySFg5Rps+IYrkkUc8Aq+ZzauU3NMgO1UsErInxHnvMVgohyyhJcYWpb9QBiUFoGm2HoVTlsBtgPI69GXR9ObZ2rxziuojiCs6khUPY+kUNhnsJZa/psyPUbR2d5WZ+INuzWEBk/50/lMcfL1IKxXVxKjO/gXSNja2PJ1DfRHv1ajl0GFJLZQV2LOX6pLZ2rxyC3CxReuZBAj7utN5BZYatY5yy3QRx2JHeRdetpYvcyNnSn6UCIgd0flmevPgpSylEicP4ZWi5TeBXbNV5Ua5gtMYcOgQ6zqE94CeYkrIc/NwjpaaAuvnJk1I+YXu6PoH9z+gqN/BD2Y73BES+2vkleeYZh6UQR2BGnQ4kTPvL1avXbKUOgSqHvJH0H88r5lDcQw0SZl6CpzC5nV+dwwjPqOm4YQnbm3Zp3xq6yXWcL9vwvoDIQTwtz3U+pG29k7Km4q5J4ayL9uqVhFDlkCfS+nN7zREURgkF9WgLdfSrc9SMqjCzwYYlbG/aIZhX012u4QLpxwcCIl/r8rw8z8hwA+n8BHrQMy/VcghzD0UJiSqHPJG3YF+YTJswvWa/673uk1SPN2zZAwfKp/SQ/+NiqWO5gMgh/F1e3Pab3mUHKc1i5Rc0EkFzaCYAACAASURBVLyuTqRTp7blhI05KEqCqHLIE3lLE4xqMXjV2UuBVFf738fpiy+lJxxQxqpVIlcd9x/py0cCIofzV3mJfePPjhrFeiouo6ZGpEePLZ+d2UpqFShlRJVDnsib5RC0BKify6kwQ6iIv4IJuk+KPvRPPhG58kqRPn2soo/80hJ5ebvR4YWvXxwm7KC8sN93Eqm5ihIBVQ55IsgFUu6eY9hYQU1N23OcI66THIuRgLL8+GORyy8X6d3bKm70aJGpU2MUFMUdFnUkeJTrNeispIAqh7zhpQCyyERxu2dtbdtgcdDqaEHTcXhdH0V4hmDFCpEf/1ikVy+rmDFjRKZPT+AZhQ3se82BFUb5pZXqqygeqHLIgjgWQFYup+IsomIrIWzcwStjKuzUEiW0+aOPRC69VKRnT+vyY44RmTEjwWcUluJn0NCQnOWh01coCaPKodzEtQD8BEO58POx+wlyr7ZFdSdFtJY+/FDkoous+K4xIscdJzJzZlIPIyFKsTzUclBSRJVDuYlrAQRl/JSDIKEdVaBHCdIGKRoHH3wgcsEFIt27W7cYO1Zk1qzkH0dJxLEeo04HoigloMqh3ARlAEVd07lcloPfHEhxp3+IYjmEyPVftkzkRz8S6dpVpKpK5IQTRGbPTuFZlEqp8SNNaVXKgCqHchM23z3OKOMs6g2tA9Re5xSv1FaIYbhN49G9e6S2vveeyA9/KNKli6UUTjpJZM6copPyJFB1DiSlAshUOQCTgQ+BWY59VwBLgRn2doTj2CXAPOAt4NAw98idcoibtZL1vDlRXUBuCsTNJVJT0zb7KeRgwCVLRM46S6RzZ8u7Nn68yNy5LnVvbPRf3CgN/ASzzp6qVABZK4ev2OtQFyuH813OHWIvRdoZGAzMB6qD7pE75SDSVnAE9ba9rivnHz1O8DjsVtxjDuhZv/uuyA9+YMn3Tp1EvvMdkXnzfOrulSrrHKiXJEGCuRyWQ9aWplLxZO5WAgaFVA6XAJc4Pj8OfDmo/Fwqh2Ly9EeOMs4iqc1NCboI10U3/llOP32LUvje90QWLAjRJr97p0HQ9+k2cDDp+ZPyNhWLUnH4KYcqsuNMY8xMY8xkY8xW9r7tgcWOc5bY+9pgjJlgjJlqjJm6fPnytOtaOhMnQrdurfd162btLydNTTBhAixaZImSRYusz01NMG4c3HYbDBwIxlivdXXJ3Le+vvXnonst3P5/+P6+M9jpgm9y++1w6qkwb551yuDByVQhNk1NMGgQVFVZr01N8O677uc69xvT+ljx51IpfqZB+xUlCl5aI8mNtpbDtkA1UAVMBCbb+38NnOg473bg2KDyK8JyEImf2pikmymqBZOENeHjB58/X+S737WshNpakTPOEFm0KKANbs/EbwR2YT2KOHi5j4JGjMd5znF+GxpzUEqAvLmVvI5R6W6lpAV5Gn/+OK6IMOs9eG0ek8a9/bbIqadacrtzZ5EzzxRZvFjcn2GYEdwNDe7zPzm37t2jPzu/SQfjLt0aZhR58VTfXr+nKL85zWxSisidcgD6O96fC9xrv9+N1gHpBVRKQDoNQZ5GnCJonIJffcMoiIJQ9Chr7lyRk0+2lEKXLiLnnCOydKl90O0Zuq1o51f3oDpG/U78lKmfsI2SzuxlhSS5Wl65rAxVQBVFpsoBuAdYBmzEiiF8F7gbeB2YCTxSpCwuw8pSegs4PMw9cqEc0hDkYXv5UXuPcUY6+9WnUCefe7/5psiJJ1pjFLp2FTn3XGvsQitKzZYqEHRedXXpU3YHfa9R0pmjbHF/T+VIiFA3V8WRueWQ9paJcoibqhqFMH/oOH/IOCOdw9aniNmzRb79besxdOsmct55Iu+/73FyKUKz0JMX8Z8HKooyjPt83Z5zlDoFtTMO5chsylNGnhIKVQ5JU+rUzGFpaEhnVHWQcvASGBEE5axZIscfbxXVvbvIhRda8yH5UqoALbS5oSHedUHPK46rJExA3y1+4fV7qquLV5dyCG5Nra04VDkkTVj3R9I+YmPaZt1E/UOGEVZhFIuHcJo5U+TYY61ievQQufhikeXLQ7Q1iQF4zjY3NFg+rKjXJU3YGEjxc3Wb6tst/hLFiknb5aOWQ8WhyiFp/HzHxdNExCXsHy3qHzLpgK3NjBki3/qWVUTPniKXXWatsRBI2J61U3AGpZEWl1+41ssySVN4hVmG1YtiheHVbrclS8OUl0YwWmMOFYUqh6SJ46+PSpRgdJQ/ZCnCqpjGRpm23RFyNA8KiPTqul5+8hNrNbYw14ayFtza4qVQgtZazkJ4JdmbDlqHOy8KQrOVKgZVDknT2Oj9J03KRRFmegZn7n9YiyXibKhevPqzx+So6kcFRPrwsVzBT+WTrv3juzjCKiunUnETlkFCsqFhiwVRyuC4sPjFp6IKzyBlGieDKmnlqMqholDlkAZRXBtx8Psjx/2TjxrlXueqqtB/4pdfFjniCOuyrVghP+PHspJe0dofxmLwchGFUSpebpas3B5+Ci3qGg9hl2x1I+2YgLqVKg5VDmmQZS8sboZS1F66g5deEjnsMOvUvn1FJnKprKJndAElEpzv7xZ492u32+b2XWQdME3i/o2N8WMnaWcTZf18lciockiLrEzoOH9yv7mHfATqCy+IfP3r1uG6OpGrrxb59FOJ5vYKO3o4SLjHWXK01OeWJEndP27HJG3hnfXzVSKjyqG9EfVPHmQ1uJTx/PNbvFD9+olce63I6tVFZcZ1e0WJOTgJo+D8hFLY5+al2ErtDCQpnOPUJW1rVy2HikOVQ3sj6p88gjvmWQ6UAw+0Pm6zjcj114usWeNTj7huL+e1YYR7Y6P7HEtVVd7jGdyEftBz8xp46DbuIKpgLbdP3u37SdPa1ZhDxaHKoT0S5U8e4I5pAXmag+QrPCcgst12Ir/8pchnn8WsW1T3Qhhl4nVOIX01rFDye26Njd51T2qMRLlckVkH3zVbqSJQ5dDR8RCsLSBP8DU5gOcFRL5glspNJ70qayffU1qPM47bK0iQBSmcJFxBcUZp59Wfri4eJQSqHNIgiR5SRr3IFpC/dx4tX952voDI9iyWX/f9sayb/Ed3QV1b676Ggld94/Ra3cYfOJ9PnJ57kgMEsxhdXQoaHFZCoMohaZIw2ZM2+4MUTWOjtNQPlL9yhOxbO11AZIcdRG65ReTzzx3nRek9BwnmsIrP7VmEWcch6HlF7T17nV9Ira0kf7paDkoIVDkkTRJ/vKQzV3wEV0uLyF/+IrL33vYtqhfLrUyQz+t3Li1dNO38eK8efFhLK8pqbF7P0TnmopL86RocVkKgyiFpkjDZkyjDOfLWZWupHygPPSSy557WrkH9VsvvahtkPQ4XUdgpwL0EbBICM+r4hbD4WQJeQrOSFEAQ7aktSiqockiaclkOQZk1HmMFmjHyJ74pu/MfAZEvflFk8mSRDfVfDHfPMDEHY6yBEEn0TqNaDk6iPqM01t3IkuL2h1l3WlFsVDkkTTliDkHHXQRqM0Ye4BgZzgwBkZ06LZA77hDZuNG+Z1hrxS047Jb/72dRlPo8/TanxRQ0V1Gx8PQqsxIDtWGem7qSFB9UOaRB2tlKQZaFQyhuokru5X9lN14XENmFOXJ37Xdk451Nre8X1lpxU0pRRifHEbRhMpPAfdW0KMqpPQVqw1pcldg2pSyocvAjr37ZoF7+wIGyiSr5I2Plv3lDQGRXZksT35ZN9YO9B4D5TXkRJ8/fS4CX8kyDFlMqRTm1p0BtWEuuEq0ipSyocvAiz4LCp4e7caNIY8MLsqt5U0BkCLPkHo6XTZ06t17Xwc3/3NjYWsDW1bmnaZayRRkTEaXthbaV2lPOokOQxj3VclBKRJWDF34uhnLPS1OMi+La2LWn3Pn9F2Xnna1dQwd8Ivdv3SDNVFmCM8y4ADdFEEbghnHnJCWg/JR2kEDMi3J3klYnRGMOSolkqhyAycCHwCzHvr7Ak8Db9utW9n4D/AqYB8wE9gxzj9jKIWjZxeLecNzF3f0IyrYZOFA2UCN/qDtPdtp2lYDI8OEiU6aINDc7ygnbi/Tz53ttxrRVjnV1bS2EoDLCttvvuF8GUp7cgk68vpsoYza80GwlpQSyVg5fAfYsUg7XAhfb7y8GrrHfHwH8zVYS+wEvh7lH4pZDFAFaiske0KPcsEHk978X2XFH69Aee4g8+GCRUigQdaxA1C3Ikgp6ZmGC3mEFWV7jRF6E+W60h69kQOZuJWBQkXJ4C+hvv+8PvGW/vxX4ttt5flviMYeoveoo9ysoJB9hur5+J7ntNpFBg6xdI0aIPPywNdLZtbwwwrmweU1vHRQELiWQXRhlHHRumr7xLBWKxgaUnJJH5bDS8d4UPgOPAgc4jj0N7OVR5gRgKjC1vr4++lNxE9TOWEPSlkMI//Dn1Mpv+b7Us1DAmu7i0UddlELI8jyVg5t7rKHBPRPIL401ShwirPJKg6wTD8J+V0lkFVWaVaVkSq6Vg/35E4moHJxbZMshzgC0UmMOPgpnHZ3lNzTIAN4VENm3dro89piHUghR3mZh7BegLlaIzmdTLFzSdlk5hWMawiwPYxvCWHml1idrJahUHHlUDtm6leJOXVFKr8xFwK6js/yKM2V7FguI7M8L8njno6Tl7hDlJiGw3QSHWxuTGv8Qdku6x5u36avTirnkQQkqFUUelcN1RQHpa+33RxYFpF8JU35k5RB34ZhScPxx19JFbuRs6c9SAZGR/EOe4mBpqY9wr6QEdphAcdLjIOIqrjg0NpZvLYYov5u4vzE/xZI3Jajknqyzle4BlgEbgSXAd4E622X0NvAU0Nc+1wC/AeYDr4dxKUkc5RA0viGlnPTPutbJ/+Nc2ZZlAiJf5Vl5hgOlBUcqZthUxLgxBz/B4fVcqqqiza2U1FZXV/Iz93xGSbtbyuXS8fvtquWgRCRzyyHtLdGYQwp/sDVrRK67TmSbXmsFRA7mKXmu6qAtwjlqDzpk1tNmwR4mQ6ix0b+cTp38j0fJmApbJpQmXP1SlcsptJPEzzrQmIMSEVUObniZ9UELxET4o61eLfKLX4hsvbV1+de+JvLPfzpOiJPiGCf91u+6QiwlidHPUSyLurpw8YygOJAf5XSz+LU9SeEcpIQ0W0mJgCqHKCQwPcOnn4r83/9tyQA99FCRF190nBAnyBuU6RLGr+4lOKLMuFqKoPd6hn5WizMOFLVXXI7efLmn9GhocL9HYbU6RYmAKocohOlFewiXlStFfv5zkb59rdMOP1zkpZdilO8mIMMK3ahCKcidFHYLG7guLBJUPPlf9+7+zzqOoE/bzRLlu0xKIWlcQUkQVQ5RCeoNFrklPvlE5MorRfr0sQ5/4xsir7ziUXZaaaHOtNMoLoUk61MIqMeZv6lTJ//ZXOO6iNJ0s0R5dkm5sjQjSUkQVQ5xCeilffyxyE9/KtK7t7V79GiRqVM9ygo7zYTfZ68trIUQNcZS2KK4nUqJXfitA5HHHnOUGEu5LQeNPSghUOUQFw+3xIpJ98uPfyzSq5e165vfFJk+3XGN2+C5oBlMvXr+fhk3YQfoxcnOgnDC2bl5zdsUdqu0RXq8nknQ0qWlEOY55PFZKblElUMpOITu8gFfkkuOel169LCe3DHHiMyYUXSu25/Sy58e5o/rVmbxFNV+g9eC0lj9Bk8FZUk5t6C1JLwEZ5Tedd56w0HPPa16Bj2HPFpZSi5R5VAiH34octFFlow3RuR//1fk9dddTozjvw8jPJw9fLdeqZfbJ8jtUeiph+nJNza2vk+PHq1XnfNzPTmtnIYGdyuqtjacEM2jgshTfUQ0LqGERpVDTN5/X+T88y35a0yLjO32sMxiN28hEGcEcRSSDmaHzQQK46aIkudfrGgKqbBBqLskHGo5KCFR5RCRZctEzj1XpGtXy40+bv8FMrvLHsFCKarwdk4PEaYHmuT0FUGz0DqP+7WrkF/vdU6pU2CEeb55EXp5sSJUiSohUeUQkqVLRc45R6RLF0spnHSSyJw5Ei1DJGy2jtONEiauID71iLpVVXlPyVFYAtTpMgoqr7CQT6kCKUi4JuEuSUuApyWQ49Y3L4pKyTWqHAJYskTkrLNEOne23OOnnCIyd67jhChCyfmn9BOoUTKBkprmwrl5Eece1dVt2x5VIIURrmEzt0q5R1zSsGrUAlBSxk85GOt4ZbPXXnvJ1KlTI1+3ZAlcfTX8/vfQ0gInnwyXXgpf/GLRiYMGwaJFbQsYOBAWLvS+gd91EyfCZZfBu+9af/sgCvdqatpyXX09rFkDK1YEX19MYyOMG2e9d5ZZVQXNzdHLK/V3FOYZNzXBhAmwdq13Od26wW23bWlb1HvEparK/RkYY/244pBmfW02btzIkiVL+PzzzxMpT8knXbp0YcCAAdTU1LTab4yZJiJ7uV7kpTUqaYtrOUyZYg3M/d73RBYs8Dkxbg/OL9UxqbWq405/kaQ1UrAcSiGsdea0TqKu05BmFk8alkMZso4WLFggy5cvlxbfZQeVSqalpUWWL18uC1yEHOpWcqe5WWTRopAnR3WZONNPS1mj2imAve4Zd+K8gQOTiWMkMelbmPEYxUQVnmkGtNNwAZUhAD979mxVDB2AlpYWmT17dpv9qhzKTZCgiJt1ZIy7II7b+zcmXF3CTAke9rl4KcygUeRuo4CjWg5p+/CTDgKXIebgJjCU9okqhzzg1eMrdXqJgkB3Ew5hA+HFQjRskNdvqvAw+Cmw2lpL6QWNsi4Ifb9V6YKEZ6Vl8aRcX1UOHQdVDnkgjmWQ5CRuYVxFfjEHNwFb6joCQXUKozgLitHrWaWxwls7J7JySEFZVVVVye677y5DhgyR4cOHy/XXXy/Nzc2+17zzzjvS1NRU8r07Eqocyo3bnyWqH78wrURYBRFmmupigV9b23rsQrGLxmuMg3MsRvGynp06hRcOSQzgC4qR6PQQ/rj8ViMph5TcXN27d9/8/oMPPpBRo0bJT3/6U99rnn32WTnyyCNLum9HQ5VDOUkiG8kp0MIqiDABySg9vKC5m/wUXtjgaKmB70I9Spm4ryPj8Vud/eqr4ctIKUDuVA4iIvPnz5e+fftKS0uLvPPOO3LAAQfIHnvsIXvssYe8aC+puO+++0qvXr1k9913lxtuuMHzPGULqhzKid+fJawF4Tf1dmH614R7apuJooziDgT0mzm2eCsOSEcZJe4Vi4lLUu6TvMQ4PJ7b7CeeCF9GSqm1xcpBRKR3797y/vvvy2effSbr1q0TEZG5c+dK4b9ebDl4nadsoaKUA7AQeB2YUagk0Bd4Enjbft0qqJxyTdnd5s8d5s8SxZ3iJvjTEi5eMQSv9oRNNfVzPTQ2+qfderm+it1ebkokyTWUk3Kf5GmEs8fvcPbf/ha+jDJZDiJblMPKlSvlxBNPlKFDh8ruu+8uXbt2FZG2ysHrPGULlagcti7ady1wsf3+YuCaoHLKvdhPYC/WuaJZ1CUzy+EaiTpwriCog9ZzCOt+ipKGGjV+kgRJCcE8TRSYhOVQhpiDSGu30uWXXy7nnXeeNDc3y8aNG6Xazo4rVg5e5ylbiKocqpIYmp0wRwN32u/vBMZkVpPLLms7VcPatdZ+sKbA6Nat9fGaGli92pr2QMR9Koria5y8+268ujY1WdMtVFVZr01N3udNmBC+3G7drHaOG2dNSzFwoPt5GzZY84+4TfcArds1bpz3lBLF7Xf7DjZsgJUrw9U/Dl7fQdTvJqlyksDtt9qtG2y1VfgynL8BY6xXr6lKYrJ8+XJOP/10zjzzTIwxrFq1iv79+1NVVcXdd99Ns/1/6tmzJ6tXr958ndd5Sgl4aY1ybMA7wHRgGjDB3rfScdw4P3ttqVkOYdxGxW6foNHKQfGIOL3KKD26KIFhr9RQv2vCrCrnV4/i88IO0kvSemiPloNI6dlKKVGcynrddddtTmWdO3euDBs2TIYPHy4XXnjhZitjw4YNctBBB8nw4cPlhhtu8DxP2UKluZW2t1+3AV4DvlKsDIBPPK6dAEwFptbX1yfy8NoQ588dNpvGbURwTU08IRelnmFjIH4CN+jaMGsoBym0qCnBSQrc9hhz8CAPykEpDxWlHFpVBK4AzgfeAvrb+/oDbwVdm1nMwY2wefiNjW1HBLstlRkmIO1n4USxbIqzo6qr3QO9YeZyChNE92pbnOlAkh7j0N6ylTxQ5dBxqBjlAHQHejre/ws4DLiO1gHpa4PKyixbyev8MK6VsIHbMMrJLzBefH1NTVulVCgz7CjooHmQSu3F+ynYqPMpKb6ocug4VJJy2NF2Jb0GvAFcZu+vA57GSmV9CugbVFbucprdxg+EXXPZ2QP26qGHyejp1s2/h1888Z1ItPmTvNJSw7pN4qYIV4CrppJQ5dBxqBjlkOSWO+UgEmxxBFkOfummYQeeBcUXioWq37lx2+l1TZwUYeezybGrppJQ5dBxUOVQKQRZF0GDzsIQJqDrLKvUmVfDEkb4q3VQFlQ5dBzawziH9k9TE9x5pyX2ChgD48dvyRn3y4WfODHcfdxy24tx3sdr/EOUcRFhCMr/L0M+vaIo/qhyyAK3gV0i8NhjWz7X17tfW1cXXkgGDVwrvs8tt0BDA1RXW5+rq63Pt9wS7n5h8Wqbc/+4cdY6yS0t1qsqhnZLjx492uybNGkSd911Vwa1yRePPPIIv/jFL7K5uZdJUUlbxbmVwg6uS9K1kidXTZJjPJSSyINbKYsBay0tLb5rRmzatKmk8jdu3FjS9WmgbqVKIGzPOUnXSt5cNcb4f1bKzg9/CAcemOz2wx/Gq8sVV1zB9ddfD8CBBx7IRRddxD777MMuu+zCP//5TwCam5u54IIL2HvvvRk+fDi33norAGvWrGHUqFHsueeeDBs2jIcffhiAhQsX8l//9V+cfPLJDB06lMWLF7e656BBg7jooovYc889eeCBB5g/fz6HHXYYI0aMYOTIkcyZMweA+fPns99++zFs2DB+/OMfb7Z8nnvuOUaOHMno0aMZMmQIAI2Njeyzzz586Utf4vvf/z7Nzc00NzdzyimnMHToUIYNG8Yvf/lLAH71q18xZMgQhg8fztixYwG44447OPPMMzfX/+CDD2b48OGMGjWKd2037CmnnMLZZ5/N/vvvz4477siUKVPiPfQiOiVSihKNiRMtP77TtVSYw8jJuHHJCu+ky4vLZZdZ8yM52bDB2p+H+im5Y9OmTbzyyis89thjXHnllTz11FPcfvvt9O7dm1dffZX169fzP//zPxxyyCHssMMOPPjgg/Tq1YuPPvqI/fbbj9GjRwPw9ttvc+edd7Lffvu53qeuro7p06cDMGrUKCZNmsTOO+/Myy+/zBlnnMEzzzzDOeecwznnnMO3v/1tJk2a1Or66dOnM2vWLAYPHsybb77Jfffdx4svvkhNTQ1nnHEGTU1N7LbbbixdupRZs2YBsNKeJ+wXv/gF77zzDp07d968z8lZZ53F+PHjGT9+PJMnT+bss8/moYceAmDZsmW88MILzJkzh9GjR3PssceW/MxVOWRBQQBedpkVhK2v3zK5XUcgTxPSKZu58casa+DNt771LQBGjBjBwoULAXjiiSeYOXPm5p7yqlWrePvttxkwYACXXnopzz//PFVVVSxdupQPPvgAgIEDB3oqBoDjjz8esKyPf/3rXxx33HGbj61fvx6Al156abNQPuGEEzj//PM3n7PPPvswePBgAJ5++mmmTZvG3nvvDcC6devYZpttOOqoo1iwYAFnnXUWRx55JIcccggAw4cPZ9y4cYwZM4YxY9rON/rSSy/x5z//GYCTTjqJCy+8cPOxMWPGUFVVxZAhQza3tVRUOWRFXnrx5aSpyVKIziwtJ17uNqXD07lzZwCqq6vZtGkTYMVLb775Zg499NBW595xxx0sX76cadOmUVNTw6BBg/j8888B6N69u+99CsdbWlro06cPM2bMiFRPZ/kiwvjx47n66qvbnPfaa6/x+OOPM2nSJO6//34mT57MX//6V55//nn+8pe/MHHiRF5//fXQ9y08n8J9k0BjDkp5KEwV7jWlt5tbTVF8OPTQQ/ntb3/Lxo0bAZg7dy6fffYZq1atYptttqGmpoZnn32WRV6/OR969erF4MGDeeCBBwBL4L722msA7LfffvzpT38C4N577/UsY9SoUUyZMoUPP/wQgI8//phFixbx0Ucf0dLSwjHHHMNVV13F9OnTaWlpYfHixRx00EFcc801rFq1ijVr1rQqb//99998v6amJkaOHBm5XVFQy0EpD27puwUGDuxYbjWlFWvXrmXAgAGbP//oRz8Kdd1pp53GwoUL2XPPPRER+vXrx0MPPcS4ceM46qijGDZsGHvttRe77rprrHo1NTXR0NDAVVddxcaNGxk7diy77747N954IyeeeCITJ07ksMMOo3fv3q7XDxkyhKuuuopDDjmElpYWampq+M1vfkPXrl059dRTabHXNLn66qtpbm7mxBNPZNWqVYgIZ599Nn369GlV3s0338ypp57KddddR79+/fjDH/4Qq11hMUmZIFmy1157ydSpU7OuhuJHVZW7O8kY74V/lNR58803+e///u+sq1FRrF27lq5du2KM4d577+Wee+7ZnBGVZ9y+a2PMNBHZy+18tRyU8lBf7+5S0jiDUmFMmzaNM888ExGhT58+TJ48OesqpYIqB6U8hE3fVZScM3LkyM3xh/aMBqSV8pC3QXjKZtqDa1nxJ853rJaDUj46YvpuzunSpQsrVqygrq4Oo6PU2yUiwooVK+jSpUuk61Q5KEoHZsCAASxZsoTly5dnXRUlRbp06dIqIywMqhwUpQNTU1OzeUSvojjRmIOiKIrSBlUOiqIoShtUOSiKoihtaBcjpI0xywHnCKutgY8yqk456SjthI7TVm1n+yPPbR0oIv3cDrQL5VCMwqbQ4gAABTBJREFUMWaq15Dw9kRHaSd0nLZqO9sfldpWdSspiqIobVDloCiKorShvSqH27KuQJnoKO2EjtNWbWf7oyLb2i5jDoqiKEpptFfLQVEURSkBVQ6KoihKGypWORhj+hpjnjTGvG2/buVx3t+NMSuNMY8W7R9sjHnZGDPPGHOfMaa2PDWPRoR2jrfPedsYM96x/zljzFvGmBn2tk35ah+MMeYwu37zjDEXuxzvbH8/8+zva5Dj2CX2/reMMYcWX5s34rbVGDPIGLPO8R1OKnfdoxCinV8xxkw3xmwyxhxbdMz1d5xHSmxns+P7fKR8tY6AiFTkBlwLXGy/vxi4xuO8UcBRwKNF++8HxtrvJwENWbcpbjuBvsAC+3Ur+/1W9rHngL2ybodH26qB+cCOQC3wGjCk6JwzgEn2+7HAffb7Ifb5nYHBdjnVWbcppbYOAmZl3YYE2zkIGA7cBRzr2O/5O87bVko77WNrsm5D0FaxlgNwNHCn/f5OYIzbSSLyNLDauc9YE9cfDEwJuj4HhGnnocCTIvKxiHwCPAkcVqb6lcI+wDwRWSAiG4B7sdrrxNn+KcAo+/s7GrhXRNaLyDvAPLu8vFJKWyuJwHaKyEIRmQkULx5eSb/jUtpZEVSycthWRJbZ798Hto1wbR2wUkQ22Z+XANsnWbkECdPO7YHFjs/F7fmDbb7+JGfCJqjerc6xv69VWN9fmGvzRCltBRhsjPmPMeYfxpiRaVe2BEr5XirpOy21rl2MMVONMf82xuSyY5rr9RyMMU8B27kcusz5QUTEGFOxObkpt3OciCw1xvQE/gSchGXmKpXDMqBeRFYYY0YADxljdhORT7OumBKbgfb/ckfgGWPM6yIyP+tKOcm1chCRr3kdM8Z8YIzpLyLLjDH9gQ8jFL0C6GOM6WT30AYAS0usbmwSaOdS4EDH5wFYsQZEZKn9utoY80csczgvymEpsIPjs9v3UDhniTGmE9Ab6/sLc22eiN1WsZzU6wFEZJoxZj6wCzA19VpHp5TvxfN3nENK+v05/pcLjDHPAXtgxTByQyW7lR4BCtkM44GHw15o/9meBQoZBJGuLzNh2vk4cIgxZis7m+kQ4HFjTCdjzNYAxpga4BvArDLUOSyvAjvbmWO1WEHY4swNZ/uPBZ6xv79HgLF2hs9gYGfglTLVOw6x22qM6WeMqQawe5o7YwVr80iYdnrh+jtOqZ6lEruddvs62++3Bv4HmJ1aTeOSdUQ87obli30aeBt4Cuhr798L+L3jvH8Cy4F1WH7BQ+39O2IJk3nAA0DnrNtUYju/Y7dlHnCqva87MA2YCbwB3ETOMnqAI4C5WL2my+x9PwNG2++72N/PPPv72tFx7WX2dW8Bh2fdlrTaChxjf38zgOnAUVm3pcR27m3/Fz/DsgLf8Psd53WL205gf+B1rAyn14HvZt0Wt02nz1AURVHaUMluJUVRFCUlVDkoiqIobVDloCiKorRBlYOiKIrSBlUOiqIoShtUOSiKoihtUOWgKIqitEGVg6KkgDHmWWPM1+33Vxljbs66TooShVzPraQoFczlwM/sxZX2AEZnXB9FiYSOkFaUlDDG/APoARwoIquDzleUPKFuJUVJAWPMMKA/sEEVg1KJqHJQlISxp1ZvwloZbI0xJq+rmSmKJ6ocFCVBjDHdgD8D54nIm8DPseIPilJRaMxBURRFaYNaDoqiKEobVDkoiqIobVDloCiKorRBlYOiKIrSBlUOiqIoShtUOSiKoihtUOWgKIqitOH/A2fl0Iy3iVIOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.plot(X_train, y_train, 'r.', markersize=12)\n",
    "X_edge = np.array([np.min(X_train, 0), np.max(X_train, 0)])\n",
    "plt.plot(X_edge, model.predict(X_edge), 'b-')\n",
    "plt.legend(('Data', 'Linear regression'), loc='lower right')\n",
    "plt.title('Linear regression')\n",
    "plt.xlabel('$x$')\n",
    "plt.ylabel('$y$')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once trained, the model can be used to make predictions on the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make predictions using the testing dataset\n",
    "prediction = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step (not shown here) is to evaluate the performance of the trained model.\n",
    "\n",
    "Note that the `scikit-learn` interface works by first initializing an object from the class that implements the machine learning model (linear regression in this case) and then fitting the initialized model using the data in the training set. Finally, the trained (fitted) model can be used to make predictions on unseen data. In fact, all models implemented in this library follow the same *initialize-fit-predict* programming interface. For example, a $k$-NN classifier can be trained in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import datasets, neighbors\n",
    "\n",
    "breast_cancer = datasets.load_breast_cancer()\n",
    "\n",
    "X = breast_cancer.data\n",
    "y = breast_cancer.target\n",
    "\n",
    "# make use of the train_test_split() utility function instead\n",
    "# of manually dividing the data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=40)\n",
    "\n",
    "# initialize a 3-NN classifier\n",
    "model = neighbors.KNeighborsClassifier(n_neighbors=3)\n",
    "\n",
    "# train the model using the training dataset\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# make predictions using the testing dataset\n",
    "prediction = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the features in the breast cancer dataset have different scales (some have on average very small absolute values, and some very large), which means that the distance metric used by $k$-NN will me dominated by the features with large values. You can use any of the number of feature transformation methods implemented in `scikit-learn` to scale the features. For example, you can use the `sklearn.preprocessing.StandardScaler` method to transform all features to a have a zero mean and unit variance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)\n",
    "\n",
    "X_train_scaled = scaler.transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scaler has its own parameters which are the means and standard deviations of the features estimated from the training set. If you train a model with the scaled features, you will have to remember to also apply the scaling transformation every time you make a prediction on new unseen and unscaled data. This is somewhat prone to error. One option for making the code more robust is to create a processing pipeline that includes the scaling and $k$-NN models in a sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "knn = neighbors.KNeighborsClassifier(n_neighbors=3)\n",
    "\n",
    "model = Pipeline([\n",
    "                 (\"scaler\", scaler),\n",
    "                 (\"knn\", knn)\n",
    "                ])\n",
    "\n",
    "# train the model using the training dataset\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# make predictions using the testing dataset\n",
    "prediction = model.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are curious, more information about the design of the `scikit-learn` application programming interface (API) can be found [in this paper](https://arxiv.org/pdf/1309.0238.pdf). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "### Bias-variance decomposition\n",
    "\n",
    "Show that the mean squared error of the estimate of a parameter can be decomposed into an expression that includes both the bias and variance (Eq. 5.53-5.54 in \"Deep learning\" by Goodfellow et al.). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Polynomial regression\n",
    "\n",
    "For this exercise we will be using generated data to better show the effects of the different polynomial orders.\n",
    "The data is created using the make_polynomial_regression function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x134c84e10>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZzklEQVR4nO3dfawcV3nH8d9jxw640GJfu6lJcu9NUFoalcqBqwgVCRJANLWqJKiUBl2oaVOZXKAKLUgN9R9ElaJSVIioWkINSZPmWgEaoLhqEApJEEIlaa+jvEchBuyU1Ngm5lWmKdhP/5hZZ7ye2ZnZPTM7Z/f7kVa7d3Z259kzu8+ce86ZM+buAgDEZ9W4AwAADIcEDgCRIoEDQKRI4AAQKRI4AETqtDY3tnHjRp+fn29zkwAQvT179nzP3Tf1L281gc/Pz2tlZaXNTQJA9Mxsf95ymlAAIFIkcACIFAkcACJFAgeASJHAASBSJHAAaNKuXdL8vLRqVXK/a1ewt251GCEATJVdu6Tt26WjR5O/9+9P/pakxcWR354aOAA0ZceO55J3z9GjyfIASOAA0JSnnqq3vCYSOAA0ZXa23vKaSOAA0JTrrpPWrTt52bp1yfIASOAA0JTFRWnnTmluTjJL7nfuDNKBKTEKBQCatbgYLGH3owYOAJEigQNApEjgABApEjgARKo0gZvZ2WZ2j5k9ZmaPmtnV6fJrzexpM3sgvW1tPlwA6JgG5zopU2UUys8lvdfd7zezF0raY2Z3ps9d7+5/21x4ANBhDc91Uqa0Bu7uB9z9/vTxjyU9LunMpgMDgM4rmuvk6qtbqZXXagM3s3lJF0i6L130bjN7yMxuMrP1Ba/ZbmYrZrZy+PDhkYIFgE4pmtPkmWeS2rj7c7XyBpJ45QRuZi+Q9FlJ73H3H0m6QdJLJG2RdEDSh/Ne5+473X3B3Rc2bdoUIGQA6Iiqc5oEnIEwq1ICN7M1SpL3Lnf/nCS5+0F3P+buxyV9QtKFwaMDgC7Lm+ukSKAZCLOqjEIxSTdKetzdP5JZvjmz2hslPRI8OgDosry5TmZm8tcNNANhVpUa+KskvU3Sa/uGDH7IzB42s4ckXSzpz4JHBwBdt7go7dsnHT+e3H/0o43OQJhVOozQ3b8myXKeuiN4NAAwCZ7//OdGp8zMJEm9gWGFzEYIAKH0jwuXpJ/+tLHNcSo9AITS8DUw+5HAASCUhq+B2Y8EDgChbNhQb/mISOAAECkSOACEcuRIveUjIoEDQChFJ+s0cBKPRAIHgHDyTq1v6CQeiQQOAOHknVq/c2djc4NzIg8AhLS42MrFHCRq4AAQLRI4AESKBA4AkSKBA0CkSOAAECkSOABEigQOAFXs2iXNz0urViX3DVxlvi7GgQNAmf4LNezfn/wttTbmOw81cAAo0/KFGqoigQNAmZYv1FAVCRwAyrR8oYaqSOAAECkSOACUaflCDVWRwAGgTMsXaqiKBA4AZVq+UENVJHAAKNPyhRqq4kQeAKiixQs1VEUNHAAiRQIHgKwOznlShCYUAOjp6JwnRaiBA0BPR+c8KVKawM3sbDO7x8weM7NHzezqdPkGM7vTzJ5M79c3Hy4ANKijc54UqVID/7mk97r7+ZJeKeldZna+pGsk3eXu50m6K/0bAOLV0RN2ipQmcHc/4O73p49/LOlxSWdKukzSLelqt0i6vKkgAaAVHT1hp0itNnAzm5d0gaT7JJ3h7gfSp74r6YygkQFA2zp6wk6RyqNQzOwFkj4r6T3u/iMzO/Gcu7uZecHrtkvaLkmzHf03BABO6OAJO0Uq1cDNbI2S5L3L3T+XLj5oZpvT5zdLOpT3Wnff6e4L7r6wadOmEDEDAFRtFIpJulHS4+7+kcxTuyVtSx9vk/SF8OEBAIpUaUJ5laS3SXrYzB5Il/2lpA9K+oyZXSlpv6Q3NxMiACBPaQJ3969JsoKnXxc2HABAVZyJCQCRIoEDQKRI4AAmx6CZBCOaZbAqZiMEMBkGzSQoRTXLYFXmnnv+TSMWFhZ8ZWWlte0BmCLz80li7jc3l9wXPbdvX5NRBWFme9x9oX85NXAAk2GYmQQ7OstgVbSBA5gMg2YSjGyWwapI4AAmw6CZBCObZbAqmlAATIZeZ+SOHUnTyOxskqCznZSDnosQnZgA0G/Xrk4lezoxAaCKiC5sTBs4AGRFdGFjEjgAZEV0YWMSOABkRTTkkAQOAFkRDTkkgQNAVkQXNmYUCgD0i+TCxtTAASBSJHAAiBQJHAAiRQIHgEiRwAEgUiRwAN03gdezDIFhhAC6LaLJpdpGDRxAt0U0uVTbSOAAui2iyaXaRgIH0G0RTS7VNhI4gG6LaHKptpHAAXRbRJNLtY1RKAC6L5LJpdpGDRwAIkUCB4BIlSZwM7vJzA6Z2SOZZdea2dNm9kB629psmACAflVq4DdLuiRn+fXuviW93RE2LABAmdIE7u5flXSkhVgAADWM0gb+bjN7KG1iWV+0kpltN7MVM1s5fPjwCJsDAGQNm8BvkPQSSVskHZD04aIV3X2nuy+4+8KmTZuG3BwAoN9QCdzdD7r7MXc/LukTki4MGxYAoMxQCdzMNmf+fKOkR4rWBQA0o/RMTDO7TdJFkjaa2XckfUDSRWa2RZJL2ifpHQ3GCADIUWUUylvcfbO7r3H3s9z9Rnd/m7u/zN1/090vdfcDbQQLYApw9Z3KmAsFQHdw9Z1aOJUeQHdw9Z1aSOAAuqPoKjv799OckoMEDiC8YduxB11lp9ecQhI/gQQOIKxeO/b+/ZJ7vcSbd/WdLJpTTkICBxDWKO3Y2avvFOFixieQwAGENepV5BcXpX37ipM4FzM+gQQOIKxQV5HnYsalSOAAwhqUeOt0bnIx41KcyAMgrF6C3bEjaTaZnX2u1lz3JB0uZjyQuXtrG1tYWPCVlZXWtgegQ+bnk6Tdb24uafNGITPb4+4L/ctpQgHQjlE7N3EKEjiAdoTq3MQJJHAA7WBUSXAkcADtYFRJcCRwAO3pnaRz/HhyPyh5My94KRI4gPaVJedR5lOZIiRwAO2qkpyZF7wSEjiAdlVJzgw5rIQEDqBdVZIzQw4rIYEDaFeV5MyQw0pI4ADaVSU5M+SwEiazAtCuosmu+pMzE1mVIoEDaB/JOQiaUAA8h5NnokINHECiNz67znzdGCtq4AASnDwTHRI4gAQnz0SHBA4gwckz0SGBA0hw8kx0SOAAEpw8E53SUShmdpOk35V0yN1/I122QdKnJc1L2ifpze7+/ebCBNAKxmdHpUoN/GZJl/Qtu0bSXe5+nqS70r8BAC0qTeDu/lVJR/oWXybplvTxLZIuDxwXAKDEsG3gZ7j7gfTxdyWdESgeAEBFI3diurtL8qLnzWy7ma2Y2crhw4dH3RyAJnEqfVSGTeAHzWyzJKX3h4pWdPed7r7g7gubNm0acnMAGhf6OpQcDBo3bALfLWlb+nibpC+ECQfA2IQ8lZ6LErfCkhaQASuY3SbpIkkbJR2U9AFJ/yrpM5JmJe1XMoywv6PzFAsLC76ysjJiyAAasWpVkmz7mUnHj9d7r/n5JGn3m5uT9u0bJrqpZmZ73H2hf3mVUShvcffN7r7G3c9y9xvd/Rl3f527n+fur6+SvAE0pL+p4p3vHK7pIuSp9EXzp+QldQyNMzGBmOU1Vdxww3BNFyFPpS9K+mY0owREAgdiltdu3a9qO3bIU+mvuy55j37uTE8bUGkbeEi0gQOBFbVb9xumHXtUeQl8XLFEbug2cAAdVrV9ehxTws7N5S9netpgSOBAzPLarfsVtWM3PU6b6WkbRwIHRjXOE1by2q2XlsrbsdsYp830tI2jDRwYRf+FgKWkltn1RMU47ajQBg40IdYLAXP9y4lAAgdGEWsi5PqXE4EEDoyiSiJsqo28zvv2r7t1Kx2Mk8DdW7u94hWvcEyh5WX3uTl3s+R+eXncEYWzvOy+bp170hWY3NatS5YvL7vPzJz8XPb5prZbdd2lpcndLxNG0orn5FQ6MdGsWDv56ti1K2nzfuqppObdq8X2f+6sUTsL63RC0mEZvaJOTBI4mjWtyaPoc/eMejbioDMw5+aSJpI77kgOKkXrcUZkNBiFUgcT0YcTayffqMo+36idhYNe3z+hVVMxYOxI4P26NBH9JBxImhrt0PWy2bCh+LkQnYVVzsAchA7LyZDXMN7ULYpOzLm5UzudpGR5m+p0UnVZE5+j62WzvOy+Zk3+92hmJlycvc7hvO0U3UbpsJzkzuiOU0EnJgm8n1nxF79NIQ8k4/7hhd7+oLIZ92cdFN/MTLvbC1kJ6fpBc8KRwKvqSg081IFkEn94RWXT+2zj/qxtVwLy9nHetkcph678LqZUUQKnDbxfV2ZQC9V2HOup3oMUlcHq1d34rG2f5ZidNEo6dR5uM+mqq0YbtjmtndEdRwLv15UZ1EIdSCbxh1dUNseO5a+/f3+7nZxbt9ZbHsLiYjIs01269daTv7+33ip97GOjvT+n3ndTXrW8qVsUTShdEqI9d1L/9c0rm0FtwW02pUximU9iU1xERBt4JEJ3wk3TD6+sLbitBNqVjvDQutBBPKVI4O6jfwGb/gI3lWyn6Ye3vFycwIdJoMOUXYga+DTtM5QigY+aHNuoyU7iv97jEKocy/Z5UZItm+CqLDFP039NqIQEPuqPuo3kOqn/erctVAIsG29eN7lXjYsDOfoUJfDpmcyqaPKfqhP6jPr6KqZ14qcm5M0QWHck0aB9Pjtbf19V3b9tfNcQFSazGnUYVBvDqLoyBj0GZXOh9IbVHT+e3A8zDHTQPh9meGbV1zBkDxVNTwIfNTm2kVy7Mga969qacGzQPh8myVZ9DQdyVJXXrtLULYpRKIPWmdaRAcN+7qbKq8024mE6Kge9V52r6Ezjdw25NPWdmFXQ+3+qpaVTO1erlEmTZVk0TFAa/b3rGCbJDjogkLBRgASeVTRCYPXqwTW7Jk6y6fKPdnm5eGRMWW23yVpy0X5avfrU+Ltcvj1UHFCCBN6T92NZs8Z97driWl1vJreQP7IYfrSDTk0vG9rY5JDIKjXwGMq3h2GDKNFIApe0T9LDkh4o2kD21okEXncC/N4PKfSPLIYf7aBpW8vmtm7y81V579Dbb7I2z/h/lCjKryFGoVzs7ls8Z4xiJ9Wdha/X+x96Vr8YZgkcZdha6JEU2WGDP/mJtHbt4PcOWb5Nj3opKudVq7p3qTh0S15Wr3pTUgPfWHX9KGvgS0uDXzfsVWBiqIGPOq9IqFprUbPXzEzxe4cs36b31aBJuLra7INWqaEmlG9Lul/SHknbC9bZLmlF0srs7GxrH7hQ3Tbw7BwWeW2qS0vDtbXG0kY7MzP+A80wCTRk+bbRxFGlEz27bgydswimqQR+Znr/y5IelPTqQet3ogbuHm4UytJS9R9d1Ti6pgsHmmETaKjybeu/pSqfM8T+iOF7h5M0ksBPeiPpWknvG7ROsATe1BewTqIom3t6kjqgxv2DH3dzU1sHsTY6Z7twQEZtwRO4pF+Q9MLM4/+QdMmg1wRJ4E1+Aev8OMra0rvUlh2r3oGjd0AcV9JZXj65KWlmppltV/luj9qcM+6DIYbSRAI/N202eVDSo5J2lL1mqATeX/sbpk22ag2yzsGh7Mro1GhGk7cvemXe5n8BbddYy76royZghixGqfEmlCq32gm8rJmiyhew7g+warIv+iGtXh1X8m6qeWTUOWW6UlNsI446+2DUA0pXyhW1xJnA6wz5K/oCNvWFnYS2xKY+w6D3rbrNrtQUm45j2Emxhj3oTsL3dgrFmcAHNVP033rjtau+R4gfYJ2mmS72+jd1cBv0vlW32ZWaYtNxjONzdvX7iEJxJvAu18Cr6nKNp6mD26D3rbrNrpRb03F05T8NdFpRAu/2BR3yTscuUnSKdJOT45ddFUZKLut19OjJy44eTZY3pUpcUnNXfhn0vlW32ZWLWzQdB1ffwSjysnpTt86PQqkbVxfbcuvUGIs6iUcdJheiDXxaDDrDl2YOpBRlE0qepaX8hFjUBj6sUMO52j4VvW6TUf8Y51BJddRRKNMk7wxfDnLImJwEHnpSqTxlNchBbfP9pz3nzbGyZk312Op+pmFq/OPuJ8DJ2B/oMzkJvOwEmhC1lqIf0MxM+bj0Kqc9l82l3TNMc8Og8elFBwE60rqF/YE+RQm8252YeYo6d1avDtdZWNQh+swzp24jq+qc1EeOVItjmA7Qoo7fY8eSNJA3lzUdac2o2pncj/2BqvKyelO3RudCqdKkUVWd4Yv9TThV3qeJ056zTS0zM8/NlV1ltkQ6FsMbpUzZH+ijiWlCcc9vFw7Zblj0AyrrkAzdGVX1Mw36wdcZdz2JHYvj+lwhZg2cxP2BoUxWAs8TutaS9wMaZnjcKMPBqn6mEGc+TqJx1mRpx0ZAk5/A3duptRRto8k5V8o+06BkMc3/jo/z4DXNB04ENx0JfJzGWeMqSxax/js+atzj3CfTfOBEcEUJPL5RKONQZTTBOEcOlE0XsLgo7dsnHT+e3Ld9OvowQlwJfpz7pCtTAWCy5WX1pm5R1sCr1qTGXeOqW1vteq08RBPEuPcJEIhoQhlSnUTS9aTYE0NiC9X8Ecs+AQYoSuCWPNeOhYUFX1lZaW17QaxalaSOfmZJk0SM5ueTJol+c3NJE0sXxBAj0BIz2+PuC/3LaQMvM4lnxRWdIVq0fByanAYYmBAk8DKTmEhiOCjRCQiUIoGXmcREEstBKcbRM0CLTht3AFFYXJys5NH7LDt2JM0ms7NJ8p6kzwhMARL4tJq0gxIwhWhCAYBIkcABIFIkcACIFAkcACJFAgeASLV6Kr2ZHZaUc350qY2Svhc4nBC6GpdEbMPoalwSsQ2jq3FJ9WObc/dN/QtbTeDDMrOVvHkAxq2rcUnENoyuxiUR2zC6GpcULjaaUAAgUiRwAIhULAl857gDKNDVuCRiG0ZX45KIbRhdjUsKFFsUbeAAgFPFUgMHAPQhgQNApDqTwM3s983sUTM7bmaFw2vM7BIze8LM9prZNZnl55jZfenyT5vZ2kBxbTCzO83syfR+fc46F5vZA5nb/5rZ5elzN5vZtzPPbQkRV9XY0vWOZba/O7O8kTKrGpuZbTGzr6f7/SEz+4PMc0HLreh7k3n+9LQM9qZlMp957v3p8ifM7LdHiWPI2P7czB5Ly+guM5vLPJe7b1uK6+1mdjiz/T/JPLct3fdPmtm2kHFVjO36TFzfMLMfZJ5rssxuMrNDZvZIwfNmZn+Xxv2Qmb0881z9Msu7UOY4bpJ+XdKvSfqKpIWCdVZL+qakcyWtlfSgpPPT5z4j6Yr08cclLQWK60OSrkkfXyPpb0rW3yDpiKR16d83S3pTQ2VWKTZJPylY3kiZVY1N0q9KOi99/GJJByS9KHS5DfreZNZ5p6SPp4+vkPTp9PH56fqnSzonfZ/VAcupSmwXZ75PS73YBu3bluJ6u6S/z3ntBknfSu/Xp4/Xtxlb3/p/Kummpsssfe9XS3q5pEcKnt8q6YuSTNIrJd03Spl1pgbu7o+7+xMlq10oaa+7f8vd/0/SpyRdZmYm6bWSbk/Xu0XS5YFCuyx9v6rv+yZJX3T3o4G2P0jd2E5ouMwqxebu33D3J9PH/yPpkKRTzjYLIPd7MyDe2yW9Li2jyyR9yt2fdfdvS9qbvl9rsbn7PZnv072Szgq4/aHjGuC3Jd3p7kfc/fuS7pR0yRhje4uk2wJuv5C7f1VJBa7IZZL+2RP3SnqRmW3WkGXWmQRe0ZmS/jvz93fSZTOSfuDuP+9bHsIZ7n4gffxdSWeUrH+FTv2yXJf+u3S9mZ0eKK46sT3PzFbM7N5e046aLbM6sUmSzOxCJbWpb2YWhyq3ou9N7jppmfxQSRlVee0o6r7/lUpqcD15+7bNuH4v3Ue3m9nZNV/bdGxKm5vOkXR3ZnFTZVZFUexDlVmrV+Qxsy9L+pWcp3a4+xfajCVrUFzZP9zdzaxw3GV6JH2ZpC9lFr9fSQJbq2Ts519I+quWY5tz96fN7FxJd5vZw0oS1EgCl9utkra5+/F08UjlNonM7K2SFiS9JrP4lH3r7t/Mf4fg/k3Sbe7+rJm9Q8l/MK9tadtVXSHpdnc/llk2zjILqtUE7u6vH/EtnpZ0dubvs9Jlzyj5V+S0tPbUWz5yXGZ20Mw2u/uBNNEcGvBWb5b0eXf/Wea9e7XQZ83snyS9r2pcoWJz96fT+2+Z2VckXSDpsxqhzELFZma/KOnflRzE782890jl1qfoe5O3znfM7DRJv6Tke1XltaOo9P5m9nolB8bXuPuzveUF+zZEMiqNy92fyfz5SSX9Hr3XXtT32q8EiKlybBlXSHpXdkGDZVZFUexDlVlsTSj/Jek8S0ZPrFWyc3Z70gtwj5L2Z0naJilUjX53+n5V3veUtrY0efXanC+XlNs73VRsZra+1/xgZhslvUrSYw2XWdXY1kr6vJI2wdv7ngtZbrnfmwHxvknS3WkZ7ZZ0hSWjVM6RdJ6k/xwhltqxmdkFkv5R0qXufiizPHffthjX5syfl0p6PH38JUlvSONbL+kNOvm/0sZjS+N7qZIOwa9nljVZZlXslvSH6WiUV0r6YVpZGa7MmuqNrXuT9EYl7T7PSjoo6Uvp8hdLuiOz3lZJ31ByxNyRWX6ukh/WXkn/Iun0QHHNSLpL0pOSvixpQ7p8QdInM+vNKzmKrup7/d2SHlaSgJYlvSBgmZXGJum30u0/mN5f2XSZ1YjtrZJ+JumBzG1LE+WW971R0iRzafr4eWkZ7E3L5NzMa3ekr3tC0u808N0vi+3L6W+iV0a7y/ZtS3H9taRH0+3fI+mlmdf+cVqWeyX9Udtllv59raQP9r2u6TK7Tcloqp8pyWdXSrpK0lXp8ybpH9K4H1ZmxN0wZcap9AAQqdiaUAAAKRI4AESKBA4AkSKBA0CkSOAAECkSOABEigQOAJH6f2kLP5py0zQoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "def generate_dataset(n=100, degree=1, noise=1, factors=None):\n",
    "    # Generates a dataset by adding random noise to a randomly\n",
    "    # generated polynomial function.\n",
    "    \n",
    "    x = np.random.uniform(low=-1, high=1, size=n)\n",
    "    \n",
    "    factors = np.random.uniform(0, 10, degree+1)\n",
    "    \n",
    "    y = np.zeros(x.shape)\n",
    "    \n",
    "    for idx in range(degree+1):\n",
    "        y += factors[idx] * (x ** idx)\n",
    "\n",
    "    # add noise\n",
    "    y += np.random.normal(-noise, noise, n)\n",
    "    \n",
    "    return x, y\n",
    "\n",
    "# load generated data\n",
    "np.random.seed(0)\n",
    "\n",
    "X, y = generate_dataset(n=100, degree=4, noise=1.5)\n",
    "\n",
    "plt.plot(X, y, 'r.', markersize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement polynomial regression using the `sklearn.preprocessing.PolynomialFeatures` transformation. Using the `sklearn.grid_search.GridSearchCV` class, perform a grid search of the polynomial order hyperparameter space with cross-validation and report the performance on an independent test set.\n",
    "\n",
    "Plot a learning curve that show the validation accuracy as a function of the polynomial order. \n",
    "\n",
    "<p><font color='#770a0a'>Which models have a high bias, and which models have high variance? Motivate your answer.</font><p>\n",
    "    \n",
    "Repeat this experiment, this time using the diabetes dataset instead of the generated data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC curve analysis\n",
    "A common method to evaluate binary classifiers is the receiver operating characteristic (ROC) curve. Similar to the week one practicals, implement a $k$-NN classifier on the breast cancer dataset, however, his time use the $k$-NN pipeline from the preliminary. Train the model for different values of $k$ and evaluate their respective performance with an ROC curve, use the `sklearn.metrics.roc_curve` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $F_1$ score and Dice similarity coefficient\n",
    "\n",
    "The Dice similarity coefficient is a very popular evaluation measure for image segmentation applications. Assuming that $A$ is the ground truth segmentation of an object represented as a binary image, and $B$ is the binary output of an image segmentation method, the Dice similarity coefficient is computed as:\n",
    "\n",
    "$\\text{Dice}(A,B) = \\frac{2|A\\cap B|}{|A| + |B|}$\n",
    "\n",
    "where $|\\cdot|$ represents the cardinality of the objects (e.g. $|A|$ is the number of non-zero pixels in the ground truth segmentation).\n",
    "\n",
    "For example, the Dice similarity can be computed in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate some test objecys\n",
    "A = np.zeros((32, 32))\n",
    "A[10:-10, 10:-10] = 1\n",
    "B = np.zeros((32, 32))\n",
    "B[5:-15, 5:-15] = 1\n",
    "\n",
    "dice = 2*np.sum(A*B)/(np.sum(A)+np.sum(B))\n",
    "\n",
    "# display the results\n",
    "plt.plot()\n",
    "plt.imshow(A)\n",
    "plt.imshow(B, alpha=0.7)\n",
    "\n",
    "print(dice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p><font color='#770a0a'>Show that the $F_1$ score, which is the harmonic mean of precision and recall, is equivalent to the Dice similarity coefficient</font><p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}