t81_558_class_05_1_reg_ridge_lasso.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_1_reg_ridge_lasso.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 5: Regularization and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 5 Material\n",
    "\n",
    "* **Part 5.1: Part 5.1: Introduction to Regularization: Ridge and Lasso** [[Video]](https://www.youtube.com/watch?v=jfgRtCYjoBs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_1_reg_ridge_lasso.ipynb)\n",
    "* Part 5.2: Using K-Fold Cross Validation with Keras [[Video]](https://www.youtube.com/watch?v=maiQf8ray_s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_2_kfold.ipynb)\n",
    "* Part 5.3: Using L1 and L2 Regularization with Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=JEWzWv1fBFQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_3_keras_l1_l2.ipynb)\n",
    "* Part 5.4: Drop Out for Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=bRyOi0L6Rs8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_4_dropout.ipynb)\n",
    "* Part 5.5: Benchmarking Keras Deep Learning Regularization Techniques [[Video]](https://www.youtube.com/watch?v=1NLBwPumUAs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_05_5_bootstrap.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5.1: Introduction to Regularization: Ridge and Lasso\n",
    "\n",
    "Regularization is a technique that reduces overfitting, which occurs when neural networks attempt to memorize training data, rather than learn from it.  Humans are capable of overfitting as well.  Before we examine the ways that a machine accidentally overfits, we will first explore how humans can suffer from it.\n",
    "\n",
    "Human programmers often take certification exams to show their competence in a given programming language.  To help prepare for these exams, the test makers often make practice exams available.  Consider a programmer who enters a loop of taking the practice exam, studying more, and then taking the practice exam again.  At some point, the programmer has memorized much of the practice exam, rather than learning the techniques necessary to figure out the individual questions.  The programmer has now overfit to the practice exam.  When this programmer takes the real exam, his actual score will likely be lower than what he earned on the practice exam.\n",
    "\n",
    "A computer can overfit as well.  Although a neural network received a high score on its training data, this result does not mean that the same neural network will score high on data that was not inside the training set.  Regularization is one of the techniques that can prevent overfitting. A number of different regularization techniques exist.  Most work by analyzing and potentially modifying the weights of a neural network as it trains.  \n",
    "\n",
    "### L1 and L2 Regularization\n",
    "\n",
    "L1 and L2 regularization are two common regularization techniques that can reduce the effects of overfitting (Ng, 2004).  Both of these algorithms can either work with an objective function or as a part of the backpropagation algorithm.  In both cases the regularization algorithm is attached to the training algorithm by adding an additional objective.  \n",
    "\n",
    "Both of these algorithms work by adding a weight penalty to the neural network training.  This penalty encourages the neural network to keep the weights to small values.  Both L1 and L2 calculate this penalty differently.  For gradient-descent-based algorithms, such as backpropagation, you can add this penalty calculation to the calculated gradients.  For objective-function-based training, such as simulated annealing, the penalty is negatively combined with the objective score.\n",
    "\n",
    "We are going to look at linear regression to see how L1 and L2 regularization work.  The following code sets up the auto-mpg data for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LassoCV\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "names = ['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']\n",
    "x = df[names].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=45)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple function to evaluate the coefficients of a regression\n",
    "%matplotlib inline    \n",
    "from IPython.display import display, HTML    \n",
    "\n",
    "def report_coef(names,coef,intercept):\n",
    "    r = pd.DataFrame( { 'coef': coef, 'positive': coef>=0  }, index = names )\n",
    "    r = r.sort_values(by=['coef'])\n",
    "    display(r)\n",
    "    print(f\"Intercept: {intercept}\")\n",
    "    r['coef'].plot(kind='barh', color=r['positive'].map(\n",
    "        {True: 'b', False: 'r'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression\n",
    "\n",
    "To understand L1/L2 regularization, it is good to start with linear regression.  L1/L2 were first introduced for [linear regression](https://en.wikipedia.org/wiki/Linear_regression).  They can also be used for neural networks.  To fully understand L1/L2 we will begin with how they are used with linear regression.\n",
    "\n",
    "The following code uses linear regression to fit the auto-mpg data set.  The RMSE reported will not be as good as a neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.0019345985860784\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.427721</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007255</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.005491</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.020166</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.138575</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.783047</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>1.003762</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.427721     False\n",
       "weight       -0.007255     False\n",
       "horsepower   -0.005491     False\n",
       "displacement  0.020166      True\n",
       "acceleration  0.138575      True\n",
       "year          0.783047      True\n",
       "origin        1.003762      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -19.101231042200112\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAD4CAYAAABIQCkOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZIElEQVR4nO3deZhkdX3v8fdHQBEHWWRAQsBRBAliBKZEUFkUQogaxYACLmELc9HE9UIefVyuG66Jxrghcg3ojWhQQK4xsskwiAzSA7MACihL1HC1CYSIhkX43j/qtCna7unume6uU93v1/P006fO+Z3f+daZnv7071enTqWqkCSpTR7V7wIkSRrNcJIktY7hJElqHcNJktQ6hpMkqXU27HcBc8VWW21VixYt6ncZkjQwVqxYcWdVLRxrm+E0TRYtWsTQ0FC/y5CkgZHk9vG2Oa0nSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJreOl5JpXkn5XIM0tM/XBFo6cJEmtYzhJklrHcJIktY7hJElqnTkfTkm+lWTzCdq8N8lBs1WTJGnt5uzVekkCpKpeOFHbqnrXLJQkSZqkgR45JXlLkuuarzclWZTkB0k+A1wDbJ/ktiRbNe3fmeSHSS5KclaSk5r1ZyQ5vFm+Lcl7klyTZE2SXfr3DCVpfhrYcEqyGDgWeDawN3ACsAXwNOCLVbVHVd3e074DHAbsAfwZ0FlL93dW1Z7AZ4GT1lLDkiRDSYaGh4fX9ylJkhoDG07A84Bzq+pXVXUvcA6wL3B7VS0fp/03quq/quqXwP9dS9/nNN9XAIvGa1RVp1VVp6o6CxeO+XlZkqR1MMjhNN57/X81xfZjub/5/hBz+HU5SWqrQQ6nZcChSTZJ8jjgZcDla2n/XeBPk2ycZAHwotkoUpI0dQM7Kqiqa5KcAXy/WXU6cPda2l+d5HxgFXA7MATcM9N1SpKmLjVTd+1roSQLqureJJvQHXktqaprpqPvTqdTQ0ND09GVZpA3fpWm1/pESJIVVTXmxWkDO3JaR6cl2RXYGDhzuoJJkjS95lU4VdUr+12DJGli8yqcpHk0iy0NtEG+Wk+SNEcZTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodPzJD84qfhKs28qNcfpcjJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juE0SUk26HcNkjRfzMlwSvK+JG/seXxKkjckOTnJ1UlWJ3lPz/bzkqxIcn2SJT3r703y3iRXAfvM8tOQpHlrToYT8L+BowGSPAo4Evg5sBOwF7A7sDjJfk3746pqMdAB3pDkCc36xwHXVdWzq+q7ow+SZEmSoSRDw8PDM/uMJGkemZPhVFW3Af+eZA/gYOBa4Fk9y9cAu9ANK+gG0ipgObB9z/qHgK+v5TinVVWnqjoLFy6ciaciSfPSXL5DxOnAMcATgS8ABwIfrKrP9TZKcgBwELBPVf06yVJg42bzfVX10GwVLEnqmpMjp8a5wCF0R0wXNF/HJVkAkGS7JFsDmwF3N8G0C7B3vwqWJHXN2ZFTVT2Q5FLgP5rRz4VJ/gC4Mt0brN0LvBr4NnBiktXAjXSn9iRJfTRnw6m5EGJv4OUj66rqE8Anxmj+J2P1UVULZqY6SdLazMlpvSS7Aj8CLqmqm/tdjyRpaubkyKmqbgCe0u86JEnrZk6GkzQePzdHGgxzclpPkjTYDCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1jh+ZoXklmf1j+jEd0tQ5cpIktY7hJElqHcNJktQ6hpMkqXVaGU5JlibpTFNfhybZtefxe5McNB19S5JmRivDaaqSbLCWzYcCvw2nqnpXVV0881VJktbVeoVTkvOSrEhyfZIlzbpDklyTZFWSS5p1C5L8Q5I1SVYnOaxZf3CSK5v2ZydZMMYxxmyT5LYk70ryXeDlSU5IcnVz3K8n2STJc4CXAB9NsjLJjknOSHJ408eBSa5t6vpCksf09P2e5phrkuyyPudJkjQ16ztyOq6qFgMd4A1JtgE+DxxWVc8EXt60eydwT1U9o6r+EPhOkq2AdwAHVdWewBDwlt7OJ9Hmvqp6XlV9BTinqp7VHPcHwPFV9T3gfODkqtq9qn7c0/fGwBnAEVX1DLrv+XptT993Nsf8LHDSWE8+yZIkQ0mGhoeHp3bmJEnjWt9wekOSVcByYHtgCbCsqm4FqKq7mnYHAZ8e2amq7gb2pjvddkWSlcDRwJNG9T9Rm6/2LO+W5PIka4BXAU+foPanAbdW1U3N4zOB/Xq2n9N8XwEsGquDqjqtqjpV1Vm4cOEEh5MkTdY63yEiyQF0Q2efqvp1kqXAKrq/9H+nOTD6ffIBLqqqo9Z2mAna/Kpn+Qzg0KpaleQY4ICJnsIE2+9vvj+Ed9KQpFm1PiOnzYC7m2Dahe4o5zHA/kmeDJBky6bthcBfjeyYZAu6o63nJnlqs26TJDuPOsZk2ozYFLgjyUZ0R04jftlsG+2HwKKRvoHXAJdN4nlLkmbY+oTTt4ENk6wG3kc3SIbpTu2d00z3jUy7vR/YIsl1zfrnV9UwcAxwVtPHcuARFx5Mpk2PdwJXARfRDZ4RXwFObi582LGn7/uAY4Gzm6nAh4FT1+VESJKmV8q7Uk6LTqdTQ0ND/S5DE/DGr1J7JFlRVWO+p3VOvM9JkjS3GE6SpNbxKjTNK06xSYPBkZMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWOH5mheWX0J+H6ERpSOzlykiS1juEkSWodw0mS1DqGkySpdaZ8QUSSdwP3Ao8HllXVxVPc/wDgpKp68VSPPduSHArcVFU39LsWSZpP1nnkVFXvmmowDaBDgV37XYQkzTeTCqckb09yY5KLgac1685Icniz/KEkNyRZneRverafmuTyJDcl+Z2RUpK9knwvybXN95G+N0jyN0nWNH2+vlm/OMllSVYkuSDJts36pUk+nmRZkh8keVaSc5LcnOT9Pcd7dZLvJ1mZ5HNJNmjW35vklCSrkixPsk2S5wAvAT7atN9xPc6zJGkKJpzWS7IYOBLYo2l/DbCiZ/uWwMuAXaqqkmzes/siYH9gR+DSJE8d1f0Pgf2q6jdJDgI+ABwGLAGeDOzRbNsyyUbAJ4GXVtVwkiOAU4Djmr4eqKr9krwR+AawGLgL+HGSjwNbA0cAz62qB5N8BngV8EXgccDyqnp7ko8AJ1TV+5OcD3yzqr42zrlZ0tTKDjvsMNGplCRN0mRec9oXOLeqfg3Q/MLu9Z/AfcDpSf4Z+GbPtn+qqoeBm5PcAuwyat/NgDOT7AQUsFGz/iDg1Kr6DUBV3ZVkN2A34KJ030m5AXBHT18jda0Brq+qO5p6bwG2B55HN7CubvZ/LPCLZp8HeupeAfzRJM4LVXUacBpAp9Px7ZySNE0me0HEuL94m5HNXsCBdEdYfwW8YJz9Rj9+H3BpVb0sySJgabM+Y7QN3dDZZ5xS7m++P9yzPPJ4w2b/M6vqbWPs+2DVb+8V8BDeOUOS+moyrzktA16W5LFJNgX+tHdjkgXAZlX1LeBNwO49m1+e5FHN6zVPAW4c1fdmwM+a5WN61l8InJhkw+YYWzb7LkyyT7NuoyRPn0T9Iy4BDk+y9UifSZ40wT6/BDadwjEkSdNgwnCqqmuArwIrga8Dl49qsinwzSSrgcuAN/dsu7FZ9y/AiVV136h9PwJ8MMkVdKfpRpwO/CuwOskq4JVV9QBwOPDhZt1K4DmTepbd53ED8A7gwqbWi4BtJ9jtK8DJzQUbXhAhSbMkNUN3vkxyBmu5mGCu6XQ6NTQ01O8yNAFv/Cq1R5IVVdUZa5t3iJAktc6MvfBfVcfMVN+SpLnNq9I0rziNJw0Gp/UkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCfNH6M/BldSaxlOkqTWMZwkSa1jOEmSWsdwkiS1zrSEU5JFSa6bjr4kSer7yCnJhv2uYTIGpU5JmgumM5w2SPL5JNcnuTDJY5PsnmR5ktVJzk2yBUCSpUk+kOQy4I1JXp7kuiSrkixr2myQ5KNJrm72/x/N+gOSLGv6uyHJqUke1Ww7Ksmapq8PN+tekeRjzfIbk9zSLO+Y5LvN8uIklyVZkeSCJNuOVec0nitJ0lpM52hgJ+CoqjohyT8BhwF/Dby+qi5L8l7gfwFvatpvXlX7AyRZA/xxVf0syebN9uOBe6rqWUkeA1yR5MJm217ArsDtwLeBP0vyPeDDwGLgbuDCJIcCy4CTm/32Bf49yXbA84DLk2wEfBJ4aVUNJzkCOAU4bnSdoyVZAiwB2GGHHdb1vEmSRpnOcLq1qlY2yyuAHen+Yr+sWXcmcHZP+6/2LF8BnNGE2jnNuoOBP0xyePN4M7oB+ADw/aoaGQGdRTdoHgSWVtVws/4fgf2q6rwkC5JsCmwPfBnYj25QnQM8DdgNuCjdN2luANwxTp2PUFWnAacBdDqdWvvpkSRN1nSG0/09yw8Bm4/XsPGrkYWqOjHJs4EXASuT7A6E7qjrgt6dkhwAjA6CatqP50rgWOBG4HK6o6J9gP8J7ABcX1X7TFSnJGl2zOQFEfcAdyfZt3n8GuCysRom2bGqrqqqdwF30h3hXAC8tpl2I8nOSR7X7LJXkic3rzUdAXwXuArYP8lWSTYAjuo53jLgpOb7tcDzgfur6h66gbUwyT7NcTZK8vTpOw2SpKma6SvQjgZOTbIJcAvd0ctYPppkJ7qjn0uAVcBqYBFwTbrzbcPAoU37K4EPAc+gGzjnVtXDSd4GXNr0862q+kbT/nK6gbesqh5K8hPghwBV9UAzdfj3STaje07+Drh+ms6BJGmKUjVYL5U003onVdWL+11Lr06nU0NDQ/0uQ2uTwID9vEtzWZIVVdUZa1vf3+ckSdJoA/fG0qpaCiztcxmSpBnkyEnzh1N60sAwnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ80PSfdL0kAwnCRJrWM4SZJax3CSJLWO4SRJap05HU5JTk+y6wRtzkhy+BjrFyV55cxVJ0kaz5wOp6r6i6q6YR13XwQYTpLUBwMRTkn+OskbmuWPJ/lOs3xgkv+T5OAkVya5JsnZSRY025cm6TTLxye5qVn3+SSf6jnEfkm+l+SWnlHUh4B9k6xM8uZZfLqSNO8NRDgBy4B9m+UOsCDJRsDzgDXAO4CDqmpPYAh4S+/OSX4PeCewN/BHwC6j+t+26evFdEMJ4K3A5VW1e1V9fKyikixJMpRkaHh4eD2foiRpxKCE0wpgcZJNgfuBK+mG1L7AfwG7AlckWQkcDTxp1P57AZdV1V1V9SBw9qjt51XVw80U4DaTLaqqTquqTlV1Fi5cuE5PTJL0uzbsdwGTUVUPJrkNOBb4HrAaeD6wI3ArcFFVHbWWLia6NcD9U2grSZphgzJygu7U3knN98uBE4GVwHLguUmeCpBkkyQ7j9r3+8D+SbZIsiFw2CSO90tg0+kqXpI0eYMUTpfTfW3oyqr6OXAf3deEhoFjgLOSrKYbVo94TamqfgZ8ALgKuBi4AbhnguOtBn6TZJUXREjS7EpV9buGWZFkQVXd24yczgW+UFXnTlf/nU6nhoaGpqs7TbeRm77Ok593aRAkWVFVnbG2DdLIaX29u7lg4jq6r1Od1+d6JEnjGIgLIqZDVZ3U7xokSZMzb8JJ85zTedJAmU/TepKkAWE4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6/h5Tm0w8hHimnl+rpM0EBw5SZJax3CSJLWO4SRJap2BCKckZyQ5vFk+PcmuU9z/3pmpTJI0Ewbugoiq+ouZ7D9JgFTVwzN5HEnS+Po6ckry50lWJ1mV5NwktybZqNn2+CS3jTzu2Wdpkk6zfG+SU5r9lyfZpln/5CRXJrk6yftG7X9ys351kvc06xYl+UGSzwDXANs3o7XrkqxJ8ubZOB+SpK6+hVOSpwNvB15QVc8EjgeWAi9qmhwJfL2qHlxLN48Dljf7LwNOaNZ/AvhsVT0L+H89xzwY2AnYC9gdWJxkv2bz04AvVtUewFbAdlW1W1U9A/iHcZ7DkiRDSYaGh4endgIkSePq58jpBcDXqupOgKq6CzgdOLbZfizjhEKPB4BvNssrgEXN8nOBs5rlL/W0P7j5upbuCGkXumEFcHtVLW+WbwGekuSTSQ4B/nOsg1fVaVXVqarOwoULJyhVkjRZ/XzNKcAj3hFZVVc0U2z7AxtU1XUT9PFg1W/fVfkQj3w+Y73bMsAHq+pzj1iZLAJ+1VPH3UmeCfwx8JfAK4DjJnxGkqRp0c+R0yXAK5I8ASDJls36L9Id9Uw0alqbK+hOCwK8qmf9BcBxSRY0x9wuydajd06yFfCoqvo68E5gz/WoRZI0RX0Lp6q6HjgFuCzJKuBjzaZ/BLbgv6fl1sUbgb9McjWwWc8xLwS+DFyZZA3wNWDTMfbfDliaZCVwBvC29ahFkjRFqZbda6x5P9NLq+o1/a5lKjqdTg0NDa3bzt5bb/a07Oddms+SrKiqzljbWvU+pySfBP4EeGG/a5Ek9U+rwqmqXt/vGiRJ/deqcJq3nGqSpEcYiHvrSZLmF8NJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWqd1ty8aVEmGgdunudutgDunuc+ZMCh1wuDUOih1wuDUOih1wuDUur51Pqmqxvy8IcOpxZIMjXffqTYZlDphcGodlDphcGodlDphcGqdyTqd1pMktY7hJElqHcOp3U7rdwGTNCh1wuDUOih1wuDUOih1wuDUOmN1+pqTJKl1HDlJklrHcJIktY7h1CJJtkxyUZKbm+9brKXt45P8LMmnZrPG5tgT1plk9yRXJrk+yeokR8xifYckuTHJj5K8dYztj0ny1Wb7VUkWzVZtY9QyUa1vSXJDcw4vSfKkftTZ1LLWWnvaHZ6kkvTlUujJ1JnkFc15vT7Jl2e7xp46Jvr33yHJpUmubX4G+vIp4Um+kOQXSa4bZ3uS/H3zPFYn2XO9D1pVfrXkC/gI8NZm+a3Ah9fS9hPAl4FPtbFOYGdgp2b594A7gM1nobYNgB8DTwEeDawCdh3V5nXAqc3ykcBX+/TvPZlanw9s0iy/ts21Nu02BZYBy4FOG+sEdgKuBbZoHm/d1nNK94KD1zbLuwK39anW/YA9gevG2f5C4F+AAHsDV63vMR05tctLgTOb5TOBQ8dqlGQxsA1w4SzVNdqEdVbVTVV1c7P8b8AvgDHfCT7N9gJ+VFW3VNUDwFeaenv11v814MAkmYXaRpuw1qq6tKp+3TxcDvz+LNc4YjLnFeB9dP94uW82i+sxmTpPAD5dVXcDVNUvZrnGEZOptYDHN8ubAf82i/X9dxFVy4C71tLkpcAXq2s5sHmSbdfnmIZTu2xTVXcANN+3Ht0gyaOAvwVOnuXaek1YZ68ke9H9y/DHs1DbdsBPeh7/tFk3Zpuq+g1wD/CEWahttMnU2ut4un+d9sOEtSbZA9i+qr45m4WNMplzujOwc5IrkixPcsisVfdIk6n13cCrk/wU+Bbw+tkpbcqm+rM8oQ3XqxxNWZKLgSeOsentk+zidcC3quonM/nH/jTUOdLPtsCXgKOr6uHpqG2iQ46xbvT7JSbTZjZMuo4krwY6wP4zWtH41lpr80fTx4FjZqugcUzmnG5Id2rvALoj0cuT7FZV/zHDtY02mVqPAs6oqr9Nsg/wpabW2fi/NBXT/n/KcJplVXXQeNuS/DzJtlV1R/NLfazphn2AfZO8DlgAPDrJvVU17gvUfaqTJI8H/hl4RzPUnw0/Bbbvefz7/O5UyEibnybZkO50ydqmLGbKZGolyUF0/yjYv6run6XaRpuo1k2B3YClzR9NTwTOT/KSqhqatSon/++/vKoeBG5NciPdsLp6dkp8RB0T1Xo8cAhAVV2ZZGO6N1vt11TkeCb1szwVTuu1y/nA0c3y0cA3RjeoqldV1Q5VtQg4ie4877QG0yRMWGeSRwPn0q3v7Fms7WpgpyRPbmo4km69vXrrPxz4TjWv6s6yCWttpso+B7ykj6+NwAS1VtU9VbVVVS1qfjaX0615NoNpwjob59G90IQkW9Gd5rtlVqvsmkyt/wocCJDkD4CNgeFZrXJyzgf+vLlqb2/gnpGp/3XWjys//Br3ipgnAJcANzfft2zWd4DTx2h/DP25Wm/COoFXAw8CK3u+dp+l+l4I3ET3Na63N+veS/eXJXT/g58N/Aj4PvCUPv6bT1TrxcDPe87h+W2tdVTbpfThar1JntMAHwNuANYAR7b1nNK9Qu8KulfyrQQO7lOdZ9G94vZBuqOk44ETgRN7zumnm+exZjr+7b19kSSpdZzWkyS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrfP/AQ55MmH0t9qpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "\n",
    "# Create linear regression\n",
    "regressor = sklearn.linear_model.LinearRegression()\n",
    "\n",
    "# Fit/train linear regression\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# L1 (Lasso) Regularization\n",
    "\n",
    "L1 Regularization, also called LASSO (Least Absolute Shrinkage and Selection Operator) is should be used to create sparsity in the neural network. In other words, the L1 algorithm will push many weight connections to near 0.  When a weight is near 0, the program drops it from the network.  Dropping weighted connections will create a sparse neural network.\n",
    "\n",
    "Feature selection is a useful byproduct of sparse neural networks. Features are the values that the training set provides to the input neurons.  Once all the weights of an input neuron reach 0, the neural network training determines that the feature is unnecessary.  If your data set has a large number of input features that may not be needed, L1 regularization can help the neural network detect and ignore unnecessary features.\n",
    "\n",
    "L1 is implemented by adding the following error to the objective to minimize:\n",
    "\n",
    "$ E_1 = \\alpha \\sum_w{ |w| } $\n",
    "\n",
    "You should use L1 regularization to create sparsity in the neural network. In other words, the L1 algorithm will push many weight connections to near 0.  When a weight is near 0, the program drops it from the network.  Dropping weighted connections will create a sparse neural network.\n",
    "Feature selection is a useful byproduct of sparse neural networks. Features are the values that the training set provides to the input neurons.  Once all the weights of an input neuron reach 0, the neural network training determines that the feature is unnecessary.  If your data set has a large number of input features that may not be needed, L1 regularization can help the neural network detect and ignore unnecessary features.\n",
    "\n",
    "The following code demonstrates lasso regression.  Notice the effect of the coefficients compared to the previous section that used linear regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.0604021904033303\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.012995</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007328</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.002715</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.011601</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.114391</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>0.708222</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.777480</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.012995     False\n",
       "weight       -0.007328     False\n",
       "horsepower   -0.002715     False\n",
       "displacement  0.011601      True\n",
       "acceleration  0.114391      True\n",
       "origin        0.708222      True\n",
       "year          0.777480      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -18.506677982383252\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAD4CAYAAAC5S3KDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYcElEQVR4nO3deZhldX3n8feHZVRoBJTWcRBsRYS0qCAFgsqiEkJcMaCCy8giPThGXAKZ+LiMG655JI5RsWVMYyaigqDGUdmkaUAaqYZeAAWVZaLjmEYICoZF+M4f9zRcylq7a/lV9fv1PPXUub/zO7/zvae661O/c889N1WFJEmt2WSmC5AkaTgGlCSpSQaUJKlJBpQkqUkGlCSpSZvNdAFzxXbbbVcLFiyY6TIkadZYsWLFrVU1f6T1BtQkWbBgAYODgzNdhiTNGkluGW29p/gkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElN8jLzRiUzXYEkjW0qPxDDGZQkqUkGlCSpSQaUJKlJBpQkqUkGlCSpSQbUOCXZdKZrkKSNyZwMqCQfSvK2vscnJzkhyUlJrkyyOskH+tZ/M8mKJNcmWdTXfmeSDya5Ath3mp+GJG3U5mRAAf8TeCNAkk2AI4BfAzsDewO7A3sm2b/rf0xV7QkMACckeWzXviVwTVU9p6ouHbqTJIuSDCYZXLt27dQ+I0nayMzJgKqqm4HfJNkDOBi4Gtirb/kqYFd6gQW9UFoFLAd26Gu/H/jGKPtZXFUDVTUwf/6In7klSVoPc/lOEqcBRwH/EfgS8CLgo1X1hf5OSQ4EDgL2rarfJ1kKPLJbfXdV3T9dBUuSHjInZ1Cdc4BD6M2czu2+jkkyDyDJ9kkeB2wN3N6F067APjNVsCTpIXN2BlVV9ya5CPi3bhZ0XpI/AS5P70Z3dwKvB74PHJ9kNXA9vdN8kqQZNmcDqrs4Yh/gVevaqurTwKeH6f7nw41RVfOmpjpJ0ljm5Cm+JAuBnwEXVtVPZ7oeSdLEzckZVFVdBzxlpuuQJK2/ORlQc8FUfsaKJM0Gc/IUnyRp9jOgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElN8uM2GtX7VHpJmri58nE9zqAkSU0yoCRJTTKgJElNMqAkSU2a8wGV5LtJthmjzweTHDRdNUmSxjZnr+JLEiBV9eKx+lbV+6ahJEnSBMzqGVSSdya5pvt6e5IFSX6c5HPAVcAOSW5Osl3X/71JfpLk/CRnJDmxa1+S5PBu+eYkH0hyVZI1SXaduWcoSRuvWRtQSfYEjgaeA+wDHAdsC+wCfLmq9qiqW/r6DwCHAXsAfwEMjDL8rVX1bODzwImj1LAoyWCSwbVr127oU5Ik9Zm1AQU8Hzinqu6qqjuBs4H9gFuqavkI/b9VVf9eVb8D/nmUsc/uvq8AFozUqaoWV9VAVQ3Mnz9/vZ6EJGl4szmgRrrXwl0T7D+ce7rv9zOHX6eTpJbN5oBaBhyaZIskWwKvBC4Zpf+lwMuSPDLJPOAl01GkJGn9zNrZQVVdlWQJ8KOu6TTg9lH6X5nk28Aq4BZgELhjquuUJK2f1Fy5q+A4JJlXVXcm2YLeDGxRVV01GWMPDAzU4ODgZAwFeLNYSetvtvxaT7Kiqka8YG3WzqDW0+IkC4FHAqdPVjhJkibfRhVQVfXama5BkjQ+G1VAzSazZYouSVNlNl/FJ0mawwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKT/LiNRs22T9T140EkTTZnUJKkJhlQkqQmGVCSpCYZUJKkJjUZUEmWJhmYpLEOTbKw7/EHkxw0GWNLkqZOkwE1UUk2HWX1ocCDAVVV76uqC6a+KknShtiggEryzSQrklybZFHXdkiSq5KsSnJh1zYvyT8kWZNkdZLDuvaDk1ze9T8zybxh9jFsnyQ3J3lfkkuBVyU5LsmV3X6/kWSLJM8FXg58MsnKJDslWZLk8G6MFyW5uqvrS0ke0Tf2B7p9rkmy64YcJ0nSxG3oDOqYqtoTGABOSPJ44IvAYVX1LOBVXb/3AndU1TOq6pnAD5JsB7wHOKiqng0MAu/sH3wcfe6uqudX1VeBs6tqr26/PwaOraofAt8GTqqq3avq531jPxJYArymqp5B7z1hb+4b+9Zun58HThzuySdZlGQwyeDatWsnduQkSaPa0IA6IckqYDmwA7AIWFZVNwFU1W1dv4OAz67bqKpuB/ahd+rtsiQrgTcCTxoy/lh9vta3vFuSS5KsAV4HPH2M2ncBbqqqG7rHpwP7960/u/u+Algw3ABVtbiqBqpqYP78+WPsTpI0Eet9J4kkB9ILnn2r6vdJlgKr6P3i/6PuwNB7DQQ4v6qOHG03Y/S5q295CXBoVa1KchRw4FhPYYz193Tf78c7bkjStNuQGdTWwO1dOO1Kb7bzCOCAJE8GSPKYru95wF+u2zDJtvRmXc9L8tSubYskTxuyj/H0WWcr4FdJNqc3g1rnd926oX4CLFg3NvAG4OJxPG9J0jTYkID6PrBZktXAh+iFyVp6p/nO7k79rTsF92Fg2yTXdO0vqKq1wFHAGd0Yy4GHXYwwnj593gtcAZxPL3zW+SpwUncxxE59Y98NHA2c2Z0WfAA4dX0OhCRp8qW8y+ekGBgYqMHBwUkbz5vFSprrkqyoqhHf8zon3gclSZp7DChJUpO8Oq1RnjKTtLFzBiVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkh+30aihn6jrx29I2tg4g5IkNcmAkiQ1yYCSJDXJgJIkNWnCF0kkeT9wJ/BoYFlVXTDB7Q8ETqyql05039MtyaHADVV13UzXIkkbm/WeQVXV+yYaTrPQocDCmS5CkjZG4wqoJO9Ocn2SC4BdurYlSQ7vlj+W5Lokq5P8bd/6U5NckuSGJH80Y0qyd5IfJrm6+75u7E2T/G2SNd2Yb+3a90xycZIVSc5N8oSufWmSU5IsS/LjJHslOTvJT5N8uG9/r0/yoyQrk3whyaZd+51JTk6yKsnyJI9P8lzg5cAnu/47bcBxliRN0Jin+JLsCRwB7NH1vwpY0bf+McArgV2rqpJs07f5AuAAYCfgoiRPHTL8T4D9q+oPSQ4CPgIcBiwCngzs0a17TJLNgc8Ar6iqtUleA5wMHNONdW9V7Z/kbcC3gD2B24CfJzkFeBzwGuB5VXVfks8BrwO+DGwJLK+qdyf5BHBcVX04ybeB71TVWSMcm0Vdrey4445jHUpJ0gSM5zWo/YBzqur3AN0v7X6/Be4GTkvyv4Hv9K37elU9APw0yY3ArkO23Ro4PcnOQAGbd+0HAadW1R8Aquq2JLsBuwHnp/cu1k2BX/WNta6uNcC1VfWrrt4bgR2A59MLrSu77R8F/Gu3zb19da8A/nQcx4WqWgwsBhgYGPCttJI0icZ7kcSIv3y7Gc7ewIvozbT+EnjhCNsNffwh4KKqemWSBcDSrj3D9A294Nl3hFLu6b4/0Le87vFm3fanV9W7htn2vqoH79VwP95hQ5Jm3Hheg1oGvDLJo5JsBbysf2WSecDWVfVd4O3A7n2rX5Vkk+71m6cA1w8Ze2vgl93yUX3t5wHHJ9ms28djum3nJ9m3a9s8ydPHUf86FwKHJ3ncujGTPGmMbX4HbDWBfUiSJsmYAVVVVwFfA1YC3wAuGdJlK+A7SVYDFwPv6Ft3fdf2PeD4qrp7yLafAD6a5DJ6p+zWOQ34P8DqJKuA11bVvcDhwMe7tpXAc8f1LHvP4zrgPcB5Xa3nA08YY7OvAid1F3F4kYQkTaPUFN2FNMkSRrnAYK4ZGBiowcHBSRvPm8VKmuuSrKiqgZHWeycJSVKTpuxigKo6aqrGliTNfV6t1ihP6Una2HmKT5LUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA6pFQz9OV5I2QgaUJKlJBpQkqUkGlCSpSQaUJKlJkxJQSRYkuWYyxpIkCRqYQSXZbKZrGI/ZUqckzRWTGVCbJvlikmuTnJfkUUl2T7I8yeok5yTZFiDJ0iQfSXIx8LYkr0pyTZJVSZZ1fTZN8skkV3bb/5eu/cAky7rxrktyapJNunVHJlnTjfXxru3VST7VLb8tyY3d8k5JLu2W90xycZIVSc5N8oTh6pzEYyVJGsNkzgp2Bo6squOSfB04DPhr4K1VdXGSDwL/HXh713+bqjoAIMka4M+q6pdJtunWHwvcUVV7JXkEcFmS87p1ewMLgVuA7wN/keSHwMeBPYHbgfOSHAosA07qttsP+E2S7YHnA5ck2Rz4DPCKqlqb5DXAycAxQ+scKskiYBHAjjvuuL7HTZI0jMkMqJuqamW3vALYid4v94u7ttOBM/v6f61v+TJgSRdsZ3dtBwPPTHJ493hreiF4L/Cjqlo3EzqDXtjcByytqrVd+z8B+1fVN5PMS7IVsAPwFWB/emF1NrALsBtwfnpvkN0U+NUIdT5MVS0GFgMMDAzU6IdHkjQRkxlQ9/Qt3w9sM1LHzl3rFqrq+CTPAV4CrEyyOxB6s69z+zdKciAwNAyq6z+Sy4GjgeuBS+jNjvYF/grYEbi2qvYdq05J0vSZyosk7gBuT7Jf9/gNwMXDdUyyU1VdUVXvA26lN9M5F3hzdwqOJE9LsmW3yd5Jnty99vQa4FLgCuCAJNsl2RQ4sm9/y4ATu+9XAy8A7qmqO+iF1vwk+3b72TzJ0yfvMEiS1sdUX5n2RuDUJFsAN9KbxQznk0l2pjcLuhBYBawGFgBXpXfubS1waNf/cuBjwDPohc45VfVAkncBF3XjfLeqvtX1v4Re6C2rqvuT/AvwE4Cqurc7jfg/kmxN75j8HXDtJB0DSdJ6SNXseumkO8V3YlW9dKZr6TcwMFCDg4OTM1gCs+znIkkTlWRFVQ2MtH7G3wclSdJwZt2bT6tqKbB0hsuQJE0xZ1At8vSeJBlQkqQ2GVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCYZUJKkJhlQkqQmGVCSpCbNus+DmtOSh5b9yA1JGzlnUJKkJhlQkqQmGVCSpCYZUJKkJs3pgEpyWpKFY/RZkuTwYdoXJHnt1FUnSRrNnA6oqnpTVV23npsvAAwoSZohsyKgkvx1khO65VOS/KBbflGS/5Xk4CSXJ7kqyZlJ5nXrlyYZ6JaPTXJD1/bFJH/ft4v9k/wwyY19s6mPAfslWZnkHdP4dCVJzJKAApYB+3XLA8C8JJsDzwfWAO8BDqqqZwODwDv7N07yn4D3AvsAfwrsOmT8J3RjvZReMAH8DXBJVe1eVacMV1SSRUkGkwyuXbt2A5+iJKnfbAmoFcCeSbYC7gEupxdU+wH/DiwELkuyEngj8KQh2+8NXFxVt1XVfcCZQ9Z/s6oe6E4HPn68RVXV4qoaqKqB+fPnr9cTkyQNb1bcSaKq7ktyM3A08ENgNfACYCfgJuD8qjpylCEyyjrohd54+0qSpsFsmUFB7zTfid33S4DjgZXAcuB5SZ4KkGSLJE8bsu2PgAOSbJtkM+Cwcezvd8BWk1W8JGliZlNAXULvtaLLq+rXwN30XiNaCxwFnJFkNb3AethrTFX1S+AjwBXABcB1wB1j7G818Ickq7xIQpKmX2ojuSlpknlVdWc3gzoH+FJVnTNZ4w8MDNTg4OCGDeLNYiVtRJKsqKqBkdbPphnUhnp/dxHFNfRet/rmDNcjSRrFrLhIYjJU1YkzXYMkafw2moCaFTytJ0kP2phO8UmSZhEDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkPw+qBf0f9b6Onw0laSPnDEqS1CQDSpLUJANKktSkWRFQSZYkObxbPi3Jwgluf+fUVCZJmiqz7iKJqnrTVI6fJECq6oGp3I8kaXQzOoNK8p+TrE6yKsk5SW5Ksnm37tFJbl73uG+bpUkGuuU7k5zcbb88yeO79icnuTzJlUk+NGT7k7r21Uk+0LUtSPLjJJ8DrgJ26GZt1yRZk+Qd03E8JEkPmbGASvJ04N3AC6vqWcCxwFLgJV2XI4BvVNV9owyzJbC8234ZcFzX/mng81W1F/D/+vZ5MLAzsDewO7Bnkv271bsAX66qPYDtgO2rareqegbwDyM8h0VJBpMMrl27dmIHQJI0qpmcQb0QOKuqbgWoqtuA04Cju/VHM0Iw9LkX+E63vAJY0C0/DzijW/7Hvv4Hd19X05sp7UovsABuqarl3fKNwFOSfCbJIcBvh9t5VS2uqoGqGpg/f/4YpUqSJmImX4MK8LB3o1bVZd3ptgOATavqmjHGuK/qwXe03s/Dn89w73QN8NGq+sLDGpMFwF19ddye5FnAnwFvAV4NHDPmM5IkTZqZnEFdCLw6yWMBkjyma/8yvdnPWLOn0VxG7xQhwOv62s8Fjkkyr9vn9kkeN3TjJNsBm1TVN4D3As/egFokSethxgKqqq4FTgYuTrIK+FS36p+AbXnoFN36eBvwliRXAlv37fM84CvA5UnWAGcBWw2z/fbA0iQrgSXAuzagFknSekg1ds+37v1Or6iqN8x0LRMxMDBQg4OD67ex9+KTtBFKsqKqBkZa39T7oJJ8Bvhz4MUzXYskaWY1FVBV9daZrkGS1IamAmqj5ek8Sfojs+JefJKkjY8BJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJalJztzqarZKsBW6ZxCG3A26dxPEmS6t1Qbu1tVoXtFtbq3VBu7W1WheMXNuTqmrEzyoyoBqVZHC0e1TNlFbrgnZra7UuaLe2VuuCdmtrtS5Y/9o8xSdJapIBJUlqkgHVrsUzXcAIWq0L2q2t1bqg3dparQvara3VumA9a/M1KElSk5xBSZKaZEBJkppkQM2wJIckuT7Jz5L8zTDrH5Hka936K5IsaKSu/ZNcleQPSQ6fjprGWdc7k1yXZHWSC5M8qaHajk+yJsnKJJcmWdhKbX39Dk9SSablcuVxHLOjkqztjtnKJG+ajrrGU1vX59Xdv7drk3ylhbqSnNJ3vG5I8m/TUdc4a9sxyUVJru7+j47+6elV5dcMfQGbAj8HngL8B2AVsHBIn/8KnNotHwF8rZG6FgDPBL4MHN7Q8XoBsEW3/ObpOF4TqO3RfcsvB77fSm1dv62AZcByYKCFuoCjgL+fjuO0HrXtDFwNbNs9flwLdQ3p/1bgSw0ds8XAm7vlhcDNo43pDGpm7Q38rKpurKp7ga8CrxjS5xXA6d3yWcCLkmSm66qqm6tqNfDAFNcy0bouqqrfdw+XA09sqLbf9j3cEpiuK5TG8+8M4EPAJ4C7G6trJoyntuOAz1bV7QBV9a+N1NXvSOCMaagLxldbAY/ulrcG/u9oAxpQM2t74F/6Hv+iaxu2T1X9AbgDeGwDdc2EidZ1LPC9Ka3oIeOqLclbkvycXhCc0EptSfYAdqiq70xTTeOqq3NYdzrorCQ7TE9p46rtacDTklyWZHmSQxqpC4Du9PaTgR9MQ10wvtreD7w+yS+A79Kb4Y3IgJpZw82Ehv5VPZ4+k20m9jke464ryeuBAeCTU1pR3y6Hafuj2qrqs1W1E/DfgPdMeVU9o9aWZBPgFOCvpqmeB3c9TNvQY/bPwIKqeiZwAQ+dTZhq46ltM3qn+Q6kN1M5Lck2DdS1zhHAWVV1/xTW0288tR0JLKmqJwIvBv6x+/c3LANqZv0C6P+L8In88ZT3wT5JNqM3Lb6tgbpmwrjqSnIQ8G7g5VV1T0u19fkqcOiUVvSQsWrbCtgNWJrkZmAf4NvTcKHEmMesqn7T9zP8IrDnFNc07tq6Pt+qqvuq6ibgenqBNdN1rXME03d6D8ZX27HA1wGq6nLgkfRuJDu86XjxzK8RX1TcDLiR3jR83YuKTx/S5y08/CKJr7dQV1/fJUzfRRLjOV570HuhducGf5Y79y2/DBhspbYh/ZcyPRdJjOeYPaFv+ZXA8laOGXAIcHq3vB2901uPnem6un67ADfT3YyhoWP2PeCobvlP6AXYiDVOS+F+jfpDfTFwQ/dL9d1d2wfp/fUPvb8wzgR+BvwIeEojde1F7y+mu4DfANc2UtcFwK+Bld3Xtxv6WX4auLar66LRQmK6axvSd1oCapzH7KPdMVvVHbNdWzlm9E5pfQq4DlgDHNFCXd3j9wMfm65jNYFjthC4rPt5rgQOHm08b3UkSWqSr0FJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkpr0/wGc3+l42MRneQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "# Create linear regression\n",
    "regressor = Lasso(random_state=0,alpha=0.1)\n",
    "\n",
    "# Fit/train LASSO\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1e-08, 100000000.0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAF3CAYAAAA4gEgdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXyU1fX48c+ZLXsyIQRISNh3kAREVCQKIrjvrXttq9bW5dtWbf1Zu7m29Wur/Vpt1dpatRbUVisogjvuAkKQfZF9J4QkkG22+/vjmQhhQubJMgmZOe/Xa5zJnGdmDjGZObnPvfeIMQallFJKJS5HZyeglFJKqc6lxYBSSimV4LQYUEoppRKcFgNKKaVUgtNiQCmllEpwWgwopZRSCc7V2QnEQvfu3U2/fv06Ow2llFKqQ3zxxRdlxpjc1j4+LouBfv36sXDhws5OQymllOoQIrKpLY/X0wRKKaVUgtNiQCmllEpwWgwopZRSCU6LAaWUUirBaTGglFJKJTgtBpRSSqkEp8WAUkopleC0GFBKKaUSnBYDSimlVILTYkAppZRKcFoMKKWUUgkuLnsTBAKwZ0/j+7p1A6cTqquhpibyMTk54HAcOd69O4jA/v1QVxcZzw23h6iqgvr6xjER6/EAlZXg8zWOOxzW6wNUVIDf3zjudFr5A+zbZ/37DuVyQXa2dbu8HILBxnG3G7xe6/bevRAKNY57PJCVZd0uKwNjGseTkiAz07p9+PcVIDkZMjKsx5WVRcZTUyEtzXrdvXuPHA8GrfwPl54OKSnWv3vfvsh4RoaVg99vff8Ol5lp/Rt8vqbjWVlWvK7O+v93uOxs63tYX2/9bDgc1kXEuk5Otq5DIet7IHLwopRSXYIxJu4ucKyx3pYPXpas8JmKap/55V2BiBgYs36zFb/1J03Hd++z4t/7QWTc7Q6ZimorfvmVwYh4dreD8XPPj4wX9jkYn3xqZHzEiIPx8cdHxscfH/w6PmJEKCI++dSD8cI+kfFzzz8Yz+4WGb/8yoNxtzsy/r0fBExFtc/s3udr8nt360+s+PrNTcd/eZcVX7Ki6fgDv7fiH3/edPyxx/2motpn5r7jbzL+zPNW/OVXm46//KoVf+b5puNz37Hij/6l6finC32mqtZnHvh9458NkZBxOEJm+Sq/qa73m/t+EzApKSGTlhYyGRkhk5UVMl5vyGzbGTC1voC5+56g6dkzZHrlhUx+fsj0LgiZgsKQqToQNPX+oPnVr4Nm0OCQGTI0ZIYOC5kRI0KmqDhk/IGgCQRD5he/DJmxY0Nm3LiQGT8+ZE44IWTOOCNkQqGQMcaYOXOM+ewzY8JfKqXiCLDQtOFzMy5HBnrlh7jmhtpG91XjY3M5jDzOwZ33Rv6z99b7OFAOYyY6uTPLGRHfVunD6YQTpzjJ7d04Lg7YXG79uT/5HCf9hzWOu5MMm8utP/fPuNjFMeMan51JTT8Yv+BKFydObhzP9B6MX3qNi9POaRzP6RFic7k1XHD1DW6qKhr/Sdoz/2D8+7e4qTnQON6778H4D+9w469vHO83KMjmcmu44fa7PJjDRhYGDbPiwSDcea+Hww0/xorX1zcdHzk2wObyENU0HR842ooHU4Q77w1ExHsPseKuLOHOe90R8e59rHhaT+HO+yLj6b38bC43dO/j4M77In82XFlWvPcQB7ff5cKErFGAhrKo3uVnY5mhYIiTm25zfX1/w3H7/PUEd0PPAU4u/bbbGkFoeHwItlTWsdcPWXkuTp7qIhQSa/TGWI9ft6cWlwvcmW4Gj3AdfKwRnA7Dyh3Wz3q9eEjPdmHCjzMh8Ass22YNdd3603RWLHUycqTh2muFb33r4IiVUiqxiVVQxJeRo8eYGbPf7+w0lDqqVB+AObPcvDLdw5eLXbjdhnvuEe64o7MzU0q1lYh8YYwZ19rHx+XIgFIqUlo6XHy5n4sv97N2lYNXZnjw9g6ye7+D6nIPzzzt4Lvfhb59OztTpVRH09UESiWgwcNC3H5XHSdN9rOrsp4Zr9Zxzz2G/v0Np59uePHFyImwSqn4pcWAUopzLvLzxif7+f6P61m63HDppdC3r2lyZY1SKv7oaQKlFAD5BYYbb63n+z+q57MPXaxe6WBHdZBuxsPdP3czcqRw2WXWUk6lVHzRkQGlVCNOJ5w0KcA1N/ioqQ+ybnsts+eGuP56yMszXHMNfPxx5H4USqmuS4sBpVSzklPg328e4LlXD3D6eX5efNEwcSI88WQo+oOVUl2CniZQSkUlAkVjgxSNreX2X9cyd5abEScG2FLu4q1ZSbw+08l118G0adbIglKqa9GRAaVUi6SmwYWX+cnuZqio8bNhu4/33g9x1lnQr5/hV7+CDRs6O0ulVEtoMaCUapNLvuXj7QX7+cMT1fQdFOC++wwXXNiwNXhkLwyl1NFHTxMopdrM7YGpZwWYelaAnduFst0OVu4I4Qq6OW1iMhdeIFx7LYwe3dmZKqWaoiMDSql21SvfMKo4SDBk2LTTz8hiP3953FBUBOPHG554wur+qZQ6esTlyEDFvnJefelfje5z5/QhKW8wJhigeuW8iMd4evTH02MAIX89Nas/ioz3GoSne19C9dXUrP0sIp6UPwx3t94Ea6uo/WpBRDy5YCQuby+C1fuo3bAoMt5nNK7MXAJVe6jb/GVEPKX/WJxp2QQqdlK3dXlkfOBxOFMy8Zdvo377qoh46uATcCSl4SvbhG/nusj40Ik43En4dq/HtzvyhG/a8FMQp4v6HWvx790cEU8fNQWA+m0r8e/b3jjocJI2YpIV37KMQOWuRmFxeUgdWgJA3aZSggca9zl2eFJIGTwBg6FuwyJCNY37GDuSM0geON56/FfzCdY1/qRxpmaT1H+sFV/7KSFfeCed8NI4Z0Z3PH2KrPiajzD+xlvvObN64ikYhQjUrpwHoYZmSVZDJ1d2Psm9h1strpe+c0jE4sntQ3LeEAgFqVr+XkQ8uddAUnoNxPjrqFj+wcHvS/g6tfdQUnv2I1hXzb6VH3O4zD4jSM0txF9dSfmqw342Bbz9R5Oak0+Oo5aNSyIfP3LcSeT0zGfvru0sXxgZH33CKXhzerB722ZWlX4eER9z0hQyvN3YsXk9a5d+ERG/64EzuPO+TN54eR0Vu5fw2WeCz2/ISBeCQbjssktIS0ultLSU0tLSiMdfeeWVuN1uFixYwPLlkT/73/nOdwD45JNPWLNmTaOY2+3myiuvBGDevHlsOGwyQ2pqKpdccgkAb7/9Nlu3bm0Uz8zM5KKLLgJgzpw57Ny5s1E8JyeHc889F4BZs2ax97Ae3b169eKMM84A4OWXX6bqsB7ZBQUFnHbaaQC8+OKL1By2y1P//v055ZRTAHj++efxH9bffMiQIUyYMAGAf/zjHxxu5MiRHHfccfj9fp5//vmIeHFxMcXFxdTU1PDiiy9GxMeNG8eoUaOorKzklVdeiYifeOKJDB06lLKyMl577bWI+Mknn8yAAQPYuXMnc+bMiYhPmTKFwsJCtmzZwjvvvBMRP+OMM+jVqxfr16/ngw8+iIifc845dO/endWrV/Ppp59GxC+88EKysrJYtmwZCxcujIhfcsklpKbGx89eW3VqMSAiZwD/BziBp4wxvzss/h3gQWBb+K5HjTFPRXteXzDE1n2Nf6k2lu1m1QpwEmSqJ3JbtXVlu1i3NEgSfiY3EV9dtpMNQR9p4qPEHRlfvmc7W0I1ZEodE5qIL9mzjR2h/XSTGsY3EV+0Zyu7QxX0cBxgrCsyPn/3ZspNGXmOKoqaiH/y6SaqTDKFjgpGNhH/8OONVBsP/Z3lDHVGxt/74CvqcTPIWcagJuJvvb+OIE6GOXfTr4n4nHfWAjDKVUaBo3E8iIO3wvEi117yDovX4+O9bVb8WNc+cg+L15gAH2yx4uPdFXSTxh0pq4zhk01WgTPBXUWm1DWKl5cL8zdY8ZPd+0kVX6P4nvJ9fPGVFZ/sOUASjTsj7igvZ8laKz7VU42TxifBt+4tY9kqK78zmvjZ2Vi2m1XLmvnZ27ODdaX+I//s7dnGhmBt+GevOiK+fNcWtoT2h3/2IuNLdm5iR2gfee46zu7mJzs1snNjrHmzDSWTA6xfGaSmRnC4DbU+Yf06ByeeaLjqKjj++A5PSykV1mldC0XECawBpgJbgQXA5caYFYcc8x1gnDHm5pY895BRReaxF99sx2xVa0n0Q1p0nN0DxeaBYvP5Gn5NDF/faLhlXR8SP/RX6tDHfX33oY895OCDxx7y30bHHvq8hzzfIfEj5ecPhpg+fzPLt1dx0sAcbpw0iMyUji8KDvf6K25e+qeHRfNdOJ2Gc88Vbr4Zpkzp7MyU6lq6ctfC8cA6Y8x6ABGZAZwPrGj2UTa4HQ56ZSa39WmUiiuj8rN4efFW/vX5ZlbuWMyPThvM2D7ZnZrT2Rf6OftCPxvWOXjlBQ+z/u2md58QJac4cIqDTZtgwIBOTVGphNCZEwh7A1sO+Xpr+L7DXSwiX4rIv0WksGNSUyr+OB3CN48t5PffLCItycmvZy7niQ++oj4Q7OzU6D8oxK0/r+PN+fu58gfVrN65n+f+XcfAgXDqqYbnn4fa2ujPo5Rqnc4cGWhqgPbwkc9ZwHRjTL2I/AB4Bji1yScTuR64HiCvt9YMSh3JwNx0Hr60mGc+2cisL3ewZEsFt04dyqAe6Z2dGm63dQHoM8jHzT+FV1/0cNVVgtdruOIK4bHHrPicObB6tbXjoctlXdLS4PLLrfinn8KuXdb9Dcekp8OJJ1rx1auhuvrgY10uSE2FggIrXl5unXo5NN7wXErFm86cM3AicJcx5vTw1z8DMMb89gjHO4FyY0xWtOceOXqMmTH7/XbMVqn4tHjzPv74zlqqav1cMb4PF40twOmwPYOjQ4RCsOBTJ6/M8PDJPDeL1liTLG+5IYlX/93475nuuYbFq2sREa65wsObbzT+5O7bL8T8JT4Q+Ma5Hj6c13hwdNQxIT74NIAITDnZxaIvGsdPnGB4890gAkw43slX6xoXCpNPNTzzrHXs1ClCWVnj+Gmnwf33H13fXxUf2jpnoDOLARfWBMIpWKsFFgBXGGOWH3JMnjFmR/j2hcD/M8acEO25tRhQyr79dX4ee/8rPl5Xxoi8TG6ZOuSonXMTCoEj/PlcWwv+eggEhWAAgiHAWPscAGzbIuyvEoIBIRCEUND6QD5mjHVaZMkXTsr3WssbgwEhEID0DMMpp1mrSebOclO2RwgFIRA+pkevEOd9w1re9/TjHvbudnz93MEgDBgc4sprrNUq9/4smYp9DoKBhsfD2PFBrru5nuqdqQwZ4CYnp2O/fyp+ddliAEBEzgL+iLW08O/GmPtF5B5goTFmpoj8FjgPCADlwA3GmMhF9IfRYkCpljHG8N7qPTzxwVcYA98/eQCnDuuB2F1uoWzbuN7Beadk8NhjcOONnZ2NihdduhiIFS0GlGqd3VV1PPT2GpZvr2LCwBxuOkqWIMYTY+C8U9IZOtjB3LlabKn20dZiQLcjVkp9rUdmMvdfcAzfmdCP+RvK+Z/pi1m0aV/0ByrbRGDStADvvQeHbUioVKfRYkAp1YjTIVw8toA/fLOI9GQXv561nCfmHR1LEOPFpKl+/H5h7tzOzkQpixYDSqkmDchN5+FLijmvKJ/Xlu7glhdKWbf7QGenFReKjg3izQ7RDlvKK9UutBhQSh2Rx+XgeyUDuPf8UdT4gvzk30t4aeEWgqH4m2vUkVwu+POzNfz+YR1tUUcHLQaUUlEVF3r50+VjOHFADs9+tok7X1nKzqq66A9URzSqOAgef/QDleoAcbmaIMt7rJlwcuN2lnfcW0e3HMM7b7iYOytydvSv/7eWtHR441U3782N3JjxN4/U4nLBf19w88m8xnGHE373J2uv1Bee9fDFZ403OklJg7sftOLPPulhWWnjeHaO4Wf3Wm+sTz6SxLpVjWu0nvmG235hxR/7fRKb1jeO9x0Q4qafWG13/3BfMru2N56hPGhYiOt/aMV/+8tk9u1tHB9VHOTq66210b/+aQq1hzW+O/aEIJdebcXv+J8UQof9MTPhlAAXXOonEIA7f5jy9f0NP1qTTw9w1gV+qg9Yz3+4M8/zM+XMAHvLhPt/Hhm/8FIfJacG2L5VePCecPyQH9vLvlPP8ScF2bDOwf/9LnJ9/HdvqKfo2CArlzn4y0OR8RturWP4qBClC5387bGkiPitP6+j/6AQn33o5J9/SzrYUCh8/fP7a+ldaHhvrosXnvNE5Hffw7V072GY/V83/33RHX7swf8HDz9ZTXoG/Ge6m9n/jXz8489X43bDc095eHdOw+OtmMsNT82w/oc9+UgSH7/vahRPzzD85Tlrk549u4ScXGO7OVNTjDG8v2YPj8+zliBef/IApugSxFab8fdk8rolcXOLWrEpFakrNyqKmfo6Yf2axh/4fbNd9M6DDwOwfk3kG9eA7i6ys2FObdPxIT1deDwQPBAZd7thWJ71rfRVCusbt7XGm3UwXrsvMt47H4blWfkeKIuMO0IH4+U7IuMZKTAsz/oQ2bNVWL+ucbxX94PxnZuEbdsbxwf2dTMsz/oQ3LZeqKhsHD9m+MH4pnXCYS3VOf5YF8PykvH5Ir83IuCcZMUrKmDzV5H9BJOCLobnwbYQ7NgU+b1PFyfD88C5H3ZuORhv+Pzp5rHivt1C2Y6Ih5Ob4mJEPlRshIo9kfG89HRG5MOONKjaGxkvyEpnZD6sT4XqisavLQIDcjIYlA/L0yB4yB/LDccM7ZVBXh4sygBHMDI+PC+TrCz4OAOSnZH/vmMKsnC7oXe29f/60LjLZcUBCrtDt8zG8cxMGNU7ix07YMpZhjPO83HbL+taXRCICJOH9mBkXiYPv72G/3tnLfM3lHPT5EFk6RLEFvtwnoNtGw033SRtKtKUaqu4HBkYN26cWbhwYWenodRRwxj44Q/h0UfhOz+o55Y7W18QNAiGDK+WbuO5zzaRkeziR1OGcGzfzu2C2NW89E8P9/4shWXLYOTIzs5GdWW6z4BSKioReOQRuOEG+MfjSTzywMHTHa3ldAgXjS3goUuKyEh2c9es5Tw+7yvq/Dopzq5TTrOG2WbO7OREVMLTYkCpBCFijQx8//vwt8eSeeWF9hnW79/dWoJ4flE+ry/dwS0v6hJEu3r0MowqCjBzZvyN0KquRYsBpRKIwwF//jM8/DBcdUX7naT2uBxcF16CWBtegviCLkG05bSz/aRnmoi5OEp1JJ0zoFQCW7OllhdeNFxwaft9Eu2v8/OXeV/x4doyhvfK4NapQ+mVdXR2QTxaZKe5KchO7ew0VBemcwaUUq323+kp/OonqTzxx8glla2Vkezmp9OGctvUIWwur+GHMxbz9opdxOMfHu1lf12A/fs7OwuVyLQYUCqB3XYbXH01PPaHZP76p/YrCESESUN78MjlYxjUI53/e3ctv31jFZW1OhbelKcfd5OXZ6iujn6sUrGgxYBSCczphL//Ha66Cv70v8n8/c+edn3+HhnJ3HfBKL47oR8LNpbzP9MXsXBTebu+RjwYPDxIdbXw9tudnYlKVFoMKJXgnE74xz/g8sthxj+SqKqM+pAWcUjDEsRiMpPd3D1rBX/RJYiNHHt8kIxMo0sMVafRYkAphdMJzz4LC+Y76Jsfm50E+3dP46FLirmgOJ/ZS3fw4xdKWbtLT5SDtYvpxMl+Zs0yBLVGUp1AiwGlFGBtbVxQAL29qTx8byrPPdW+pwzAWoJ47cQB3HfBKOoDQX76ny95YcFmXYIITJoaYM8e4fPPOzsTlYi0GFBKNRIMQtVeFw/encK/nm7/ggCgqMDLny4by0kDu/PPzzdzx8tfsqOyNiav1VVMnOznV7+pZ9Cgzs5EJSItBpRSjbhcMH26cOGFht/9KoUZz8SmIEhPdvHT060liFvKa/jRjFLeXLEzYZcgZmTCN6+uo1tOqLNTUQlIiwGlVAS3G2bMEM4/3/CbX6Tw0j9jUxAAXy9BHNwjnT+9u47fvLEyYZcgHtgPf/1biM2bOzsTlWi0GFBKNcnjgRdfFC6+2NC3MLZvFT0ykrn3glFcc1I/Fm7cx83TF7FwY+ItQazYJ9z4fRcvvdTZmahEo9sRK6WiMsawpbyWFWsC9C6M7XvGhrJqHnprNRv31nDmqF5cc1J/kt3OmL7m0eQb09Lpletg3rz26x2h4p9uR6yUijkRYW1pCueenMErM2Kz9LBB/+5p/OGbxVxQ3Js5y3by4xdKWZNASxAnT/Pz0UdQVtbZmahEosWAUsqWk04SpkyBu25P4dWXYlsQWEsQ+4eXIIb46b+XMCNBliBOmhogFBJmz+7sTFQicXV2AkqpriE5GV55xZpU+KvbUnA44NyLYzvRb3SBlz9dPobH533F859vZsHGcsb17YY31Y03xY031YM31U1WipsUtxORrj+0PmJ0kJ55IRYvdnD11Z2djUoUOmdAKdUiNTVw7rmG99+HGbMPMGxkxyyF+2DNHp7+ZANlB3xNxj0uB94UN9nhAsGb4iYr1RMuGg4WDtkpHtKSju7CoaoSxg5OJ8WTOHMlVNu0dc6AjgwopVokNRVmzRKmTzeMG+vgQH3HFAMnD8nl5CG5BIIhKmv9VNT6qajxU1HjC9/2fX3frqo6Vu/aT1Wtn6bOLLgcQtahRULKodeNb2cku3E6OrZwyMyCqjq/FgOqw2gxoJRqsdRUuPZaIRRK5c2Pa1m5ynDamYEOeW2X00FOehI56dFbLgdDhgP1AatQqPGz75CCobLWFy4m/GzaW01FjZ9AE5WDQyAzObJI8KZ6DhYUKR6yw6crXM72mYp10/ed9C+ABx9sl6dTqllaDCilWs3hEJ54KIVZs+CBx2qYdnbHFAR2OcMjAFkpbvrmNH+sMYZqX/DrwuHr0YbDbu+orKWixk99oOkRkYwkF1mHzWloVESkeOjtTSE9ufm338oq+Ne/DA88IDh0qreKMS0GlFJt8uyzwplnGu64ORWHo6bDRgjam4iQnuQiPclFQXb04+v8wa9PU+w7pFioPOSUxYayaipqfFT7Grci7J7u4W/fPg5HM/MWJk318/ZsN4sWwbhWnwlWyh4tBpRSbZKRAbNnC2ecYbj9xlR+/5caTj2jaxYELZHsdtIry0mvrOSox/oCoa+LhM83lPPCwi1sLKtmQG76ER9TcmoAh8Mwc6ZoMaBiTgeflFJtlpkJc+YIxx4L/30hiThcpNQmHpeD3IwkBvfM4MxRvQAo3VLR7GOyuxnGHBdk5kz9ZqrY05EBpVS7yMyEuXMFl8vJrhonB+qCHMWr9zpNTnoSfbqlUrqlgovGFjR77MVX+CjbkkQw6MSpCwtUDOnIgFKq3WRlQVqa4HWmcd2l6Xz4nv690ZTiQi/Lt1fhO8IkxAbnXOTnxz+r10JAxZwWA0qpdiciBOsd3PK9VD5+XwuCwxUXevEFQ6zcWRX12LJKP/Pn66kCFVtaDCil2l12Nrz1ljBiOPzoulQ+/UALgkONzM/E6RCWRJk3APDnh5I46SSoiH6oUq2mxYBSKia6dYO33xaGDYUfXpvK/E90rLtBqsfF0J4ZLLZRDJw8JUAgIMyZ0wGJqYSlxYBSKmZycuCdd4TTpkD/fjqb8FDFhV6+2n2A/XXNN3s6ZkyQbjkhZs7soMRUQtJiQCkVU927w2uvCaeMSyXJ5WDDOn3bAasYMMCXWyubPc7phFNOCzB7tsEf2yaRKoHpb6VSqkM4HcL0J9O57Kx0Fn6mpwyG9Mwgxe2Mut8AWLsRVlYKH3zQAYmphKTFgFKqw/zg+0LfvnDzt9NYND+xCwKnQxhdkGWrGDjh5AAvvFbDpEmxz0slJi0GlFIdpmdPePddobAQbro6jdKFiV0QFBd62VlVx87KumaPS0mB4UV+QnRMu2iVeLQYUEp1qLw8qyDIz4cfXZtKTXVnZ9R5igq9ACzZGn10YOcO4cabDCtWxDorlYi0GFBKdbj8fHjvPWHGdOjmTdy3oQJvCjlpHltLDF1O+NuTDl5+uQMSUwkncX8LlVKdqndvmDbNQf/uacyd6WFZaeKdMhARigu9fLmlglCU7k7dexiOGaONi1RsaDGglOpUAb+DJx5O5gdXpbHiy8R7Syou9LK/PsD6PdHPl0yaGmDBAmH79g5ITCWUxPvNU0odVZKTrZ0Ku2XD969MZ+WyxHpbapg3YHeJIcCsWTFNSSWgxPqtU0odlfr2teYQZGXC9Veks3pF4rw1Zad66JeTSumWfVGPHTgkRNHYADU1eqpAta/E+Y1TSh3V+vWzCoLMdJj/kbuz0+lQxYVeVuyooj4QbPY4EXju1Wquv7H545RqKS0GlFJHjQEDYMkS4b5fefC4HAQT5DOvqNCLP2hYuWO/reOragNUJ/CSTNX+tBhQSh1VsrPB43JQuSWNb0xNZ93q+H+bGpWfhcshtk4VGANTT3Fz440dkJhKGPH/W6aU6pKyMh3UHHDwvcvSWL82vt+qkt1OhvXKsDWJUAT6DQzy+uuGQKADklMJIb5/w5RSXdbQodZOhS6ncN2laXHf7bC40Mv6PdVU1kZvTTh5mp+9e4VPP+2AxFRCiO/fLqVUlzZsmDWp0CHCdZelsXWzdHZKMVNcmB1uaRx9dGDCKQHcbsPMmbHPSyUGLQaUUke14cOtgmDaVKuvgTEwd5ab1Ssc1NV2dnbtZ1CPdNI89loap2fAcRMCuhuhajeuzk5AKaWiGTEC/vmcg2Aog01bg/z0RuutS8RQ0NcwYFCQS672UTI5QCAAvnpITevkpFvIamnspXRLBcYYRJofBfnOD+rxBCEUcuPQP+tUG2kxoJTqMpwOoU++i6VLYcUKWLFCWLlSWL5CSAk56J0dZP6CEGdOTiavd4gBg4MMHByi/+AgJZMD9Oh1dP8lXVTo5dP1e9lZVUdeVkqzx54wMUhmig+HI7H2ZFCxocWAUqpLcblg1CjrcpAATsDJiAFw332wcqWDFSuEF5+DujrhxVd9DOzn5913hCcf9TBgUJABg0MMHBxkwJAQ3XI6v1AoLji4NXG0YgBg2YoQ76w1XHNN/M6lUB1DiwGlVFzp0zed9KEAACAASURBVAd+/vOGr4RgEDZtgl69PKSmeuiZDsZnmPVvJwcOHPwQfeOjAxT0C7LgEyerVzqtImFwiNyehigj9u0m35tMbkYSpVsqOHNUXtTjX/+vm8cfhnPOgR49OiBBFbc69UyTiJwhIqtFZJ2I3NFEPElEXgjHPxeRfh2fpVKqK3M6rZ0NU1Otr88+Gz7/XKiqEjZvhrlz4aGHYMr4dEbmZ1L6SSoP/DqF669I57TjMpk4KpNvXZCGP7zib+N6B9u3CqFQ++cqIhQXePlyayXBUPSRiklT/RgjvP56++eiEkunjQyIiBN4DJgKbAUWiMhMY8yKQw67FthnjBkkIpcBDwCXdny2Sql4IwKFhdZl2rSv7+WPDwl33tEwJ8Gal7Bnj5MRBenUB0Lc/gMHc153kpJi6B8ePRhVFOSK7/oAa7VDW0YSigu9vLVyF1/tOcCQnhnNHjtsZIhe+SFmznTw3e+2/jWV6szTBOOBdcaY9QAiMgM4Hzi0GDgfuCt8+9/AoyIixpjOP7mnlIpLItCzp3WZPPnrewEnyW4n990NF57XMHnRxeLPDbu3urj1xw7q/EEuOiuJqioaTV4cNiJEn/72hhJGF2QB1ryBaMWAiDU6MPMlD7W1Qkr0aQZKNakzi4HewJZDvt4KHH+kY4wxARGpBHKAssOfTESuB64H6NOnTyzyVUopjj3Wuhwk+HyCx5MEwNlnwMKFsHypgzkzrSGCs88L8H9P1VHvD/HzW5IZc1yACy9reqdBb6qHAd3TWLKlgkvGFUbNZ9LUAP/5l4clS+CEE9r6r1OJqjOLgaYG0g7/i9/OMdadxjwJPAkwbtw4HTlQSnUYj+fg7bvuarglVFfD6tXgcrkYmJtOTQ3M/8jw2stuTj/P//U8hsMVFXqZtWQ7df4gyW5ns6993IQAn684wJhBzY8iKNWczpxAuBU4tOwtALYf6RgRcQFZQHmHZKeUUm2UlgZjx8Lo0dbXqanw178KgYCwrPTIH/LFBV4CIcOK7VVRX8PtBldyiPpAgvR7VjHRmcXAAmCwiPQXEQ9wGXD4TtszgW+Hb38DeFfnCyilurIJE6ydExfNP/LA7Ij8TKulsY0+BQBfrXFQMlGYP7+9slSJptOKAWNMALgZmAusBF40xiwXkXtE5LzwYX8DckRkHXArELH8UCmluhKv1xopWDz/yCMDyW4nI/IybfUpAOiea1i0UHj11fbKUiWaTt10yBgzG5h92H2/OuR2HfDNjs5LKaVi6ac/FXbvDzR7THGhl2c/20RFjQ9vqqfZY7OyDWPHB5k508n99+tuhKrltL2FUkp1sCuvhEsva/6YokJra+IlWyttPeekaX6WLRPWr29rdioRaTGglFKdYNsGF2tXHfkteGBuOulJLpbYPFUw6TRrpGHm4TOvlLJBiwGllOoEl33DxZ8fSj5i3GppnMXicEvjaAr7hbj4Ch/9+ukca9VyzRYDIuIUkQc7KhmllEoUEycKi+c7ae5zvrjQS9mBerZX1Nl6zl8/UMuUM5qfi6BUU5otBowxQeBYkY7q2aWUUomhpATK9zrYuP7Ib8PF4XkDdpcYAmzcGtB5A6rF7JwmWAy8KiLfEpGLGi6xTkwppeJZSYl1vaiZJYa9MpPpkZFE6ZZ9tp7TGJh6chI/+1l7ZKgSiZ1ioBuwFzgVODd8OSeWSSmlVLwbMgRycw2Lm9l8SEQYU+hlqc2WxiIwcXKAN94w+Hztma2Kd1H3GTDGaGNMpZRqZyIwa5bgT61v9riiQi9zV+xi7e79DOuVGfV5J031859/eXj//UNbMyvVvKgjAyJSICKviMhuEdklIv8RkYKOSE4ppeLZ8cdDn/zmGxGNLvAiYHuJ4fEnBUhJMbrEULWIndMET2P1CMjHaik8K3yfUkqpNqipgb/92cMXnx+5IMhKcTMgN8321sTJKXDiyQFmzjTNrlRQ6lB2ioFcY8zTxphA+PIPIDfGeSmlVNzzeOCh/3Uyd5a72eOKC72s2rmfWp+9zoT/c3sd/53tQ9eBKbvsFANlInJVeM8Bp4hchTWhUCmlVBu4XHDiibB4QfPTt4oLswmEDMt32NuaeOCQEN166X4Dyj47xcA1wCXATmAHVivha2KZlFJKJYqJE4U1Kx1UNfM5PzwvA7dTKN1sf7+BOXPgttv0PIGyJ+oOhMDFxpjzjDG5xpgexpgLjDGbOig/pZSKayUlYIyw5Isjjw4kuZyMzM9iSQs2H1q9wsFDDwlbtrRHlire2dmB8PwOykUppRLO8cdDWpph25bmB2qLCrxs3FvDvmp7GwhMnmadJpg1q80pqgRg5zTBxyLyqIiUiMjYhkvMM1NKqQSQmgplZXD5d5r/kC/+uqWxvdGBfgND9O0fZOZMPVWgoou66RAwIXx9zyH3GawdCZVSSrVRcrKQ7HY2u1pgQG4aGUkuSrdUMGloj6jPKQKnTA0w/WkPVVWQGX2/IpXAos0ZcAB/McZMPuyihYBSSrWTVavgygtSKV145P0GHCKMLvRSarOlMcDkaX4GDjY6b0BFFW3OQAi4uYNyUUqphJSbC/M/dbDg0+YHa8cUetlb7WNrRa2t5x07Psh/36lm5Mj2yFLFMztzBt4SkZ+ISKGIdGu4xDwzpZRKEDk5MHKkYfGC5rcmLmpoaWxziaEI+AIhKg8ECdrbr0glKLv7DNwEfAB8Eb4sjGVSSimVaCZOFEoXupr90O6VmUxeVnKLlhh+udhJQb6DDz9shyRV3IpaDBhj+jdxGdARySmlVKIoKYED+4W1q6IvMfxyayWBYMjW8w4aEqS+Hm1cpJplp2thqoj8QkSeDH89WETOiX1qSimVOE4+GaaeHiIUZTi/uNBLrT/I2t0HbD1vaprVyfDVV7VxkToyu10LfRxcYrgVuC9mGSmlVAIqLITXXoMRo5v/i390QRYCtrsYgrUB0fr1wooVbUxSxS07xcBAY8z/An4AY0wtoL2wlFKqnXlcDqr3S7N/wWckuxnYI71FxcDJU/yAnipQR2anGPCJSArWRkOIyECgPqZZKaVUApo+HU4ckcnWTc2/NY8p9LJ6135qfPY6E/bMM9xxdy1nnqnnCVTT7BQDvwbmAIUi8jzwDnB7TLNSSqkENHq0db1ofvQlhsGQYdm2KtvPfcU1PgYM07bGqml2VhO8BVwEfAeYDowzxrwf27SUUirxDB8O3boZFi1ofvOh4b0y8bgcLVpiGArBrNeDfPxxW7NU8chObwKMMXuB12Oci1JKJTSHA046SVgcZWTA43IwMi+TxS2YNyACt//Yw7FjdO6AimTnNIFSSqkOUlICG9c72bun+XnaxYVetpTXsPeAvSlcIjBpmp+33jLU1LRHpiqeaDGglFJHkfPPhwf/6MftaX6y38GWxpW2n3vSVD91dcJbb7UpRRWHjlgMHNqHoKlLRyaplFKJYsgQuPEGITOr+eP6dU8jK8VN6ZZ9tp/72OODZGQaPU2gIjQ3MtDQg+ALYA+wBlgbvv1F7FNTSqnEVLbTyftvNT+lyyFCUUEWS7ZU2m5p7HbDxMl+PvtcdyNUjR2xGDikB8Fc4FxjTHdjTA5wDvByRyWolFKJ5qmnhB9fl0p1lB2Hiwq9lNf42FxufxLAnffW8c5HPkS3jlOHsDNn4DhjzOyGL4wxbwCnxC4lpZRKbCUlEAoJS75ofnTg4LwB+6sKsrINNX5/m/JT8cdOMVAWblTUT0T6isjPgb2xTkwppRLVCSeA02mibj7UIyOZ/KxkFm+2XwwAPP2Uk9NP1/ME6iA7xcDlQC7wSviSC1wWy6SUUiqRZWRAcTEsjrL5EFinCpZtt9/SGMDvhzffFNasaUuWKp7YKQamGGN+ZIwZY4wZa4z5MXBarBNTSqlEVlIiLC11Em1Ef0yhlzp/iNW79tt+7klTrSedNastGap4YqcY+JnN+5RSSrWT226DTxbV4XY3f9wxBV4c0rKWxr0LDUOGB5k5U08VKMsRx6BE5EzgLKC3iDxySCgT0G4XSikVQwUFkFHrYvNeX7PHpSe5GNwjgyVbKrjy+L62n3/SVD9PPZrE3r2Qk9PWbFVX19zIwHasfQbqsPYVaLjMBE6PfWpKKZXYXv23k789lhT1uKJwS+Pqevt/p512lp+LLw1SXd2WDFW8aG6fgSXGmGeAQcaYZ8K3ZwLrjDH2t7xSSinVKh/Mc/CPx5MIRZkbWFzoJWRg2Xb7WxMPGxninodq6NOnjUmquGBnzsBbIpIZ3oJ4CfC0iDwU47yUUirhlZRAZYXw1Zrm36qH9cogyeWgtIVLDH1+w8LFQert9TpSccxOMZBljKkCLgKeNsYci64mUEqpmCspsa4XzW9+iaHb6WBkfhalLdh8CODzj5wcN9bJu++2NkMVL+wUAy4RyQMuAV6LcT5KKaXC+veH/HzD4gXNbz4E1hLDrftqKbPZ0hhgzHFBUlO1cZGyVwzcg9WfYJ0xZoGIDMBqWKSUUiqGRGDSJKG2JnojgaLw1sQtWWKYlAwTJgWYOVMbFyW6qMWAMeYlY8xoY8yN4a/XG2Mujn1qSiml/vlPeGZ69F4C/XJS8aa4WdKCYgCsJYbbtwuLFrU2QxUP7IwMKKWU6iQikOqJfppARCgq9FK6tcJ2S2OAklMDOByGV19tS5aqq9NiQCmljnLXXOXhf+9KjnpccaGXiho/m/bab2mc3c3w5PRqbr1NzxMkMi0GlFLqKOf3CZ9+aKNpUUHL5w0AjJ8QRNy6sWwia1ExICK6mkAppTpYSQl8tcZJxb7mJxLmZiTR25vS4iWGwSD85reG//ynLVmqrqylIwO9Y5KFUkqpI2rYb2DxfHtLDJdtq8TfgpbGTie8NMPJ44/rqYJE1dJiYHFMslBKKXVExx0HSUmGRQtsnCoo9FIfCLFqp/2WxgCTp/l5/32otL+jsYojRywGRORJEblQRDIa7jPGXNMxaSmllGqQlAQ33ggDhwSjHntM7ywcQiuWGAYIBIQ33mhtlqora25k4O9AETBbRN4Rkf8nIkUdlJdSSqlDPPSQcMW3og/9pyW5GNIzo8WTCI8ZE6RbTkh3I0xQzXUt/MwYc5cxpgRrK+LNwG0islhE/i4il3RYlkoppTD1LiqjTCIE61TB2t37OdCClsZOJ0w500+9z/5cAxU/bM0ZMMbsNcZMN8ZcbYwZAzwGDI5takoppRpUV8OIAUlMf8YT9dgx4ZbGS1u4quAXv6njL0/7Wpui6sJsTyAUkakNt40xXxhj7o9NSkoppQ6XlgYjR8IiGysKhvTMINntoHRry2YDikBVbYC6utZmqbqqlqwmeCBmWSillIpq4kThyy9cBKKM/rudDkblZ7V4EiHA/b92M3KkNi5KNJ2yA6GIdBORt0Rkbfg6+wjHBUWkNHzRaS1KqYRWUgI1NcKqZdFHB4oLvWyrqGX3/pb9md+nf5D164Vly1qbpeqKmi0GROTp8GTBp4E+4dt/F5G/t/F17wDeMcYMBt4Jf92UWmNMcfhyXhtfUymlurSJE61rO6cKisMtjVu8xPA0a9hBGxcllmgjA/8Anglf7wvfbri0xfmHPMczwAVtfD6llIp7+fnw5F9DTJoafZVAn26pZKe6W7zEsHsPw+gxAWbO1PMEiaTZ7ayMMfMabovI/kO/bqOexpgd4dfYISI9jnBcsogsBALA74wx/22n11dKqS7pe9c5WL0TfFHqgYaWxos3VxAyBodEX5LYYNK0AI884GL7dqsAUfEv+t6WB7VovYmIvA30aiL08xY8TR9jzHYRGQC8KyJLjTFfHeH1rgeuB+jTp09LUlVKqS6jshLemuVhWHE9PfOa/+t9TKGX91fvYWNZNQNy022/xunn+Ome7SAlJfoyRhUfbE8gNMac0JInNsacZowZ1cTlVWCXiOQBhK93H+E5toev1wPvA2Oaeb0njTHjjDHjcnNzW5KqUkp1Gbt2wf9cn8SH78WupXFhvxAXf6sOr1dPFSSKTllNAMwEvh2+/W0gYqqKiGSLSFL4dnfgJGBFh2WolFJHocGDoUcPw2IbTYty0pMo7JbKkhZuPgSwr9zw5FMhDhxoTZaqq+msYuB3wFQRWQtMDX+NiIwTkafCxwwHForIEuA9rDkDWgwopRKaiLXfwOL59s7yFhdksWx7Fb5Ay7YZXrXCyQ+ud/Lmm63JUnU1nVIMhLc3nmKMGRy+Lg/fv9AYc1349ifGmGOMMUXh6791Rq5KKXW0KSmBrZsd7NoRfVJgcWE2vkCIVTurWvQaY44LkuU12rgoQdgqBkSkr4icFr6dcmhbY6WUUh2rpMS6Xloafb+BUb0zcUjL5w243TBxsp/XXjMEo3dOVl1c1GJARL4H/Bt4InxXAaBL/JRSqpMUFcGqNUFOOzP6fgOpHhdDe2W2uBgAmDzNz969wqeftiZL1ZXYGRm4CWvyXhWAMWYtcKR9AZRSSsWYywVDBjlw2DzRO6bQy7rdB9hf52/R60w4JYDbbbQYSAB2fpTqjTFf7zEgIi5A15sopVQn+uIL4Y6b06iy8Qd/UaEXA3zZwi6G6Rnw5vz93HJryyYfqq7HTjEwT0TuBFLCbYxfAmbFNi2llFLNOXAAXnvFRekX0VcVDOmRTorb2aolhjndDfvrop+OUF2bnWLgDmAPsBT4PjAb+EUsk1JKKdW88ePB7Ta2mha5nA6O6Z3VqnkDfj9cdYWDRx9tTZaqq4jWtdAJPGuM+asx5pvGmG+Eb+tpAqWU6kSpqXDssdjafAisLoY7KuvYWdWylsZuN2xYL8yYoW/78azZYsAYEwRyRUQ3qFZKqaNMSYmwbImTehuf761taQwwaZqfTz6BPXta/FDVRdg5TbAR+FhEfikitzZcYpyXUkqpKE45BQYPNuzeGf2tvCA7hW5pnladKpg01Y8xwuuvtyZL1RXYKQa2A6+Fj8045KKUUqoTnX02fLowSGG/6LP9RYTiQi9LtlotjVti2MgQvfJCvBrRRUbFi6gnm4wxdwOEdx00xhhtW6GUUkeJNI8LY6yeBdEUF3p5d9Vu1u+pZlAP+y2NReDSb/vISvbQeS1tVCzZ2YFwlIgsBpYBy0XkCxEZGfvUlFJKRfPnx4TTT8iwtWVwcbilcWuWGF57Uz033uKLfqDqkuyUeE8Ctxpj+hpj+gK3AX+NbVpKKaXsyMmBndsdrFkZ/e08O81D326prZo3AFBW4WeF9o6NS3aKgTRjzHsNXxhj3gfSYpaRUkop2xqaFi2y29K40Mvy7ZXUB1refeiOW5KYMsUQ0g0J446dYmB9eCVBv/DlF8CGWCemlFIqusJC6NvXsHhB9M2HwCoG/EHDyh37W/xaJ00KsHOnsHBhix+qjnJ2ioFrgFzg5fClO/DdWCallFLKvokThUXzrYmE0YzMz8LlkFadKpg4KYDTaXRVQRyys5pgH/DDDshFKaVUK1x2GeT0CuL3gSep+WNTPE6G9sqgdMs+oF+LXicr2zB2fJCZM53cf7+N5Quqy7CzmuAtEfEe8nW2iMyNbVpKKaXsOucc+PXdJmoh0KC40Mv6PdVU1raspTFYuxEuWyasX9/ih6qjmJ3TBN2NMV+PJ4VHCnrELiWllFItZfxONqyztwdAcbil8dJtLWtpDHDW+X5efqOOvn1b/FB1FLPzkxMSkT4NX4hIX0A7Viil1FHkuu86+eE1qbaOHdwjgzSPk9LN+1r8Ojm5hiGj63E49GMgntgpBn4OfCQiz4nIc8AHwM9im5ZSSqmWOOkk2LTBSdnu6OfynQ7hmIIsSlux+RDA+nUObrjJsK/ltYQ6StmZQDhHRMYCJwAC3GKMKYt5ZkoppWz7er+BBU6mnR2IenxxgZfP1pezo7KWvKyUFr1WVaXwxF8cbN9qLW0EOPlkuPRSCAbhh01MOZ86FS64AKqr4fbbI+PnngtnnAHl5fDLX0bGv/ENmDwZtm+H+++PjF95JUyYABs2wO9/Hxm/9loYOxZWroRHH42M33QTjBgBpaXw1ya21bv1Vhg4ED7/HJ59NjJ+553QuzfMmwcvvhgZv/tu6N4d3nyTJldj/O53kJEBM2fC3CZm5f3xj1Y76Zdegvffbxxz2ltV2qyoxYCInASUGmNeE5GrgDtF5P+MMZva/vJKKaXaw9ixkJJiWLzAZasYKAq3NC7dUtHiYmBUUZCxxwf46OODg8s+8TNmkg+/H6bPiNyXzpXuZ8SJPiormo5n9fQxYIyfnduF6TMiT3fkDfDRe4Sfr9Y7mD4jMt8Bo+rpPjDA0jVNx0eOrye9d4CFK51Mn5EcER8/uQ5XTpDPlzYdn3ROLcH0EJ+Wupg+I3Km5lmX1FLtCvHRIjfTZ3gi4hd/p4aCoGHe/Kbj37qphm45hvc+9TB9hjsi/oPbq0lKhrc/9PDSYXF35OEtJibKwlQR+RIoAkYDzwJ/By4yxpzS9pePjXHjxpmFuiuGUirBnHqqYWdZiBdmR+8nZ4zhmmcWMqRnOj87c3gHZKdiRQSOKfB+YYwZ19rnsLN/ZcAYY0TkfOARY8zfROTbrX1BpZRSsXHffcLuA/W2jrVaGmfx2fpygiGD06H7BiQyOxMI94vIz4CrgNdFxAm0w6CEUkqp9jRhAkw4wX6L4eLCbA7UB1i/RzvTJzo7PzWXAvXAtcaYnUBv4MGYZqWUUqpV3pnr5oN37DUtKirIAmh1F0MVP6IWA8aYncaYh4wxH4a/3myMaWIupVJKqc728IMO/vG4va0Ivake+ndPa/USQxU/7I8nKaWUOupNnCgsLXXiszd1gKICLyu2V1Hnb3lLYxU/tBhQSqk4UlIC9XXCiqX2WxoHQoYVO6pinJk6mh2xGBCRn4hIYUcmo5RSqm0mTrSuF823VwyMzM9sdUtjFT+aGxnoDXwiIh+IyA0i0r2jklJKKdU6ubkwdKhhzUp7xUCy28nwvEyWaDGQ0I5YDBhjbgH6AL/E2nDoSxF5Q0SuFpGMjkpQKaVUy3z0kfDQX2xOGiDc0risdS2NVXxods6AscwzxtwAFAJ/BG4BdnVEckoppVque3dIS7K/YX1xeGtiHR1IXLYmEIrIMcA9wGOAD7gzlkkppZRqvdpa+OnNScz+r7394QbmppOW5NQlhgnsiDtTiMhg4HLgMiAIzACmGWPWd1BuSimlWiE5GT5430FltYuzLog+9O90CKN7eyndUoExBhHdmjjRNDcyMBdIAi41xhxjjLlfCwGllDr6iUBJibB4gYsovei+VlzoZc/+enZU1sU2OXVUaq4YOB14wxiz9NA7RaRERAbGNi2llFJtUVICO7c72L7V3l/5DfMGFuu8gYTUXDHwMNDULhS1WBMJlVJKHaUO7jdgr09BXlYyPTKSdBJhgmquGOhnjPny8DuNMQuBfjHLSCmlVJuNGgXjjrN5joCGlsZevtxaQTBk/3EqPjRXDCQ3E0tp70SUUkq1H6cTFswXzv+m/b0Digu9VPuCrNutLY0TTXPFwAIR+d7hd4rItcAXsUtJKaVUe0lxuQgE7B07usCaN6BLDBNPc8XAj4Hvisj7IvKH8GUecB3wo45JTymlVGutXAnjhqXywdv25g1kpbgZkJtG6eZ9Mc5MHW2a2454lzFmAnA3sDF8udsYc6IxZmfHpKeUUqq1BgyAmhpYtMBeMQBQXOBl1c792tI4wUTdgdAY854x5k/hy7sdkZRSSqm2S0qC8eNhsc0OhnCwpfGy7ZUxzEwdbWxtR6yUUqprKikRVi5zUlNj7/gR+Zm4naJLDBOMFgNKKRXHSkogEBC+XGRvdCDJ5WREXialWgwkFC0GlFIqjk2YAD++NUjPPPt7BxQVetm4t4Z9Nb4YZqaOJloMKKVUHMvMhN8/6KD/wJDtxxQXaEvjRKPFgFJKxTm/T1i2yI3f5v5DA3LTyUhy6amCBKLFgFJKxbmZM+GK81NZtczevAGnQxhdkMWSrVZLYxX/tBhQSqk4V1JiXS9q0RLDbMoO+NhaURujrNTRRIsBpZSKc3l5MHCgYXFLNh8q1HkDiUSLAaWUSgAlJcLiBU7sjvr3ykqmV2ayzhtIEFoMKKVUAigpgX3lDjass/+2X1ToZem2Sm1pnAC0GFBKqQRw7rkw+y0/hX1bsMSw0EuNL8jaXftjmJk6GmgxoJRSCSA3F06d5MDtsf+Y0b2zEGCxniqIe1oMKKVUgvhysZO/PpJk+/jMFDcDc9NZslWLgXinxYBSSiWIjz+GPz2YzM4dYvsxxYVWS+Nan7Y0jmdaDCilVIL4er+Bz1u2xDCoLY3jXqcUAyLyTRFZLiIhERnXzHFniMhqEVknInd0ZI5KKRVvioogPd2weIH9zYeG52XicTp0iWGc66yRgWXARcAHRzpARJzAY8CZwAjgchEZ0THpKaVU/HG5rC6Gi1qw+ZDH5WBEvrY0jnedUgwYY1YaY1ZHOWw8sM4Ys94Y4wNmAOfHPjullIpfJSXC3t0O6lqwy3BxoZfN5TWUV2tL43h1NM8Z6A1sOeTrreH7miQi14vIQhFZuGfPnpgnp5RSXdFPfgLL19WTnGL/MQ1bE+voQPyKWTEgIm+LyLImLnb/um9quusRt8EyxjxpjBlnjBmXm5vbuqSVUirOJSdDRor90wQA/bunkZns0j4FcaxlPxEtYIw5rY1PsRUoPOTrAmB7G59TKaUS3qN/dPLh5yn89hF75wocIhQVeikNtzQWsb80UXUNR/NpggXAYBHpLyIe4DJgZifnpJRSXd7evcKbr7lbNG+gqMBLebWPLfu0pXE86qylhReKyFbgROB1EZkbvj9fRGYDGGMCwM3AXGAl8KIxZnln5KuUUvGkpAT8fmFZqf0lhmN03kBc66zVBK8YYwqMMUnGmJ7GmNPD9283xpx1yHGzjTFDjDEDjTH3d0auSikVb046ybpuyRLDHpnJ5GUlU7plX4yyUp3paD5NoJRSKga6dYNRrP3YTgAADZRJREFUowyL5tsfGQBrVcGybVUEgvY7H6quQYsBpZRKQJdcIgwYeMQFWk0qLvRS6w+yWlsax52YrSZQSil19PrlL2FbRZDyA/YfM7q3F4fAki0VjMzPil1yqsPpyIBSSiWoVLeT6hYUA+nJLgb1SKd0qzYtijdaDCilVIKaNsnNr36S2qLHFBV4Wb2zihpfIEZZqc6gxYBSSiWoYcOExfOdmBZMHRhT6CVkYNk2HR2IJ1oMKKVUgpo4Ecr2ONi80f5HwbC8TDwuB4t1v4G4osWAUkolqJIS63rR5/aXGLqdDkblZ2qfgjijxYBSSiWo4cMhJ8ewuAWbD4G1xHDLvlrKDtTHKDPV0XRpoVJKJSgReOABwaT5W/S4hpbGj723jp6ZybFITXUwLQaUUiqBXXstbNoLVS3oP9Q3J40ReZms3rVfNyA6CrRHD0ktBpRSKoEFg7ByiRufM0T/gfa2GXaI/P/27j9I7rq+4/jzvfcjiYHw23SEhGg5iBWMhDQEydHgROogVo11StuZBrWNM60//ihj6TBVW6cKjEINMEBoRVoHqgVRQCXVUUCxDLlcEggJCRFQQ1og8iOaEHOX+/SP3YTs3e3l9m73vrff7/Mxc7N7+/nu5v3O93a/r/3+5MoPvLXJlWm0IuCMfxzfa7jPgCQV2MAAfOA9HXz93zuzLkUZMgxIUoF1dMCiRbDuEVcUF5lhQJIKrrs72LKpxG/c/F9YhgFJKrjFi2FgIFjf49qBojIMSFLBLVoEbW2J3kdGf/Ih5YsxUJIK7ogj4KGHoHScJxEqKtcMSJI4++zgmBmuGSgqw4Akieeeg2uvmsLWzS4WisjNBJIkSiVY+aUOomM/p77ZzQVFYwSUJHHCCTB3bmLdGjcVFJFhQJIElM83sL6nnYHRnZVYOWIYkCQB0N0Nu14Jtm1x0VA0znFJElAOA0fOSOzY7qKhaNyBUJIEwMknw3PPJ7a90J91KZpgxj9JElC+FO60KSU62iPrUjTBDAOSpIMeeADe944j2P4LA0GRGAYkSQcdcwxs2Vyi10saF4phQJJ00Omnw9FHJ9atMQwUiWFAknRQqQTnnhtewbBgDAOSpCrd3fD0tjZe/JX7DRSFYUCSVGXpUlj2wf28uifrSjRRDAOSpCpnnQX/8bXEibNS1qVoghgGJElDTG1v44Xn3ExQFIYBSdIQV1wRXHD2kezZnXUlmgiGAUnSEPPnw/79waO9HlVQBIYBSdIQb387lEqJXs83UAiGAUnSEDNmwLx5eCbCgjAMSJKG1d0dPNbbRl9f1pWo2Yx8kqRhLV8Oc+f1kQayrkTNZhiQJA1r/nz43bnBL17MuhI1m5sJJEk1Pb2tjfu/7/fGvDMMSJJq+vI1JT596TQG3FSQa4YBSVJN3d3w8oslnt7m4iLPnLuSpJq6u8u369Z48qE8MwxIkmo65RSYOTN5voGcMwxIkmqKKJ9vYMNaw0CeOXclSSO6+mrYtX8v7kOYX64ZkCSNaNYsmHm8+wzkmWFAknRYN1/XwW23dGZdhprEMCBJOqwH7g/uvM0wkFeGAUnSYS1eHDz5RBuvvBRZl6ImMAxIkg7r4PkGetx3II8MA5Kkw1q4EDo6EuvWeBBaHhkGJEmHNW0aLFkCffuyrkTNYMSTJI3K6tXBUzv72PPbrCtRo7lmQJI0KhEwvbOdlLKuRI1mGJAkjcq+fXDhOzr5t+unZF2KGiyTMBARH4yIxyNiICIWjDDdMxHxWESsj4ieiaxRklStsxP69gU9/+MRBXmT1ZqBjcAy4MFRTHt+SultKaWaoUGSNDEOXLSovz/rStRImYSBlNLmlNKWLP5tSdLYdXfD7t3B1k1uZc6TyT43E/DfEbE2IlZkXYwkFd2Bkw+tfcSD0fKkaXMzIn4A/M4wQ5enlL49ypc5N6W0IyJeD3w/Ip5IKQ27aaESFlYAzJ49e0w1S5JGdtJJsOKjiTlv8oLGedK0MJBSWtqA19hRuX0+Iu4CFlJjP4OU0ipgFcCCBQs88EWSmuSmG4Mt/zfAPvcbyI1Ju5kgIqZHxJEH7gMXUN7xUJKUsV072/n1rqyrUKNkstEnIt4PXAucAHwnItanlP4wIt4A/GtK6UJgJnBXRByo87aU0n1Z1CtJes3WrbDwjGksPr+dWSe/trngU5/dS1sbrL6ng95Hqg8/bGsrjwPcc2cHG9dXj79ueuKTl5VPbXjn7R1s3VQ9PuPoxN/8bXn89q928szPqr/LnjAz8ZcfK4/felMnO7ZXj584a4C/WFE+l/KqlVP41QvVV1984ykDXLy8PH79F6ew65Xq8dPesp9lF/cBcM3np7L31er/kzPO3M9Fy8rjV3xmKmnQVpSzFvVzwbv76euDL/7TVAY757x+lryzn92/gZVXDh1f8s5+zjmvn5dfCm64uvo8D20NONIzkzCQUroLuGuYx3cAF1buPwXMm+DSJEmH0dUFixcnNm1oZ9OG1x6/6fp2Ojrglm3B6rurF6adU+DmG8qLnJ9vHjp+7LFw3TUdADy5IVj9verxWbPgX64qjz/eU+InP66uae6b4crPlcfXP1xiXW/1+ILfh65/6ARgzY9LPLm1evwPlkDXp8rjP/1hiWefrR5vH0h0fbK8EL5/dYlXXq4enzEt0TWzPP6De0tDDr18w/EddM1M7N0Lq+8eulL+1Dnl8Z1tw4+feXp5/OfDPL+jY8jkdYuUw/NKLliwIPX0eI4iSVIxRMTa8ZyPZ9LuMyBJkiaGYUCSpIIzDEiSVHCGAUmSCs4wIElSwRkGJEkqOMOAJEkFZxiQJKngDAOSJBWcYUCSpIIzDEiSVHCGAUmSCs4wIElSweXyqoUR8WtgS9Z1NMnxwM6si2gi+2tt9te68twb5L+/01JKR471ye2NrGQS2TKeSzlOZhHRk9fewP5anf21rjz3BsXobzzPdzOBJEkFZxiQJKng8hoGVmVdQBPluTewv1Znf60rz72B/Y0olzsQSpKk0cvrmgFJkjRKhgFJkgrOMCBJUsEVKgxExOyIuDsivhIRl2VdT6NFRCki/jkiro2I5VnX0wwRMT0i1kbERVnX0mgR8b6IuDkivh0RF2Rdz3hV5tWtlZ7+POt6Gi1v82s4OX+/5frzst7lXcuEgUpDz0fExkGPvysitkTEtlE0fCrwnZTSh4Hfa1qxY9Cg/t4LnAj0AdubVetYNKg/gL8DvtGcKseuEf2llL6VUvor4BLgT5pY7pjV2ecy4I5KT3804cWOQT39tcL8GmwMf6eT8v1WS539TdrPy1rq7K++5V1KqSV+gPOA+cDGQx5rA34GvAnoBDZUmj4DuHfQz+uB44AfAT8EPpR1T03o7zLgo5Xn3pF1T03obylwMeUP34uy7qnR/R3yvC8B87PuqQF9/j3wtso0t2Vde6P7a4X5Nc75N2nfbw3qb9J+Xjaov7qWdy1zOuKU0oMRMWfQwwuBbSmlpwAi4j+B96aUvgAMWa0VEZcCn6m81h3ALc2tevQa1N92YF/l1/3Nq7Z+DervfGA65T/0VyPiuymlgaYWPkoN6i+AK4DvpZR6m1vx2NTTJ+VvWycB62mRtZD19BcRm5nk82uwOuffEUzS91stdfb3Sybp52UtdfbXRx3Lu5YJAzWcSHmGHrAdOHuE6e8DPhsRfwY808S6GqXe/r4JXBsR3cCDzSysQerqL6V0OUBEXALsnOwfTNQ//z5O+dvYURFxSkrpxmYW10C1+lwJXBcR7wbuyaKwBqnVX6vOr8GG7S+l9DFoqfdbLbXm35dprc/LWmr1dyN1LO9aPQzEMI/VPItSSmkj8MfNK6fh6u1vD/CR5pXTcHX1d3CClL7a+FKaot75t5LyArTVDNtnSmk38KGJLqYJavXXqvNrsBH/Tlvo/VZLrfnXap+XtdTqr67lXUusuhvBdmDWIb+fBOzIqJZmsL/Wlvf+Dsh7n/bX2uxvFFo9DKwBuiLijRHRSXlnl7szrqmR7K+15b2/A/Lep/21Nvsbjaz3jqxjL8rbgf/ltcNAPlJ5/EJgK+W9KS/Puk77s7889leUPu3P/ibzTzP780JFkiQVXKtvJpAkSeNkGJAkqeAMA5IkFZxhQJKkgjMMSJJUcIYBSZIKzjAgqS4R8UxEHD/eaSRNHoYBSZIKzjAgqaaI+FZErI2IxyNixaCxORHxRETcGhGPRsQdEfG6Qyb5eET0RsRjETG38pyFEfHTiFhXuT1tQhuSNCzDgKSRfDildBawAPhERBw3aPw0YFVK6a3ALuCvDxnbmVKaD9wAXFp57AngvJTSmcCngc83tXpJo2IYkDSST0TEBuBhyldG6xo0/suU0kOV+18DFh8y9s3K7VpgTuX+UcB/RcRG4BrgLc0oWlJ9DAOShhURS4ClwDkppXnAOmDqoMkGX9zk0N9/W7ndD7RX7n8O+FFK6XTgPcO8nqQMGAYk1XIU8FJKaU9lm/+iYaaZHRHnVO7/KfCTUbzms5X7lzSkSknjZhiQVMt9QHtEPEr5G/3Dw0yzGVhemeZYyvsHjOQq4AsR8RDQ1shiJY2dlzCWNCYRMQe4t7LKX1ILc82AJEkF55oBSZIKzjUDkiQVnGFAkqSCMwxIklRwhgFJkgrOMCBJUsEZBiRJKrj/ByPDeVRQ9WQpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.linear_model import LassoCV\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "lasso = Lasso(random_state=42)\n",
    "alphas = np.logspace(-8, 8, 10)\n",
    "\n",
    "scores = list()\n",
    "scores_std = list()\n",
    "\n",
    "n_folds = 3\n",
    "\n",
    "for alpha in alphas:\n",
    "    lasso.alpha = alpha\n",
    "    this_scores = cross_val_score(lasso, x, y, cv=n_folds, n_jobs=1)\n",
    "    scores.append(np.mean(this_scores))\n",
    "    scores_std.append(np.std(this_scores))\n",
    "\n",
    "scores, scores_std = np.array(scores), np.array(scores_std)\n",
    "\n",
    "plt.figure().set_size_inches(8, 6)\n",
    "plt.semilogx(alphas, scores)\n",
    "\n",
    "# plot error lines showing +/- std. errors of the scores\n",
    "std_error = scores_std / np.sqrt(n_folds)\n",
    "\n",
    "plt.semilogx(alphas, scores + std_error, 'b--')\n",
    "plt.semilogx(alphas, scores - std_error, 'b--')\n",
    "\n",
    "# alpha=0.2 controls the translucency of the fill color\n",
    "plt.fill_between(alphas, scores + std_error, scores - std_error, alpha=0.2)\n",
    "\n",
    "plt.ylabel('CV score +/- std error')\n",
    "plt.xlabel('alpha')\n",
    "plt.axhline(np.max(scores), linestyle='--', color='.5')\n",
    "plt.xlim([alphas[0], alphas[-1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# L2 (Ridge) Regularization\n",
    "\n",
    "You should use Tikhonov/Ridge/L2 regularization when you are less concerned about creating a space network and are more concerned about low weight values.  The lower weight values will typically lead to less overfitting. \n",
    "\n",
    "$ E_2 = \\alpha \\sum_w{ w^2 } $\n",
    "\n",
    "Like the L1 algorithm, the $\\alpha$ value determines how important the L2 objective is compared to the neural network’s error.  Typical L2 values are below 0.1 (10%).  The main calculation performed by L2 is the summing of the squares of all of the weights.  The bias values are not summed.\n",
    "\n",
    "You should use L2 regularization when you are less concerned about creating a space network and are more concerned about low weight values.  The lower weight values will typically lead to less overfitting.  Generally L2 regularization will produce better overall performance than L1.  However, L1 might be useful in situations where there are a large number of inputs and some of the weaker inputs should be pruned.\n",
    "\n",
    "The following code uses L2 with linear regression (Ridge regression):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): {score}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.421393</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007257</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.005385</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.020006</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.138470</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.782889</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>0.994621</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.421393     False\n",
       "weight       -0.007257     False\n",
       "horsepower   -0.005385     False\n",
       "displacement  0.020006      True\n",
       "acceleration  0.138470      True\n",
       "year          0.782889      True\n",
       "origin        0.994621      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -19.07980074425469\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAD4CAYAAABIQCkOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZB0lEQVR4nO3deZgldX3v8fdHQBEHWWRAQsBRBAliBOaIoLIohBA1igEFXMIW5qKJ64U8+rhcN1wTjXHDkWsGvRENCsg1RjYZBpFBemAWQAFliRquNoEQ0bAI3/vHqTaHtteZ7j7V3e/X8/Rz6lT96lffqjnTn/7VqVMnVYUkSW3yqH4XIEnScIaTJKl1DCdJUusYTpKk1jGcJEmts3G/C5grttlmm1q0aFG/y5CkWWPVqlV3VtXCkZYZTlNk0aJFDAwM9LsMSZo1ktw+2jJP60mSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLreCm55o2k3xVIc890fbGFIydJUusYTpKk1jGcJEmtYzhJklpnzodTkm8l2XKcNu9NcshM1SRJGtucvVovSYBU1QvHa1tV75qBkiRJEzSrR05J3pLkuubnTUkWJflBks8A1wA7JrktyTZN+3cm+WGSi5KcleSUZv6yJEc207cleU+Sa5KsS7Jb//ZQkuanWRtOSRYDxwPPBvYFTgK2Ap4GfLGq9qqq23vad4AjgL2APwM6Y3R/Z1XtDXwWOGWMGpYkGUgyMDg4uKG7JElqzNpwAp4HnFtVv6qqe4FzgP2B26tq5Sjtv1FV/1VVvwT+7xh9n9M8rgIWjdaoqpZWVaeqOgsXjvh9WZKk9TCbw2m0z/v/apLtR3J/8/gQc/h9OUlqq9kcTiuAw5NsluRxwMuAy8do/13gT5NsmmQB8KKZKFKSNHmzdlRQVdckWQZ8v5l1BnD3GO2vTnI+sAa4HRgA7pnuOiVJk5earrv2tVCSBVV1b5LN6I68llTVNVPRd6fTqYGBganoStPEG79KU29DIiTJqqoa8eK0WTtyWk9Lk+wObAqcOVXBJEmaWvMqnKrqlf2uQZI0vnkVTprf5tEZbGnWm81X60mS5ijDSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa3jV2Zo3vCbcNV2fq3Lf3PkJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwmKMlG/a5BkuaLORlOSd6X5I09z09L8oYkpya5OsnaJO/pWX5eklVJrk+ypGf+vUnem+QqYL8Z3g1JmrfmZDgB/xs4FiDJo4CjgZ8DuwD7AHsCi5Mc0LQ/oaoWAx3gDUme0Mx/HHBdVT27qr47fCNJliQZSDIwODg4vXskSfPInAynqroN+PckewGHAtcCz+qZvgbYjW5YQTeQ1gArgR175j8EfH2M7Sytqk5VdRYuXDgduyJJ89JcvkPEGcBxwBOBLwAHAx+sqs/1NkpyEHAIsF9V/TrJcmDTZvF9VfXQTBUsSeqakyOnxrnAYXRHTBc0PyckWQCQZIck2wJbAHc3wbQbsG+/CpYkdc3ZkVNVPZDkUuA/mtHPhUn+ALgy3Zus3Qu8Gvg2cHKStcCNdE/tSZL6aM6GU3MhxL7Ay4fmVdUngE+M0PxPRuqjqhZMT3WSpLHMydN6SXYHfgRcUlU397seSdLkzMmRU1XdADyl33VIktbPnAwnaSR+V440e8zJ03qSpNnNcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLr+JUZmjeS/mzXr+qQJs+RkySpdQwnSVLrGE6SpNYxnCRJrdPKcEqyPElnivo6PMnuPc/fm+SQqehbkjQ9WhlOk5VkozEWHw78Npyq6l1VdfH0VyVJWl8bFE5JzkuyKsn1SZY08w5Lck2SNUkuaeYtSPIPSdYlWZvkiGb+oUmubNqfnWTBCNsYsU2S25K8K8l3gZcnOSnJ1c12v55ksyTPAV4CfDTJ6iQ7J1mW5Mimj4OTXNvU9YUkj+np+z3NNtcl2W1DjpMkaXI2dOR0QlUtBjrAG5JsB3weOKKqngm8vGn3TuCeqnpGVf0h8J0k2wDvAA6pqr2BAeAtvZ1PoM19VfW8qvoKcE5VPavZ7g+AE6vqe8D5wKlVtWdV/bin702BZcBRVfUMup/5em1P33c22/wscMpIO59kSZKBJAODg4OTO3KSpFFtaDi9IckaYCWwI7AEWFFVtwJU1V1Nu0OATw+tVFV3A/vSPd12RZLVwLHAk4b1P16br/ZM75Hk8iTrgFcBTx+n9qcBt1bVTc3zM4EDepaf0zyuAhaN1EFVLa2qTlV1Fi5cOM7mJEkTtd53iEhyEN3Q2a+qfp1kObCG7i/932kODP+cfICLquqYsTYzTptf9UwvAw6vqjVJjgMOGm8Xxll+f/P4EN5JQ5Jm1IaMnLYA7m6CaTe6o5zHAAcmeTJAkq2bthcCfzW0YpKt6I62npvkqc28zZLsOmwbE2kzZHPgjiSb0B05Dflls2y4HwKLhvoGXgNcNoH9liRNsw0Jp28DGydZC7yPbpAM0j21d05zum/otNv7ga2SXNfMf35VDQLHAWc1fawEHnHhwUTa9HgncBVwEd3gGfIV4NTmwoede/q+DzgeOLs5FfgwcPr6HAhJ0tRKeVfKKdHpdGpgYKDfZWgM3vhVapckq6pqxM+0zonPOUmS5hbDSZLUOl6FpnnD02vS7OHISZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsevzNC8MdI34fo1GlI7OXKSJLWO4SRJah3DSZLUOoaTJKl1Jn1BRJJ3A/cCjwdWVNXFk1z/IOCUqnrxZLc905IcDtxUVTf0uxZJmk/We+RUVe+abDDNQocDu/e7CEmabyYUTknenuTGJBcDT2vmLUtyZDP9oSQ3JFmb5G96lp+e5PIkNyX5nZFSkn2SfC/Jtc3jUN8bJfmbJOuaPl/fzF+c5LIkq5JckGT7Zv7yJB9PsiLJD5I8K8k5SW5O8v6e7b06yfeTrE7yuSQbNfPvTXJakjVJVibZLslzgJcAH23a77wBx1mSNAnjntZLshg4GtiraX8NsKpn+dbAy4DdqqqSbNmz+iLgQGBn4NIkTx3W/Q+BA6rqN0kOAT4AHAEsAZ4M7NUs2zrJJsAngZdW1WCSo4DTgBOavh6oqgOSvBH4BrAYuAv4cZKPA9sCRwHPraoHk3wGeBXwReBxwMqqenuSjwAnVdX7k5wPfLOqvjbKsVnS1MpOO+003qGUJE3QRN5z2h84t6p+DdD8wu71n8B9wBlJ/hn4Zs+yf6qqh4Gbk9wC7DZs3S2AM5PsAhSwSTP/EOD0qvoNQFXdlWQPYA/gonQ/TbkRcEdPX0N1rQOur6o7mnpvAXYEnkc3sK5u1n8s8ItmnQd66l4F/NEEjgtVtRRYCtDpdPw4pyRNkYleEDHqL95mZLMPcDDdEdZfAS8YZb3hz98HXFpVL0uyCFjezM8IbUM3dPYbpZT7m8eHe6aHnm/crH9mVb1thHUfrPrtvQIewjtnSFJfTeQ9pxXAy5I8NsnmwJ/2LkyyANiiqr4FvAnYs2fxy5M8qnm/5inAjcP63gL4WTN9XM/8C4GTk2zcbGPrZt2FSfZr5m2S5OkTqH/IJcCRSbYd6jPJk8ZZ55fA5pPYhiRpCowbTlV1DfBVYDXwdeDyYU02B76ZZC1wGfDmnmU3NvP+BTi5qu4btu5HgA8muYLuabohZwD/CqxNsgZ4ZVU9ABwJfLiZtxp4zoT2srsfNwDvAC5sar0I2H6c1b4CnNpcsOEFEZI0Q1LTdOfLJMsY42KCuabT6dTAwEC/y9AYvPGr1C5JVlVVZ6Rl3iFCktQ60/bGf1UdN119S5LmNq9K07zhKTxp9vC0niSpdQwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hpPlhpK/BldRahpMkqXUMJ0lS6xhOkqTWMZwkSa0zJeGUZFGS66aiL0mS+j5ySrJxv2uYiNlSpyTNBVMZThsl+XyS65NcmOSxSfZMsjLJ2iTnJtkKIMnyJB9IchnwxiQvT3JdkjVJVjRtNkry0SRXN+v/j2b+QUlWNP3dkOT0JI9qlh2TZF3T14ebea9I8rFm+o1Jbmmmd07y3WZ6cZLLkqxKckGS7UeqcwqPlSRpDFM5GtgFOKaqTkryT8ARwF8Dr6+qy5K8F/hfwJua9ltW1YEASdYBf1xVP0uyZbP8ROCeqnpWkscAVyS5sFm2D7A7cDvwbeDPknwP+DCwGLgbuDDJ4cAK4NRmvf2Bf0+yA/A84PIkmwCfBF5aVYNJjgJOA04YXudwSZYASwB22mmn9T1ukqRhpjKcbq2q1c30KmBnur/YL2vmnQmc3dP+qz3TVwDLmlA7p5l3KPCHSY5snm9BNwAfAL5fVUMjoLPoBs2DwPKqGmzm/yNwQFWdl2RBks2BHYEvAwfQDapzgKcBewAXpftBzY2AO0ap8xGqaimwFKDT6dTYh0eSNFFTGU7390w/BGw5WsPGr4YmqurkJM8GXgSsTrInELqjrgt6V0pyEDA8CKppP5orgeOBG4HL6Y6K9gP+J7ATcH1V7TdenZKkmTGdF0TcA9ydZP/m+WuAy0ZqmGTnqrqqqt4F3El3hHMB8NrmtBtJdk3yuGaVfZI8uXmv6Sjgu8BVwIFJtkmyEXBMz/ZWAKc0j9cCzwfur6p76AbWwiT7NdvZJMnTp+4wSJIma7qvQDsWOD3JZsAtdEcvI/lokl3ojn4uAdYAa4FFwDXpnm8bBA5v2l8JfAh4Bt3AObeqHk7yNuDSpp9vVdU3mvaX0w28FVX1UJKfAD8EqKoHmlOHf59kC7rH5O+A66foGEiSJilVs+utkua03ilV9eJ+19Kr0+nUwMBAv8vQaBKYZa91aa5LsqqqOiMt6/vnnCRJGm7WfbC0qpYDy/tchiRpGjly0vzgKT1pVjGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnzQ9JvyuQNAmGkySpdQwnSVLrGE6SpNYxnCRJrTOnwynJGUl2H6fNsiRHjjB/UZJXTl91kqTRzOlwqqq/qKob1nP1RYDhJEl9MCvCKclfJ3lDM/3xJN9ppg9O8n+SHJrkyiTXJDk7yYJm+fIknWb6xCQ3NfM+n+RTPZs4IMn3ktzSM4r6ELB/ktVJ3jyDuytJ896sCCdgBbB/M90BFiTZBHgesA54B3BIVe0NDABv6V05ye8B7wT2Bf4I2G1Y/9s3fb2YbigBvBW4vKr2rKqPj1RUkiVJBpIMDA4ObuAuSpKGzJZwWgUsTrI5cD9wJd2Q2h/4L2B34Iokq4FjgScNW38f4LKququqHgTOHrb8vKp6uDkFuN1Ei6qqpVXVqarOwoUL12vHJEm/a+N+FzARVfVgktuA44HvAWuB5wM7A7cCF1XVMWN0Md7tAe6fRFtJ0jSbLSMn6J7aO6V5vBw4GVgNrASem+SpAEk2S7LrsHW/DxyYZKskGwNHTGB7vwQ2n6riJUkTN5vC6XK67w1dWVU/B+6j+57QIHAccFaStXTD6hHvKVXVz4APAFcBFwM3APeMs721wG+SrPGCCEmaWamqftcwI5IsqKp7m5HTucAXqurcqeq/0+nUwMDAVHWnqZbAPHmtS7NFklVV1Rlp2WwaOW2odzcXTFxH932q8/pcjyRpFLPigoipUFWn9LsGSdLEzKeRk+YzT+lJs4rhJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJaZ9582WCrJf2uYH7wO52kWcORkySpdQwnSVLrGE6SpNaZFeGUZFmSI5vpM5LsPsn1752eyiRJ02HWXRBRVX8xnf0nCZCqeng6tyNJGl1fR05J/jzJ2iRrkpyb5NYkmzTLHp/ktqHnPessT9Jppu9Nclqz/sok2zXzn5zkyiRXJ3nfsPVPbeavTfKeZt6iJD9I8hngGmDHZrR2XZJ1Sd48E8dDktTVt3BK8nTg7cALquqZwInAcuBFTZOjga9X1YNjdPM4YGWz/grgpGb+J4DPVtWzgP/Xs81DgV2AfYA9gcVJDmgWPw34YlXtBWwD7FBVe1TVM4B/GGUfliQZSDIwODg4uQMgSRpVP0dOLwC+VlV3AlTVXcAZwPHN8uMZJRR6PAB8s5leBSxqpp8LnNVMf6mn/aHNz7V0R0i70Q0rgNuramUzfQvwlCSfTHIY8J8jbbyqllZVp6o6CxcuHKdUSdJE9fM9pwCP+FRkVV3RnGI7ENioqq4bp48Hq377ycqHeOT+jPSJywAfrKrPPWJmsgj4VU8ddyd5JvDHwF8CrwBOGHePJElTop8jp0uAVyR5AkCSrZv5X6Q76hlv1DSWK+ieFgR4Vc/8C4ATkixotrlDkm2Hr5xkG+BRVfV14J3A3htQiyRpkvoWTlV1PXAacFmSNcDHmkX/CGzFf5+WWx9vBP4yydXAFj3bvBD4MnBlknXA14DNR1h/B2B5ktXAMuBtG1CLJGmSUi2731jzeaaXVtVr+l3LZHQ6nRoYGFi/lb233sxo2Wtdmu+SrKqqzkjLWvU5pySfBP4EeGG/a5Ek9U+rwqmqXt/vGiRJ/deqcJq3PN0kSY8wK+6tJ0maXwwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJap3W3b5otkoyCNw+Rd1tA9w5RX1NN2udHtY6Pax1eqxvrU+qqhG/b8hwaqEkA6Pdb6ptrHV6WOv0sNbpMR21elpPktQ6hpMkqXUMp3Za2u8CJsFap4e1Tg9rnR5TXqvvOUmSWseRkySpdQwnSVLrGE4tkGTrJBclubl53GqMto9P8rMkn5rJGnu2P26tSfZMcmWS65OsTXLUDNd4WJIbk/woyVtHWP6YJF9tll+VZNFM1jeslvFqfUuSG5rjeEmSJ/WjzqaWMWvtaXdkkkrSt8ugJ1Jrklc0x/b6JF+e6Rp76hjvNbBTkkuTXNu8Dvr2TeFJvpDkF0muG2V5kvx9sy9rk+y93hurKn/6/AN8BHhrM/1W4MNjtP0E8GXgU22tFdgV2KWZ/j3gDmDLGapvI+DHwFOARwNrgN2HtXkdcHozfTTw1T4dy4nU+nxgs2b6tW2utWm3ObACWAl02lorsAtwLbBV83zbFte6FHhtM707cFs/am22fwCwN3DdKMtfCPwLEGBf4Kr13ZYjp3Z4KXBmM30mcPhIjZIsBrYDLpyhukYybq1VdVNV3dxM/xvwC2DET4FPg32AH1XVLVX1APAVujX36t2HrwEHJ8kM1ddr3Fqr6tKq+nXzdCXw+zNc45CJHFeA99H9A+a+mSxumInUehLw6aq6G6CqfjHDNQ6ZSK0FPL6Z3gL4txms75GFVK0A7hqjyUuBL1bXSmDLJNuvz7YMp3bYrqruAGgetx3eIMmjgL8FTp3h2oYbt9ZeSfah+xfhj2egNoAdgJ/0PP9pM2/ENlX1G+Ae4AkzUt0odTRGqrXXiXT/Ku2HcWtNshewY1V9cyYLG8FEjuuuwK5JrkiyMslhM1bdI02k1ncDr07yU+BbwOtnprT1MtnX9Kg2npJyNK4kFwNPHGHR2yfYxeuAb1XVT6b7j/wpqHWon+2BLwHHVtXDU1HbRDY7wrzhn5eYSJuZMOE6krwa6AAHTmtFoxuz1uaPp48Dx81UQWOYyHHdmO6pvYPojkYvT7JHVf3HNNc23ERqPQZYVlV/m2Q/4EtNrTP1f2oypuz/luE0Q6rqkNGWJfl5ku2r6o7mF/pIpxj2A/ZP8jpgAfDoJPdW1ahvTPexVpI8Hvhn4B3N8H6m/BTYsef57/O7p0GG2vw0ycZ0T5WMdapiukykVpIcQvcPgwOr6v4Zqm248WrdHNgDWN788fRE4PwkL6mqgRmrsmuir4GVVfUgcGuSG+mG1dUzU+Ij6hiv1hOBwwCq6sokm9K90Wq/TkWOZUKv6YnwtF47nA8c20wfC3xjeIOqelVV7VRVi4BT6J7XnfJgmoBxa03yaOBcujWePYO1QfeXyy5JntzUcTTdmnv17sORwHeqeTd3ho1ba3Oq7HPAS/r4vgiMU2tV3VNV21TVouY1upJuzTMdTOPW2jiP7sUmJNmG7mm+W2a0yq6J1PqvwMEASf4A2BQYnNEqJ+584M+bq/b2Be4Zehtg0vp11Yc/j7jC5QnAJcDNzePWzfwOcMYI7Y+jf1frjVsr8GrgQWB1z8+eM1jjC4Gb6L7P9fZm3nvp/rKE7n/us4EfAd8HntLHf/vxar0Y+HnPcTy/rbUOa7ucPl2tN8HjGuBjwA3AOuDoFte6O3AF3Sv5VgOH9rHWs+heffsg3VHSicDJwMk9x/XTzb6s25DXgLcvkiS1jqf1JEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6/x/OVgyXZ8+pB4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "# Create linear regression\n",
    "regressor = Ridge(alpha=1)\n",
    "\n",
    "# Fit/train Ridge\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ElasticNet Regularization\n",
    "\n",
    "The ElasticNet regression combines both L1 and L2.  Both penalties are applied.  The amount of L1 and L2 are governed by the parameters alpha and beta.\n",
    "\n",
    "$ a * L1 + b * L2 $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 3.0450899960775013\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <td>-0.274010</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>-0.007303</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>horsepower</th>\n",
       "      <td>-0.003231</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>displacement</th>\n",
       "      <td>0.016194</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>acceleration</th>\n",
       "      <td>0.132348</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>0.777482</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <td>0.782781</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  coef  positive\n",
       "cylinders    -0.274010     False\n",
       "weight       -0.007303     False\n",
       "horsepower   -0.003231     False\n",
       "displacement  0.016194      True\n",
       "acceleration  0.132348      True\n",
       "year          0.777482      True\n",
       "origin        0.782781      True"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -18.389355690429767\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAD4CAYAAABIQCkOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYRElEQVR4nO3deZhldX3n8feHhojYyCKNEgK2IkgQI9hXFJVFIYSoUQwo4DJsoUcncR3Io4/LuOGa0XGcKLaMaXTiEhSQMUY2aRqRRqqbXgAFlCXRMFoEQkTDInznj3vaXMrqrlvdVXVP3Xq/nqeee+45v/M739+tevrTv3PPPTdVhSRJbbLFoAuQJGksw0mS1DqGkySpdQwnSVLrGE6SpNbZctAFDIuddtqpFi5cOOgyJGnWWLly5Z1VtWC8bYbTFFm4cCEjIyODLkOSZo0kt29om6f1JEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXW8lFyzWjLoCqS5bbq+2MKZkySpdQwnSVLrGE6SpNYxnCRJrTP04ZTkW0m2n6DN+5IcPlM1SZI2bmiv1ksSIFX1oonaVtW7Z6AkSVKfZvXMKclbk1zX/Lw5ycIkP0jyaWAVsFuS25Ls1LR/V5IfJrk4yZeTnNasX5rkmGb5tiTvTbIqybokew9uhJI0N83acEqyCDgJeDbwHOBUYAfgqcAXqmr/qrq9p30HOBrYH/hToLOR7u+sqmcCnwFO20gNi5OMJBkZHR3d3CFJkhqzNpyA5wPnVdUvq+pe4FzgIOD2qlqxgfbfqKp/r6pfAP93I32f2zyuBBZuqFFVLamqTlV1FiwY9/uyJEmbYDaH04buDfDLSbYfz/3N40MM8ftyktRWszmclgNHJdkmyWOAlwNXbKT9d4E/SbJ1kvnAi2eiSEnS5M3aWUFVrUqyFPh+s+os4O6NtL8myQXAGuB2YAS4Z7rrlCRNXmq67trXQknmV9W9SbahO/NaXFWrpqLvTqdTIyMjU9GVJsEbv0qDtTkRkmRlVY17cdqsnTltoiVJ9gG2Bs6eqmCSJE2tORVOVfWqQdcgSZrYnAonDZ85dFZamlNm89V6kqQhZThJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGkySpdfzKDM1qfhOuNDjT+ZU1zpwkSa1jOEmSWsdwkiS1juEkSWodw0mS1DqGU5+SzBt0DZI0VwxlOCV5f5I39Tw/I8kbk5ye5Joka5O8t2f7+UlWJrk+yeKe9fcmeV+Sq4EDZ3gYkjRnDWU4Af8bOAEgyRbAccDPgD2BA4D9gEVJDm7an1xVi4AO8MYkj2vWPwa4rqqeXVXfHXuQJIuTjCQZGR0dnd4RSdIcMpThVFW3Af+SZH/gCOBa4Fk9y6uAvemGFXQDaQ2wAtitZ/1DwNc3cpwlVdWpqs6CBQumYyiSNCcN8x0izgJOBJ4AfB44DPhQVX22t1GSQ4HDgQOr6ldJlgFbN5vvq6qHZqpgSVLXUM6cGucBR9KdMV3Y/JycZD5Akl2T7AxsB9zdBNPewHMGVbAkqWtoZ05V9UCSy4B/bWY/FyX5feCqdG/Idi/wGuDbwOuSrAVupHtqT5I0QEMbTs2FEM8BXrF+XVV9EvjkOM3/eLw+qmr+9FQnSdqYoTytl2Qf4EfApVV186DrkSRNzlDOnKrqBuDJg65DkrRphjKcNHdM5/fJSBqcoTytJ0ma3QwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY5fmaFZLRl0BePzqzykzePMSZLUOoaTJKl1DCdJUusYTpKk1mllOCVZlqQzRX0dlWSfnufvS3L4VPQtSZoerQynyUoybyObjwJ+E05V9e6qumT6q5IkbarNCqck5ydZmeT6JIubdUcmWZVkTZJLm3Xzk/xNknVJ1iY5ull/RJKrmvbnJJk/zjHGbZPktiTvTvJd4BVJTk1yTXPcryfZJslzgZcCH0uyOskeSZYmOabp47Ak1zZ1fT7Jo3r6fm9zzHVJ9t6c10mSNDmbO3M6uaoWAR3gjUkeD3wOOLqqngG8omn3LuCeqnp6Vf0B8J0kOwHvBA6vqmcCI8Bbezvvo819VfX8qvoKcG5VPas57g+AU6rqe8AFwOlVtV9V/bin762BpcCxVfV0up/5en1P33c2x/wMcNp4g0+yOMlIkpHR0dHJvXKSpA3a3HB6Y5I1wApgN2AxsLyqbgWoqruadocDf71+p6q6G3gO3dNtVyZZDZwAPHFM/xO1+WrP8r5JrkiyDng18LQJan8qcGtV3dQ8Pxs4uGf7uc3jSmDheB1U1ZKq6lRVZ8GCBRMcTpLUr02+Q0SSQ+mGzoFV9asky4A1dP/R/63mwNjPzAe4uKqO39hhJmjzy57lpcBRVbUmyYnAoRMNYYLt9zePD+GdNCRpRm3OzGk74O4mmPamO8t5FHBIkicBJNmxaXsR8Bfrd0yyA93Z1vOSPKVZt02SvcYco582620L3JFkK7ozp/V+0Wwb64fAwvV9A68FLu9j3JKkabY54fRtYMska4H30w2SUbqn9s5tTvetP+32AWCHJNc1619QVaPAicCXmz5WAI+48KCfNj3eBVwNXEw3eNb7CnB6c+HDHj193wecBJzTnAp8GDhzU14ISdLUSnmHyinR6XRqZGRk0GXMOd74VZq9kqysqnE/0zoUn3OSJA0Xw0mS1DpehaZZzdNn0nBy5iRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTWMZwkSa3jV2ZoVhv7Tbh+hYY0HJw5SZJax3CSJLWO4SRJah3DSZLUOpO+ICLJe4B7gccCy6vqkknufyhwWlW9ZLLHnmlJjgJuqqobBl2LJM0lmzxzqqp3TzaYZqGjgH0GXYQkzTV9hVOSdyS5McklwFObdUuTHNMsfzjJDUnWJvmrnu1nJrkiyU1JfmumlOSAJN9Lcm3zuL7veUn+Ksm6ps83NOsXJbk8ycokFybZpVm/LMknkixP8oMkz0pybpKbk3yg53ivSfL9JKuTfDbJvGb9vUnOSLImyYokj0/yXOClwMea9ntsxussSZqECU/rJVkEHAfs37RfBazs2b4j8HJg76qqJNv37L4QOATYA7gsyVPGdP9D4OCq+nWSw4EPAkcDi4EnAfs323ZMshXwKeBlVTWa5FjgDODkpq8HqurgJG8CvgEsAu4CfpzkE8DOwLHA86rqwSSfBl4NfAF4DLCiqt6R5KPAqVX1gSQXAN+sqq9t4LVZ3NTK7rvvPtFLKUnqUz/vOR0EnFdVvwJo/sHu9W/AfcBZSf4e+GbPtr+rqoeBm5PcAuw9Zt/tgLOT7AkUsFWz/nDgzKr6NUBV3ZVkX2Bf4OJ0P3k5D7ijp6/1da0Drq+qO5p6bwF2A55PN7CuafZ/NPDzZp8HeupeCfxhH68LVbUEWALQ6XT8+KckTZF+L4jY4D+8zczmAOAwujOsvwBeuIH9xj5/P3BZVb08yUJgWbM+47QN3dA5cAOl3N88PtyzvP75ls3+Z1fV28fZ98Gq39xb4CG8c4YkDVQ/7zktB16e5NFJtgX+pHdjkvnAdlX1LeDNwH49m1+RZIvm/ZonAzeO6Xs74KfN8ok96y8CXpdky+YYOzb7LkhyYLNuqyRP66P+9S4Fjkmy8/o+kzxxgn1+AWw7iWNIkqbAhOFUVauArwKrga8DV4xpsi3wzSRrgcuBt/Rsu7FZ9w/A66rqvjH7fhT4UJIr6Z6mW+8s4B+BtUnWAK+qqgeAY4CPNOtWA8/ta5TdcdwAvBO4qKn1YmCXCXb7CnB6c8GGF0RI0gxJTdOdMpMsZSMXEwybTqdTIyMjgy5jzvHGr9LslWRlVXXG2+YdIiRJrTNtb/xX1YnT1bckabh5VZpmNU/jScPJ03qSpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpNmr7FfgytpaBhOkqTWMZwkSa1jOEmSWsdwkiS1zpSEU5KFSa6bir4kSRr4zCnJloOuoR+zpU5JGgZTGU7zknwuyfVJLkry6CT7JVmRZG2S85LsAJBkWZIPJrkceFOSVyS5LsmaJMubNvOSfCzJNc3+/7lZf2iS5U1/NyQ5M8kWzbbjk6xr+vpIs+6VST7eLL8pyS3N8h5JvtssL0pyeZKVSS5Msst4dU7hayVJ2oipnA3sCRxfVacm+TvgaOAvgTdU1eVJ3gf8N+DNTfvtq+oQgCTrgD+qqp8m2b7ZfgpwT1U9K8mjgCuTXNRsOwDYB7gd+Dbwp0m+B3wEWATcDVyU5ChgOXB6s99BwL8k2RV4PnBFkq2ATwEvq6rRJMcCZwAnj61zrCSLgcUAu++++6a+bpKkMaYynG6tqtXN8kpgD7r/sF/erDsbOKen/Vd7lq8Eljahdm6z7gjgD5Ic0zzfjm4APgB8v6rWz4C+TDdoHgSWVdVos/5vgYOr6vwk85NsC+wGfAk4mG5QnQs8FdgXuDjdD3XOA+7YQJ2PUFVLgCUAnU6nNv7ySJL6NZXhdH/P8kPA9htq2Pjl+oWqel2SZwMvBlYn2Q8I3VnXhb07JTkUGBsE1bTfkKuAk4AbgSvozooOBP4rsDtwfVUdOFGdkqSZMZ0XRNwD3J3koOb5a4HLx2uYZI+qurqq3g3cSXeGcyHw+ua0G0n2SvKYZpcDkjypea/pWOC7wNXAIUl2SjIPOL7neMuB05rHa4EXAPdX1T10A2tBkgOb42yV5GlT9zJIkiZruq9AOwE4M8k2wC10Zy/j+ViSPenOfi4F1gBrgYXAqnTPt40CRzXtrwI+DDydbuCcV1UPJ3k7cFnTz7eq6htN+yvoBt7yqnooyT8BPwSoqgeaU4f/M8l2dF+T/wFcP0WvgSRpklI1u94qaU7rnVZVLxl0Lb06nU6NjIwMuoy5JYFZ9vcr6T8kWVlVnfG2DfxzTpIkjTXrPlhaVcuAZQMuQ5I0jZw5afbylJ40tAwnSVLrGE6SpNYxnCRJrWM4SZJax3CSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJs1PS/ZE0lAwnSVLrGE6SpNYxnCRJrWM4SZJaZ6jDKclZSfaZoM3SJMeMs35hkldNX3WSpA0Z6nCqqj+rqhs2cfeFgOEkSQMwK8IpyV8meWOz/Ikk32mWD0vyf5IckeSqJKuSnJNkfrN9WZJOs3xKkpuadZ9L8r96DnFwku8luaVnFvVh4KAkq5O8ZQaHK0lz3qwIJ2A5cFCz3AHmJ9kKeD6wDngncHhVPRMYAd7au3OS3wXeBTwH+ENg7zH979L09RK6oQTwNuCKqtqvqj4xXlFJFicZSTIyOjq6mUOUJK03W8JpJbAoybbA/cBVdEPqIODfgX2AK5OsBk4Anjhm/wOAy6vqrqp6EDhnzPbzq+rh5hTg4/stqqqWVFWnqjoLFizYpIFJkn7bloMuoB9V9WCS24CTgO8Ba4EXAHsAtwIXV9XxG+liolsJ3D+JtpKkaTZbZk7QPbV3WvN4BfA6YDWwAnhekqcAJNkmyV5j9v0+cEiSHZJsCRzdx/F+AWw7VcVLkvo3m8LpCrrvDV1VVT8D7qP7ntAocCLw5SRr6YbVI95TqqqfAh8ErgYuAW4A7pngeGuBXydZ4wURkjSzUlWDrmFGJJlfVfc2M6fzgM9X1XlT1X+n06mRkZGp6k4TWX/T1zny9ysNoyQrq6oz3rbZNHPaXO9pLpi4ju77VOcPuB5J0gbMigsipkJVnTboGiRJ/Zkz4aQh4+k8aajNpdN6kqRZwnCSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1jGcJEmtYzhJklrHcJIktY7hJElqHcNJktQ6hpMkqXUMJ0lS6xhOkqTW8fuc2mD9V45r8vxeJ2koOXOSJLWO4SRJah3DSZLUOrMinJIsTXJMs3xWkn0muf+901OZJGk6zLoLIqrqz6az/yQBUlUPT+dxJEkbNtCZU5L/lGRtkjVJzktya5Ktmm2PTXLb+uc9+yxL0mmW701yRrP/iiSPb9Y/KclVSa5J8v4x+5/erF+b5L3NuoVJfpDk08AqYLdmtnZdknVJ3jITr4ckqWtg4ZTkacA7gBdW1TOAU4BlwIubJscBX6+qBzfSzWOAFc3+y4FTm/WfBD5TVc8C/l/PMY8A9gQOAPYDFiU5uNn8VOALVbU/sBOwa1XtW1VPB/5mA2NYnGQkycjo6OjkXgBJ0gYNcub0QuBrVXUnQFXdBZwFnNRsP4kNhEKPB4BvNssrgYXN8vOALzfLX+xpf0Tzcy3dGdLedMMK4PaqWtEs3wI8OcmnkhwJ/Nt4B6+qJVXVqarOggULJihVktSvQb7nFOARn6CsqiubU2yHAPOq6roJ+niw6jefwnyIR45nvE9nBvhQVX32ESuThcAve+q4O8kzgD8C/hx4JXDyhCOSJE2JQc6cLgVemeRxAEl2bNZ/ge6sZ6JZ08ZcSfe0IMCre9ZfCJycZH5zzF2T7Dx25yQ7AVtU1deBdwHP3IxaJEmTNLBwqqrrgTOAy5OsAT7ebPpbYAf+47TcpngT8OdJrgG26znmRcCXgKuSrAO+Bmw7zv67AsuSrAaWAm/fjFokSZOUatm9yZrPM72sql476Fomo9Pp1MjIyKbt7L31Nl3L/n4l9S/JyqrqjLetVZ9zSvIp4I+BFw26FknS4LQqnKrqDYOuQZI0eK0KpznLU1OS9Aiz4t56kqS5xXCSJLWO4SRJah3DSZLUOoaTJKl1DCdJUusYTpKk1mnd7YtmqySjwO2DrqPHTsCdgy5iBsyFcc6FMYLjHCb9jvGJVTXu9w0ZTkMqyciG7lk1TObCOOfCGMFxDpOpGKOn9SRJrWM4SZJax3AaXksGXcAMmQvjnAtjBMc5TDZ7jL7nJElqHWdOkqTWMZwkSa1jOA2JJDsmuTjJzc3jDuO02S/JVUmuT7I2ybGDqHWykhyZ5MYkP0rytnG2PyrJV5vtVydZOPNVbr4+xvnWJDc0v7tLkzxxEHVuronG2dPumCSVZNZddt3PGJO8svl9Xp/kSzNd41To42929ySXJbm2+bvt/1vOq8qfIfgBPgq8rVl+G/CRcdrsBezZLP8ucAew/aBrn2Bc84AfA08GfgdYA+wzps1/Ac5slo8DvjrouqdpnC8AtmmWXz+s42zabQssB1YAnUHXPQ2/yz2Ba4Edmuc7D7ruaRrnEuD1zfI+wG399u/MaXi8DDi7WT4bOGpsg6q6qapubpb/Gfg5MO6ns1vkAOBHVXVLVT0AfIXuWHv1jv1rwGFJMoM1ToUJx1lVl1XVr5qnK4Dfm+Eap0I/v0+A99P9D9d9M1ncFOlnjKcCf11VdwNU1c9nuMap0M84C3hss7wd8M/9dm44DY/HV9UdAM3jzhtrnOQAuv/b+fEM1LY5dgX+qef5T5p147apql8D9wCPm5Hqpk4/4+x1CvAP01rR9JhwnEn2B3arqm/OZGFTqJ/f5V7AXkmuTLIiyZEzVt3U6Wec7wFek+QnwLeAN/Tb+ZabW51mTpJLgCeMs+kdk+xnF+CLwAlV9fBU1DaNxpsBjf38Qz9t2q7vMSR5DdABDpnWiqbHRseZZAvgE8CJM1XQNOjnd7kl3VN7h9KdAV+RZN+q+tdprm0q9TPO44GlVfXfkxwIfLEZ54T/7hhOs0hVHb6hbUl+lmSXqrqjCZ9xTxMkeSzw98A7q2rFNJU6lX4C7Nbz/Pf47VMD69v8JMmWdE8f3DUz5U2ZfsZJksPp/mfkkKq6f4Zqm0oTjXNbYF9gWXNm9gnABUleWlUjM1bl5un3b3ZFVT0I3JrkRrphdc3MlDgl+hnnKcCRAFV1VZKt6d4UdsLTmJ7WGx4XACc0yycA3xjbIMnvAOcBX6iqc2awts1xDbBnkic19R9Hd6y9esd+DPCdat6BnUUmHGdzuuuzwEtn6XsUMME4q+qeqtqpqhZW1UK6763NpmCC/v5mz6d7gQtJdqJ7mu+WGa1y8/Uzzn8EDgNI8vvA1sBoX70P+ooPf6bsypnHAZcCNzePOzbrO8BZzfJrgAeB1T0/+w269j7G9iLgJrrvj72jWfc+uv9o0fzBnwP8CPg+8ORB1zxN47wE+FnP7+6CQdc8HeMc03YZs+xqvT5/lwE+DtwArAOOG3TN0zTOfYAr6V7Jtxo4ot++vX2RJKl1PK0nSWodw0mS1DqGkySpdQwnSVLrGE6SpNYxnCRJrWM4SZJa5/8D7xgtoq1AiKAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sklearn\n",
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "# Create linear regression\n",
    "regressor = ElasticNet(alpha=0.1, l1_ratio=0.1)\n",
    "\n",
    "# Fit/train LASSO\n",
    "regressor.fit(x_train,y_train)\n",
    "# Predict\n",
    "pred = regressor.predict(x_test)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(f\"Final score (RMSE): {score}\")\n",
    "\n",
    "report_coef(\n",
    "  names,\n",
    "  regressor.coef_,\n",
    "  regressor.intercept_)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}