Amazon_Grid_search_Random_search_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yI8hHeMFzjQp"
   },
   "source": [
    "  APPLYING LOGISTIC REGRESSION FOR AMAZON DATASET\n",
    "  ====================================\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7ppnhwuLmXQ6"
   },
   "source": [
    "To apply Logistic regression and do hyperparameter tunning by Grid-search and Random-search for both L1 an L2  regularization.\n",
    "-----------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O_Joi4dRzTjP"
   },
   "outputs": [],
   "source": [
    "!pip install -U -q Pydrive\n",
    "from pydrive.auth import GoogleAuth\n",
    "from pydrive.drive import GoogleDrive\n",
    "from google.colab import auth\n",
    "from oauth2client.client import GoogleCredentials\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plot\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import linear_model, datasets\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ettUgf1OyJYO"
   },
   "outputs": [],
   "source": [
    "\n",
    "# authenticate\n",
    "auth.authenticate_user()\n",
    "gauth= GoogleAuth()\n",
    "gauth.credentials= GoogleCredentials.get_application_default()\n",
    "drive=GoogleDrive(gauth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BhEEnuEUyN95"
   },
   "outputs": [],
   "source": [
    "train_downloaded = drive.CreateFile({'id': '1LgNo7LS202N7BuUaQQt4tVwchHRJm8S9'})\n",
    "train_downloaded.GetContentFile('final_clean.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 369
    },
    "colab_type": "code",
    "id": "a6Uyz4sixmgZ",
    "outputId": "b26e8122-9a40-48e3-cc5b-7d7603593c51"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>ProfileName</th>\n",
       "      <th>HelpfulnessNumerator</th>\n",
       "      <th>HelpfulnessDenominator</th>\n",
       "      <th>Score</th>\n",
       "      <th>Time</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "      <th>cleanedText</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>150524</td>\n",
       "      <td>0006641040</td>\n",
       "      <td>ACITT7DI6IDDL</td>\n",
       "      <td>shari zychinski</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>939340800</td>\n",
       "      <td>EVERY book is educational</td>\n",
       "      <td>this witty little book makes my son laugh at l...</td>\n",
       "      <td>b'witti littl book make son laugh loud recit c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>150506</td>\n",
       "      <td>0006641040</td>\n",
       "      <td>A2IW4PEEKO2R0U</td>\n",
       "      <td>Tracy</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1194739200</td>\n",
       "      <td>Love the book, miss the hard cover version</td>\n",
       "      <td>I grew up reading these Sendak books, and watc...</td>\n",
       "      <td>b'grew read sendak book watch realli rosi movi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Id   ProductId          UserId      ProfileName  HelpfulnessNumerator  \\\n",
       "0  150524  0006641040   ACITT7DI6IDDL  shari zychinski                     0   \n",
       "1  150506  0006641040  A2IW4PEEKO2R0U            Tracy                     1   \n",
       "\n",
       "   HelpfulnessDenominator  Score        Time  \\\n",
       "0                       0      1   939340800   \n",
       "1                       1      1  1194739200   \n",
       "\n",
       "                                      Summary  \\\n",
       "0                   EVERY book is educational   \n",
       "1  Love the book, miss the hard cover version   \n",
       "\n",
       "                                                Text  \\\n",
       "0  this witty little book makes my son laugh at l...   \n",
       "1  I grew up reading these Sendak books, and watc...   \n",
       "\n",
       "                                         cleanedText  \n",
       "0  b'witti littl book make son laugh loud recit c...  \n",
       "1  b'grew read sendak book watch realli rosi movi...  "
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.read_csv('final_clean.csv')\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "OPVA1389xmgh",
    "outputId": "6431917a-56b5-40e5-d63a-4dc439cb9ef8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3000, 11)"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1=df.sample(3000, random_state=23)\n",
    "data1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 111
    },
    "colab_type": "code",
    "id": "nCcrn9nB3JtW",
    "outputId": "61f4714c-d735-4373-f11b-eaee89d15ce9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cleanedText</th>\n",
       "      <th>Text</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10509</th>\n",
       "      <td>b'let say impress senseo machin never get coff...</td>\n",
       "      <td>Let me just say that I am so impressed with th...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>279417</th>\n",
       "      <td>b'best snappl flavor ever hard find though how...</td>\n",
       "      <td>This is the best Snapple flavor I have ever ha...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              cleanedText  \\\n",
       "10509   b'let say impress senseo machin never get coff...   \n",
       "279417  b'best snappl flavor ever hard find though how...   \n",
       "\n",
       "                                                     Text  Score  \n",
       "10509   Let me just say that I am so impressed with th...      1  \n",
       "279417  This is the best Snapple flavor I have ever ha...      1  "
      ]
     },
     "execution_count": 43,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data1.filter(['cleanedText','Text','Score'],axis=1)\n",
    "data.head(2)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zNi1FeRnxmgl"
   },
   "outputs": [],
   "source": [
    "\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from collections import Counter\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn import cross_validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A7LG1IcFCgXW"
   },
   "outputs": [],
   "source": [
    "X = data['cleanedText']\n",
    "y = data['Score']\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AX_BcCF10LBD"
   },
   "source": [
    "SPLITTING DATA IN TO TRAIN AND TEST AND FEATURIZE TEXT UDING TF-IDF\n",
    "--------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qgwSIYVInjP-"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "_WnYJHViBxLN",
    "outputId": "c64698a7-0f51-478a-d62d-9b88b6dd8ad8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2250,)"
      ]
     },
     "execution_count": 46,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_train, x_test, y_train, y_test = train_test_split(X,y)\n",
    "\n",
    "y_train.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A9FCFIwexmgt"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tfidf_vect = TfidfVectorizer(\n",
    "    analyzer='word',  # features made of words\n",
    "    ngram_range=(1,2))\n",
    "tf_idf = tfidf_vect.fit_transform(x_train)\n",
    "tfidf_xtest = tfidf_vect.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vFO_tJ4GzfV9"
   },
   "outputs": [],
   "source": [
    "w = tf_idf.todense()\n",
    "w2=tfidf_xtest.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mqJbvL-20h-o"
   },
   "source": [
    "OBTAINING OPTIMAL PENALITY(L1,L2) AND LAMBDA USING GRID-SEARCH CV\n",
    "-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6N6qfXsHxmhB"
   },
   "outputs": [],
   "source": [
    "logistic = linear_model.LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "39YRvukpxmhF"
   },
   "outputs": [],
   "source": [
    "# Create regularization penalty space\n",
    "penality=('l1','l2')\n",
    "\n",
    "# Create regularization hyperparameter space\n",
    "C = np.logspace(0, 4, 10)\n",
    "\n",
    "# Create hyperparameter options\n",
    "hyperparameters = dict(C=C, penalty=penality)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FJ3rY9ah6dY3"
   },
   "outputs": [],
   "source": [
    "cf = GridSearchCV(logistic, hyperparameters, cv=5, verbose=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 4193
    },
    "colab_type": "code",
    "id": "01LosKKQxmhc",
    "outputId": "6977ca75-018f-4747-df12-1c6c7fe194a4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
      "[CV] C=1.0, penalty=l1 ...............................................\n",
      "[CV] ...... C=1.0, penalty=l1, score=0.8492239467849224, total=   1.5s\n",
      "[CV] C=1.0, penalty=l1 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    1.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l1, score=0.8377777777777777, total=   0.9s\n",
      "[CV] C=1.0, penalty=l1 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    2.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l1, score=0.8311111111111111, total=   0.9s\n",
      "[CV] C=1.0, penalty=l1 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    4.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l1, score=0.8422222222222222, total=   0.9s\n",
      "[CV] C=1.0, penalty=l1 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    5.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l1, score=0.8396436525612472, total=   0.9s\n",
      "[CV] C=1.0, penalty=l2 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    6.3s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ....... C=1.0, penalty=l2, score=0.835920177383592, total=   0.9s\n",
      "[CV] C=1.0, penalty=l2 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    7.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l2, score=0.8355555555555556, total=   0.9s\n",
      "[CV] C=1.0, penalty=l2 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    8.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l2, score=0.8355555555555556, total=   0.9s\n",
      "[CV] C=1.0, penalty=l2 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:    9.6s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l2, score=0.8355555555555556, total=   0.9s\n",
      "[CV] C=1.0, penalty=l2 ...............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:   10.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=1.0, penalty=l2, score=0.8374164810690423, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l1 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   11.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l1, score=0.8869179600886918, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l1 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  11 out of  11 | elapsed:   13.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l1, score=0.8733333333333333, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l1 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  12 out of  12 | elapsed:   14.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l1, score=0.8688888888888889, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l1 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  13 out of  13 | elapsed:   15.3s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l1, score=0.8733333333333333, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l1 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  14 out of  14 | elapsed:   16.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l1, score=0.8619153674832962, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l2 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  15 out of  15 | elapsed:   17.6s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l2, score=0.8403547671840355, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l2 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  16 out of  16 | elapsed:   18.8s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=2.7825594022071245, penalty=l2, score=0.84, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l2 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  17 out of  17 | elapsed:   20.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l2, score=0.8355555555555556, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l2 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  18 out of  18 | elapsed:   21.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l2, score=0.8355555555555556, total=   0.9s\n",
      "[CV] C=2.7825594022071245, penalty=l2 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  19 out of  19 | elapsed:   22.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.7825594022071245, penalty=l2, score=0.8374164810690423, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l1 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed:   23.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l1, score=0.9135254988913526, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l1 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  21 out of  21 | elapsed:   24.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l1, score=0.8977777777777778, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l1 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  22 out of  22 | elapsed:   25.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l1, score=0.9044444444444445, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l1 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  23 out of  23 | elapsed:   26.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l1, score=0.8911111111111111, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l1 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  24 out of  24 | elapsed:   28.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l1, score=0.8663697104677061, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l2 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  25 out of  25 | elapsed:   29.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l2, score=0.8470066518847007, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l2 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  26 out of  26 | elapsed:   30.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l2, score=0.8444444444444444, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l2 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  27 out of  27 | elapsed:   31.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l2, score=0.8444444444444444, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l2 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  28 out of  28 | elapsed:   32.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l2, score=0.8422222222222222, total=   0.9s\n",
      "[CV] C=7.742636826811269, penalty=l2 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  29 out of  29 | elapsed:   33.8s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=7.742636826811269, penalty=l2, score=0.844097995545657, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l1 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l1, score=0.917960088691796, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l1 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l1, score=0.8888888888888888, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l1 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l1, score=0.8955555555555555, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l1 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l1, score=0.8777777777777778, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l1 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l1, score=0.8641425389755011, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l2 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l2, score=0.8536585365853658, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l2 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l2, score=0.8511111111111112, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l2 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l2, score=0.8444444444444444, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l2 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l2, score=0.8488888888888889, total=   0.9s\n",
      "[CV] C=21.544346900318832, penalty=l2 ................................\n",
      "[CV]  C=21.544346900318832, penalty=l2, score=0.844097995545657, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l1 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l1, score=0.9113082039911308, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l1 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l1, score=0.8911111111111111, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l1 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l1, score=0.8888888888888888, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l1 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l1, score=0.8755555555555555, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l1 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l1, score=0.8663697104677061, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l2 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l2, score=0.8558758314855875, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l2 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l2, score=0.8555555555555555, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l2 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l2, score=0.8488888888888889, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l2 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l2, score=0.8466666666666667, total=   0.9s\n",
      "[CV] C=59.94842503189409, penalty=l2 .................................\n",
      "[CV]  C=59.94842503189409, penalty=l2, score=0.8463251670378619, total=   0.9s\n",
      "[CV] C=166.81005372000593, penalty=l1 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l1, score=0.9090909090909091, total=   1.0s\n",
      "[CV] C=166.81005372000593, penalty=l1 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l1, score=0.8933333333333333, total=   1.0s\n",
      "[CV] C=166.81005372000593, penalty=l1 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l1, score=0.8933333333333333, total=   1.0s\n",
      "[CV] C=166.81005372000593, penalty=l1 ................................\n",
      "[CV] ..... C=166.81005372000593, penalty=l1, score=0.88, total=   1.0s\n",
      "[CV] C=166.81005372000593, penalty=l1 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l1, score=0.8685968819599109, total=   1.0s\n",
      "[CV] C=166.81005372000593, penalty=l2 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l2, score=0.8580931263858093, total=   0.9s\n",
      "[CV] C=166.81005372000593, penalty=l2 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l2, score=0.8622222222222222, total=   0.9s\n",
      "[CV] C=166.81005372000593, penalty=l2 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l2, score=0.8511111111111112, total=   0.9s\n",
      "[CV] C=166.81005372000593, penalty=l2 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l2, score=0.8488888888888889, total=   0.9s\n",
      "[CV] C=166.81005372000593, penalty=l2 ................................\n",
      "[CV]  C=166.81005372000593, penalty=l2, score=0.8463251670378619, total=   0.9s\n",
      "[CV] C=464.15888336127773, penalty=l1 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l1, score=0.9068736141906873, total=   1.0s\n",
      "[CV] C=464.15888336127773, penalty=l1 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l1, score=0.8911111111111111, total=   1.0s\n",
      "[CV] C=464.15888336127773, penalty=l1 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l1, score=0.8822222222222222, total=   1.0s\n",
      "[CV] C=464.15888336127773, penalty=l1 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l1, score=0.8777777777777778, total=   0.9s\n",
      "[CV] C=464.15888336127773, penalty=l1 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l1, score=0.8708240534521158, total=   1.0s\n",
      "[CV] C=464.15888336127773, penalty=l2 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l2, score=0.8647450110864745, total=   0.9s\n",
      "[CV] C=464.15888336127773, penalty=l2 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l2, score=0.8644444444444445, total=   0.9s\n",
      "[CV] C=464.15888336127773, penalty=l2 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l2, score=0.8555555555555555, total=   0.9s\n",
      "[CV] C=464.15888336127773, penalty=l2 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l2, score=0.8488888888888889, total=   0.9s\n",
      "[CV] C=464.15888336127773, penalty=l2 ................................\n",
      "[CV]  C=464.15888336127773, penalty=l2, score=0.8463251670378619, total=   0.9s\n",
      "[CV] C=1291.5496650148827, penalty=l1 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l1, score=0.9002217294900222, total=   1.0s\n",
      "[CV] C=1291.5496650148827, penalty=l1 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l1, score=0.8911111111111111, total=   1.0s\n",
      "[CV] C=1291.5496650148827, penalty=l1 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l1, score=0.8844444444444445, total=   1.0s\n",
      "[CV] C=1291.5496650148827, penalty=l1 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l1, score=0.8777777777777778, total=   1.0s\n",
      "[CV] C=1291.5496650148827, penalty=l1 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l1, score=0.8708240534521158, total=   1.0s\n",
      "[CV] C=1291.5496650148827, penalty=l2 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l2, score=0.8647450110864745, total=   0.9s\n",
      "[CV] C=1291.5496650148827, penalty=l2 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l2, score=0.8688888888888889, total=   0.9s\n",
      "[CV] C=1291.5496650148827, penalty=l2 ................................\n",
      "[CV] ..... C=1291.5496650148827, penalty=l2, score=0.86, total=   0.9s\n",
      "[CV] C=1291.5496650148827, penalty=l2 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l2, score=0.8511111111111112, total=   0.9s\n",
      "[CV] C=1291.5496650148827, penalty=l2 ................................\n",
      "[CV]  C=1291.5496650148827, penalty=l2, score=0.8485523385300668, total=   0.9s\n",
      "[CV] C=3593.813663804626, penalty=l1 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l1, score=0.9046563192904656, total=   1.0s\n",
      "[CV] C=3593.813663804626, penalty=l1 .................................\n",
      "[CV] ....... C=3593.813663804626, penalty=l1, score=0.9, total=   1.0s\n",
      "[CV] C=3593.813663804626, penalty=l1 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l1, score=0.8866666666666667, total=   1.0s\n",
      "[CV] C=3593.813663804626, penalty=l1 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l1, score=0.8688888888888889, total=   1.0s\n",
      "[CV] C=3593.813663804626, penalty=l1 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l1, score=0.8775055679287305, total=   1.0s\n",
      "[CV] C=3593.813663804626, penalty=l2 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l2, score=0.8625277161862528, total=   0.9s\n",
      "[CV] C=3593.813663804626, penalty=l2 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l2, score=0.8688888888888889, total=   0.9s\n",
      "[CV] C=3593.813663804626, penalty=l2 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l2, score=0.8644444444444445, total=   0.9s\n",
      "[CV] C=3593.813663804626, penalty=l2 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l2, score=0.8511111111111112, total=   0.9s\n",
      "[CV] C=3593.813663804626, penalty=l2 .................................\n",
      "[CV]  C=3593.813663804626, penalty=l2, score=0.8507795100222717, total=   0.9s\n",
      "[CV] C=10000.0, penalty=l1 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l1, score=0.9090909090909091, total=   1.0s\n",
      "[CV] C=10000.0, penalty=l1 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l1, score=0.8933333333333333, total=   1.0s\n",
      "[CV] C=10000.0, penalty=l1 ...........................................\n",
      "[CV] ................ C=10000.0, penalty=l1, score=0.88, total=   1.0s\n",
      "[CV] C=10000.0, penalty=l1 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l1, score=0.8666666666666667, total=   1.0s\n",
      "[CV] C=10000.0, penalty=l1 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l1, score=0.8619153674832962, total=   1.0s\n",
      "[CV] C=10000.0, penalty=l2 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l2, score=0.8625277161862528, total=   0.9s\n",
      "[CV] C=10000.0, penalty=l2 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l2, score=0.8688888888888889, total=   0.9s\n",
      "[CV] C=10000.0, penalty=l2 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l2, score=0.8644444444444445, total=   0.9s\n",
      "[CV] C=10000.0, penalty=l2 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l2, score=0.8511111111111112, total=   0.9s\n",
      "[CV] C=10000.0, penalty=l2 ...........................................\n",
      "[CV] .. C=10000.0, penalty=l2, score=0.8507795100222717, total=   0.9s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:  2.0min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:01:58.626006\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "startTime = datetime.now()\n",
    "\n",
    "#lets run GridsearchCV on our data\n",
    "bestmodel=cf.fit(w, y_train)\n",
    "\n",
    "print(datetime.now() - startTime)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "colab_type": "code",
    "id": "nUVCXKup8dNr",
    "outputId": "795a1037-608e-4a8f-8456-4fbe0e8dcb59"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Penalty: l1\n",
      "Best C: 7.742636826811269\n"
     ]
    }
   ],
   "source": [
    "print('Best Penalty:', bestmodel.best_estimator_.get_params()['penalty'])\n",
    "print('Best C:', bestmodel.best_estimator_.get_params()['C'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nGoMQPoCxmhm"
   },
   "outputs": [],
   "source": [
    "y_true, y_pred = y_test,cf.predict(w2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "xYmhE-_rxmhw",
    "outputId": "333b37b9-d27f-41cc-8d6d-0eeeeecb4835"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.62      0.53      0.57        99\n",
      "          1       0.93      0.95      0.94       651\n",
      "\n",
      "avg / total       0.89      0.89      0.89       750\n",
      "\n",
      "(0.8946666666666667, 0.8946666666666667, 0.8946666666666667, None)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from sklearn.metrics import precision_recall_fscore_support, classification_report\n",
    "\n",
    "print(classification_report(y_test, y_pred))\n",
    "print(precision_recall_fscore_support(y_test, y_pred, average='micro'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oj_YSyrg1A2B"
   },
   "source": [
    "OBTAINING OPTIMAL PENALITY(L1,L2) AND LAMBDA USING RANDOM-SEARCH CV\n",
    "-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KrYOo7yQxmhz"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "clf  =RandomizedSearchCV(logistic, hyperparameters, cv=5, verbose=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 2015
    },
    "colab_type": "code",
    "id": "Kl9uT0wSxmh3",
    "outputId": "019d8c9d-2967-4ce3-9e9e-30036f150dc5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n",
      "[CV] penalty=l2, C=10000.0 ...........................................\n",
      "[CV] .. penalty=l2, C=10000.0, score=0.8625277161862528, total=   1.0s\n",
      "[CV] penalty=l2, C=10000.0 ...........................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    1.3s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .. penalty=l2, C=10000.0, score=0.8688888888888889, total=   1.0s\n",
      "[CV] penalty=l2, C=10000.0 ...........................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    2.6s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .. penalty=l2, C=10000.0, score=0.8644444444444445, total=   1.0s\n",
      "[CV] penalty=l2, C=10000.0 ...........................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    3.8s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .. penalty=l2, C=10000.0, score=0.8511111111111112, total=   1.0s\n",
      "[CV] penalty=l2, C=10000.0 ...........................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    5.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .. penalty=l2, C=10000.0, score=0.8507795100222717, total=   1.0s\n",
      "[CV] penalty=l2, C=3593.813663804626 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    6.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  penalty=l2, C=3593.813663804626, score=0.8625277161862528, total=   1.0s\n",
      "[CV] penalty=l2, C=3593.813663804626 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    7.6s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  penalty=l2, C=3593.813663804626, score=0.8688888888888889, total=   1.0s\n",
      "[CV] penalty=l2, C=3593.813663804626 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    8.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  penalty=l2, C=3593.813663804626, score=0.8644444444444445, total=   1.0s\n",
      "[CV] penalty=l2, C=3593.813663804626 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:   10.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  penalty=l2, C=3593.813663804626, score=0.8511111111111112, total=   1.0s\n",
      "[CV] penalty=l2, C=3593.813663804626 .................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:   11.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  penalty=l2, C=3593.813663804626, score=0.8507795100222717, total=   0.9s\n",
      "[CV] penalty=l1, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l1, C=1291.5496650148827, score=0.8980044345898004, total=   1.0s\n",
      "[CV] penalty=l1, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l1, C=1291.5496650148827, score=0.8933333333333333, total=   1.0s\n",
      "[CV] penalty=l1, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l1, C=1291.5496650148827, score=0.8888888888888888, total=   1.0s\n",
      "[CV] penalty=l1, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l1, C=1291.5496650148827, score=0.8755555555555555, total=   1.0s\n",
      "[CV] penalty=l1, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l1, C=1291.5496650148827, score=0.8619153674832962, total=   1.0s\n",
      "[CV] penalty=l2, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l2, C=464.15888336127773, score=0.8647450110864745, total=   0.9s\n",
      "[CV] penalty=l2, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l2, C=464.15888336127773, score=0.8644444444444445, total=   0.9s\n",
      "[CV] penalty=l2, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l2, C=464.15888336127773, score=0.8555555555555555, total=   0.9s\n",
      "[CV] penalty=l2, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l2, C=464.15888336127773, score=0.8488888888888889, total=   0.9s\n",
      "[CV] penalty=l2, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l2, C=464.15888336127773, score=0.8463251670378619, total=   0.9s\n",
      "[CV] penalty=l1, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l1, C=464.15888336127773, score=0.9046563192904656, total=   1.0s\n",
      "[CV] penalty=l1, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l1, C=464.15888336127773, score=0.8911111111111111, total=   1.0s\n",
      "[CV] penalty=l1, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l1, C=464.15888336127773, score=0.8844444444444445, total=   1.0s\n",
      "[CV] penalty=l1, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l1, C=464.15888336127773, score=0.8777777777777778, total=   1.0s\n",
      "[CV] penalty=l1, C=464.15888336127773 ................................\n",
      "[CV]  penalty=l1, C=464.15888336127773, score=0.8708240534521158, total=   1.0s\n",
      "[CV] penalty=l1, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l1, C=21.544346900318832, score=0.917960088691796, total=   0.9s\n",
      "[CV] penalty=l1, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l1, C=21.544346900318832, score=0.8888888888888888, total=   0.9s\n",
      "[CV] penalty=l1, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l1, C=21.544346900318832, score=0.8955555555555555, total=   1.0s\n",
      "[CV] penalty=l1, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l1, C=21.544346900318832, score=0.8777777777777778, total=   0.9s\n",
      "[CV] penalty=l1, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l1, C=21.544346900318832, score=0.8641425389755011, total=   0.9s\n",
      "[CV] penalty=l2, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l2, C=7.742636826811269, score=0.8470066518847007, total=   0.9s\n",
      "[CV] penalty=l2, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l2, C=7.742636826811269, score=0.8444444444444444, total=   0.9s\n",
      "[CV] penalty=l2, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l2, C=7.742636826811269, score=0.8444444444444444, total=   0.9s\n",
      "[CV] penalty=l2, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l2, C=7.742636826811269, score=0.8422222222222222, total=   0.9s\n",
      "[CV] penalty=l2, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l2, C=7.742636826811269, score=0.844097995545657, total=   0.9s\n",
      "[CV] penalty=l1, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l1, C=7.742636826811269, score=0.9135254988913526, total=   0.9s\n",
      "[CV] penalty=l1, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l1, C=7.742636826811269, score=0.8977777777777778, total=   0.9s\n",
      "[CV] penalty=l1, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l1, C=7.742636826811269, score=0.9044444444444445, total=   0.9s\n",
      "[CV] penalty=l1, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l1, C=7.742636826811269, score=0.8911111111111111, total=   0.9s\n",
      "[CV] penalty=l1, C=7.742636826811269 .................................\n",
      "[CV]  penalty=l1, C=7.742636826811269, score=0.8663697104677061, total=   0.9s\n",
      "[CV] penalty=l2, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l2, C=1291.5496650148827, score=0.8647450110864745, total=   0.9s\n",
      "[CV] penalty=l2, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l2, C=1291.5496650148827, score=0.8688888888888889, total=   0.9s\n",
      "[CV] penalty=l2, C=1291.5496650148827 ................................\n",
      "[CV] ..... penalty=l2, C=1291.5496650148827, score=0.86, total=   0.9s\n",
      "[CV] penalty=l2, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l2, C=1291.5496650148827, score=0.8511111111111112, total=   0.9s\n",
      "[CV] penalty=l2, C=1291.5496650148827 ................................\n",
      "[CV]  penalty=l2, C=1291.5496650148827, score=0.8485523385300668, total=   0.9s\n",
      "[CV] penalty=l2, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l2, C=21.544346900318832, score=0.8536585365853658, total=   0.9s\n",
      "[CV] penalty=l2, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l2, C=21.544346900318832, score=0.8511111111111112, total=   0.9s\n",
      "[CV] penalty=l2, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l2, C=21.544346900318832, score=0.8444444444444444, total=   0.9s\n",
      "[CV] penalty=l2, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l2, C=21.544346900318832, score=0.8488888888888889, total=   0.9s\n",
      "[CV] penalty=l2, C=21.544346900318832 ................................\n",
      "[CV]  penalty=l2, C=21.544346900318832, score=0.844097995545657, total=   0.9s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  50 out of  50 | elapsed:   59.9s finished\n"
     ]
    }
   ],
   "source": [
    "bestmodel1 = clf.fit(w,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "colab_type": "code",
    "id": "rUzNuIn5xmh7",
    "outputId": "1b7db85a-2ee7-46c1-eb2f-1766559962dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Penalty: l1\n",
      "Best C: 7.742636826811269\n"
     ]
    }
   ],
   "source": [
    "print('Best Penalty:', bestmodel1.best_estimator_.get_params()['penalty'])\n",
    "print('Best C:', bestmodel1.best_estimator_.get_params()['C'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KkwdIcdnxmh_"
   },
   "outputs": [],
   "source": [
    "y_true, y_pred = y_test,clf.predict(w2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "ZbBxDEohxmiB",
    "outputId": "48658798-1117-4e18-a92e-923b6fd11138"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.62      0.53      0.57        99\n",
      "          1       0.93      0.95      0.94       651\n",
      "\n",
      "avg / total       0.89      0.89      0.89       750\n",
      "\n",
      "(0.8946666666666667, 0.8946666666666667, 0.8946666666666667, None)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from sklearn.metrics import precision_recall_fscore_support, classification_report\n",
    "\n",
    "print(classification_report(y_test, y_pred))\n",
    "print(precision_recall_fscore_support(y_test, y_pred, average='micro'))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oi-Mo14I54Vx"
   },
   "source": [
    "Observing Sparcity in L1,L2 Regularization\n",
    "------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "h-2ryaNbxmiL"
   },
   "outputs": [],
   "source": [
    "weights1 =[]\n",
    "for item in C:\n",
    "    lg = LogisticRegression(C=item, penalty='l1')\n",
    "    lg.fit(w[0:1000],y_train[0:1000])\n",
    "    weights1.append(np.count_nonzero(lg.coef_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6bKMZ2CRyjt5"
   },
   "outputs": [],
   "source": [
    "weights =[]\n",
    "for item in C:\n",
    "    lg = LogisticRegression(C=item, penalty='l2')\n",
    "    lg.fit(w[0:1000],y_train[0:1000])\n",
    "    weights.append(np.count_nonzero(lg.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 376
    },
    "colab_type": "code",
    "id": "CLrvWrrejObf",
    "outputId": "cd32c000-9683-4f38-9a23-7400f29bf268"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAFnCAYAAABKGFvpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl0lPXd/vH3zGQjG2SZECYsssgi\nJCSALAFMQFZpLbaIilj7lFYR3KotIqKC+qhFpVpr1WpFavWIIlV/rQ/QVhNRIhRJQgiggKiQyTIT\nQvZ97t8fkVEqIUCSmUxyvc7hnMw9y/2ZTyZc8723r8kwDAMRERHpVMzeLkBERETangJeRESkE1LA\ni4iIdEIKeBERkU5IAS8iItIJKeBFREQ6IQW8yFnau3cv119/PbNmzWLmzJlcddVV7Nq1yyPrvv76\n68nNzQXgjTfeOKfn/vWvf+XJJ588p+dcd911vPPOOwAYhsGLL77I8OHDPfZ+T2fIkCEUFBSc03PO\n571/13vvvUdFRQUAy5Yt4/333z/v1xLxND9vFyDiCwzDYPHixTz00EOkpqYCsHXrVpYuXUpaWhrd\nunVr1/WvX78egMbGRtasWcP8+fPP+rkLFy5s1brvv/9+XC4XkZGRrXodb2jte//973/PqFGjCA0N\nZc2aNW1UlYhnaAQvchZKSkpwOByMHDnSvWzGjBm88847dOvWjU2bNvHLX/6S3/zmN0ybNo0f/OAH\nfPnllwA4nU4WLVrErFmzmDp1KuvWrXO/xtSpU/nDH/7AzJkzsdvtHD16lGuvvZbp06fzk5/8xD1q\nnzp1Krt27eJ//ud/KC8vZ9asWbz66qvceOON7tdyuVwkJyezf//+U2p/+umnueeee4Cmkfm6deu4\n5pprmDx5MnfccQctXevqiiuu4KGHHsLf3/+Mj/vv91JQUMDixYuZOXMmM2fOJD093f3Y5557jgkT\nJvCTn/yEV199lalTpwKwfPly/vjHP7of99+3T3rmmWeYOXMm06ZN48Ybb6SsrMz9XleuXMm8efN4\n+eWX3e/dbrcza9Ys97+xY8dyyy23APDFF19wzTXXMHv2bKZPn87f//53AO6++26OHDnCddddx65d\nu07ZqrFjxw6uuOIKZs2axZVXXklOTg4AmzZt4tZbb2XFihXMnDmTyy67jIMHD56xbyLtRQEvchYi\nIiKIj4/npz/9KW+++SZHjx4FIDY21v2Y7du3c+211/Kvf/2LSy+9lMceewyAZ599lt69e7N582bW\nr1/PE088QX5+vvt5hYWFbNmyBZvNxr333sucOXP45z//yU033cSyZctOqePhhx/GYrGwefNmZs2a\nxSeffEJJSQkAu3fvJjw8nGHDhp3xvbz//vusW7eOLVu28Mknn7B79+4zPj4pKems+/Td93LXXXcx\ndOhQtmzZwp/+9CeWLVtGSUkJBw8e5MUXX+Sdd97htddeY/PmzWf9+tC0q+TVV1/lrbfeYuvWrdTV\n1fHXv/7VfX96ejp/+tOf+NnPfuZeZrPZ2Lx5M5s3b+a1114jJCSEX/ziFwCsWbOGKVOm8H//9388\n/PDD3HPPPdTX1/PII48A8MorrzBmzBj3a1VWVnLbbbexcuVKNm/ezC9+8Qt+/etf43K5APjwww9Z\nsGABW7ZsYdy4ce6tLyKepoAXOQsmk4l169Yxffp0/vKXvzBt2jTmzJnD1q1b3Y8ZOHAgiYmJAMyc\nOZPMzEwAVq5cyb333gtAnz59sFqtHDt2zP28k5v8a2tr2bFjBz/4wQ8AuPTSS8+4vz0qKooxY8aw\nZcsWAP75z39y2WWXtfheZs2aRVBQEMHBwVxwwQWnfNlorZPvpaqqih07drhDtl+/fowePZr09HT+\n85//MHbsWGJiYggMDOQnP/nJOa1jxIgRpKWlERoaitlsJikpyf2FC2DkyJFn3J2wYsUKrrnmGvfW\nmD/+8Y8sWrQIgNGjR1NbW4vD4Wj2+Xv27CE2NpbRo0cDTb/rkpIS8vLygKbPwYgRIwC46KKL2rS/\nIudC++BFzlJYWBi33nort956K06nk02bNnHHHXe4N9t2797d/djw8HD3ZuOcnBz3qN1sNuNwONyj\nve8+78SJE7hcLsLCwoCmLxUhISFnrGnOnDls2rSJq6++mn//+98899xzLb6P0NBQ988Wi4XGxsaz\n7EDLTr6X8vJyDMPg6quvdt9XVVXF+PHjqaqqOqVXPXv2PKd1VFdX88gjj7Bjxw4ASktL3V8svlvD\n6axfv56amhr36B1g27ZtPPvss5SUlGAymTAM45Tfz387fvw44eHhpywLCwujuLjY/fNJbd1fkXOh\ngBc5CwUFBRw7dsy9qTY6OpobbriBzZs3u/exnjhxwv340tJSd9D85je/4frrr+eaa67BZDIxefLk\n064jIiICk8lESUkJkZGRGIbB119/Td++fZuta/r06TzwwAOkp6fTrVs3Bg0a1FZvuVWioqKwWCy8\n9dZb3/uS8te//pWqqir37aKiIvfPZrP5lHAtLS393muvX7+eL7/8kk2bNhESEsLvfvc7CgsLW6xp\n//79vPTSS7z55puYzU0bL+vr67n99tt58sknSUlJoa6ujoSEhBbf23d/14ZhUFpaSlRUFF988UWL\ndYh4ijbRi5yF/Px8li5dyt69e93L9uzZg91uJz4+HoAjR46wb98+ALZs2eLehFtcXMyIESMwmUz8\n7W9/o7q6+pSAOykgIICJEyfyt7/9DWgaWd5www2YTCb3Y/z9/XG5XO5Tt8LCwpg8eTKrV69m9uzZ\n7fPmz4Ofnx8pKSm8/vrrQNOo++677yY/P5+EhAR27NjB8ePHqaur4+2333Y/z2q1cuDAAQCOHj16\n2uMDiouLGTBgACEhIeTl5ZGenn7afn5XVVUVd9xxB6tXryYmJsa9/OTv4uQm9fXr1+Pv7+9+PT8/\nP/eWmJMSEhJwOp3uXTD/+Mc/iI2NpXfv3ufaJpF2pRG8yFlISkriwQcfZNWqVZSXl+NyuYiOjuZ3\nv/sdcXFx7se8/PLL7Nq1i+DgYJ599lkAbrvtNpYuXUqPHj24+uqrueqqq7j33nt57bXXvree//3f\n/+XXv/41r732Gt27d+fxxx8/5X6r1cro0aOZMmUKzz//PKNGjXIfC3A2+9/Pxw9+8AMaGhooLCzk\nN7/5DYGBgaxZs6bFke6qVau4//77efPNNwG4/PLL6dWrF7169eKKK67giiuuoFevXlx22WW8/PLL\nAMyfP5+bb76ZGTNmcNFFFzFz5szvve7VV1/NrbfeysyZMxkyZAjLly/nlltucb/G6WzdupWjR4/y\n6KOP8uijjwIQGRnJa6+9xi9+8Qvmzp1LVFQUN910E9OmTWPx4sX8/e9/Z9asWVx99dU89NBD7tcK\nDg7mySef5MEHH6SqqorIyEjWrl17yhcxkY7ApPngRVpv06ZNvPvuu2cMmfayZ88eHnjgATZu3Ojx\ndZ8vwzDcgZiWlsaTTz55ykheRFpPm+hFfFhDQwPPPPMM1113nbdLOWvHjx9n/Pjx5OXlYRgG//d/\n/+c++0BE2o4CXsRH7du3j+nTpxMTE8Pll1/u7XLOWmRkJLfffjs/+9nPmDlzJqWlpe6LzohI29Em\nehERkU5II3gREZFOSAEvIiLSCXWq0+QcjvI2fb2IiGBKSs58fq2cmXrYeuph66mHbUN9bL227qHV\nGtbsfRrBn4Gfn8XbJfg89bD11MPWUw/bhvrYep7soQJeRESkE1LAi4iIdEIKeBERkU5IAS8iItIJ\nKeBFREQ6IQW8iIhIJ6SAFxER6YQU8CIiIp2QAl5ERKQTUsCLiIh0Qgp4ERGRdlZd20BGbgH5zkqP\nrbNTTTYjIiLSkRzJLyM9K48d+4qorW8kr7iKeZcM8Mi6FfAiIiJtqKaugR37CknLsvNVQdMsp1Hh\nQVw2oR/zZwyluqLGI3Uo4EVERNrA14XlpGfZycgtoKauEZMJEgdFk5oUx4j+kZjNJkK7+SvgRURE\nOrra+kb+s7+I9Kw8DtvLAIgIC2TGxX24ZKSNyPAgr9WmgBcRETlHec5K0jPz2L63gKraBkxA/IAo\nUhNtJAyKwmL2/jHsCngREZGzUN/g4tPPikjLzOPzY6UAhIcEMGdUP1JG2oju0c3LFZ5KAS8iInIG\nBcerSM/K4+OcAiqq6wG46IIIUhPjSLwwGj+L90frp6OAFxER+S8NjS4yDzpJy8xj/1clAIR282fW\nuL6kJNroGRHs5QpbpoAXERH5huNENR9m29mWbaesqmm0PqRPD1KSbIweHIO/X8ccrZ+OAl5ERLq0\nRpeL7EPFpGXmkXvkOAYQHOjH9DF9SEm0YYsO8XaJ50UBLyIiXdLxsho+zLbzYbadExV1AAyK605K\noo2Lh8YQ4G/xcoWto4AXEZEuw+UyyPmimPQsO9mHnRgGdAu0MHVUHKmJcfSOCfV2iW1GAS8iIp3e\niYpatn0zWi8uqwXggtgwUpPiGDesJ4EBvj1aPx0FvIiIdEouw2Dfl8dJz7STedCJyzAI9LdwyUgb\nqUk2LogN93aJ7UoBLyIinUpZZR0f5eSTnpWH40TTdd/7xISSmhTH+It60i2wa0Rf13iXIiLSqRmG\nwYGvT5CelcennzlodBkE+JmZGB9LalIcA3qFYzKZvF2mRyngRUTEZ1VU1/NxTj7pWXYKjlcBYIsO\nISXRRvKIWEKC/L1cofco4EVExKcYhsGhvFLSMvP4zwEHDY0u/Cwmxg/vSWpiHBf27t7lRuuno4AX\nERGfUFVTz/a9BaRn2clzVgLQM6IbKYlxTIyPJSw4wMsVdiwKeBER6bAMw+BIfjlpmXns3F9IXYML\ni9nExUNjSE20MbRfhEbrzVDAi4hIh1Nd28COfYWkZebxdVEFANHdg0hJtDEpwUb3EI3WW6KAFxGR\nDuOrgnLSs/LI2FdIbV0jZpOJUYOtpCbauKh/JGaN1s+aAl5ERLyqtq6RnfsLScuycyS/DIDI8EBm\nj+vL5AQbEWGBXq7QN7VrwH/++ecsWbKEn/3sZyxcuJD8/HyWLVtGY2MjVquVxx57jICAAN59913W\nr1+P2Wxm/vz5XHnlldTX17N8+XLsdjsWi4VHHnmEPn36tGe5IiLiQcccFaRn2tmem091bSMmIGFg\nFKlJcSQMiMJs1mi9Ndot4KuqqnjwwQeZMGGCe9nvf/97FixYwOzZs1m7di0bN25k7ty5PPPMM2zc\nuBF/f3/mzZvH9OnT+eCDDwgPD+eJJ57go48+4oknnuDJJ59sr3JFRMQD6uob2fVZEWlZdg4dKwWg\ne2gA00b34ZKRNqK6B3m5ws6j3QI+ICCAF154gRdeeMG9bMeOHaxevRqAKVOm8NJLL9G/f3/i4+MJ\nCwsDYNSoUezevZuMjAzmzp0LQHJyMitWrGivUkVEpJ3lF1eSnmXn45x8KmsaABjeP5LUxDhGDorC\nz2L2coWdT7sFvJ+fH35+p758dXU1AQFNRz5GRUXhcDhwOp1ERka6HxMZGfm95WazGZPJRF1dnfv5\npxMREYyfX9vOCGS1hrXp63VF6mHrqYetpx62jXPpY31DIxk5+WzO+Iqcw06gabQ+b+qFzBzfj9io\nkPYqs0Pz1GfRawfZGYbRJsu/q6SkqlU1/TerNQyHo7xNX7OrUQ9bTz1sPfWwbZxtH4tKqkjPsvNR\nTj7lVfUADO3bg9SkOEYNtjaN1l2uLvk7aevP4pm+LHg04IODg6mpqSEoKIjCwkJiYmKIiYnB6XS6\nH1NUVERiYiIxMTE4HA6GDh1KfX09hmGccfQuIiLe09DoIuugk/SsPHK/LAEgJMiPGRf3ISXRRq8u\nOlr3Jo8GfHJyMlu2bOFHP/oRW7duZfLkyYwcOZKVK1dSVlaGxWJh9+7drFixgoqKCjZv3szkyZP5\n4IMPGDdunCdLFRGRs+AsrebDbDvbsvMprawD4MLe3UlNimPMECv+bbzbVM5euwX83r17+e1vf0te\nXh5+fn5s2bKFxx9/nOXLl7NhwwZsNhtz587F39+fO++8k0WLFmEymVi6dClhYWFcdtllbN++nWuu\nuYaAgAAeffTR9ipVRETOQaPLxZ7DxaRn2ck5XIwBdAv049LRvUlNtBFnDfV2iQKYjLPZue0j2np/\njvbbtZ562HrqYeuph23DHODH394/yIfZdkrKawEYYAsnJdHG2GE9CfTXaL0lnXYfvIiI+BaXYZB7\n5DhpmXlkHy7G5TIIDLCQmhRHaqKNvj11dkJHpYAXEZHvKa2oZduefD7MtuMsrQFgQFx3JsXHMm5Y\nT7oFKj46Ov2GREQEaBqtH/iqhLTMPDIPOml0GQT4m5mc0IvUpDgujrfhdFZ4u0w5Swp4EZEurryq\njo9zCkjLyqOopBqAOGsIqYlxTBgeS3BQU1Ro3nXfooAXEemCDMPg86MnSM+ys+uzIhoaDfwsZpJH\nxJKaGMfAuHAFuo9TwIuIdCGVNfVs/2a0nl/cdPXP2MhgUpPiSB4RS2g3fy9XKG1FAS8i0skZhsFh\nexnpmXnsPFBEfYMLi9nE2GExTEmKY3CfHhqtd0IKeBGRTqq6toGM3ALSMu0cczQdHBfToxspSTYm\nxvciPFiX/+7MFPAiIp3Mkfwy0rPy2LGviNr6RixmE6OHWElNimNYvwjMGq13CQp4EZFOoKaugR37\nCknLsvNVQdOV0qLCg7hsQj8mJ/SiR2iglysUT1PAi4j4sK8Ly0nPspORW0BNXSMmEyQOiiY1KY4R\n/SMxmzVa76oU8CIiPqa2vpH/7C8iPSuPw/YyACLCAplxcR8uGWkjMjzIyxVKR6CAFxHxEXnOStIz\n89i+t4Cq2gZMQPyAKFITbSQMisJiNnu7ROlAFPAiIh1YfYOLTz8rIi0zj8+PlQIQHhLAnFH9SBlp\nI7pHNy9XKB2VAl5EpAMqPF5Fepadj3LyqaiuB+CiCyJITYwj8cJo/CwarcuZKeBFRDqIhkYXmQed\npGXmsf+rEgBCu/kza1xfUhJt9IwI9nKF4ksU8CIiXuY4Uc2H2Xa2Zdspq2oarQ/p04OUJBujB8fg\n76fRupw7BbyIiBc0ulxkHyomLSuP3C+OYwDBgX5MH9OHlEQbtugQb5coPk4BLyLiQcfLavgw286H\n2XZOVNQBMDAunNTEOC4eGkOAv8XLFUpnoYAXEWlnLpdBzhfFpGfZyT7sxDAgKMDClFFxpCbG0Scm\n1NslSiekgBcRaScnKmrZ9s1ovbisFoALYsNITYpj7LAYggL0X7C0H326RETakMsw2P9lCWmZeWQe\ndOIyDAL9LVwy0kZqko0LYsO9XaJ0EQp4EZE2UFZZx0c5+aRn5eE4UQNAn5hQUpPiGH9RT7oF6r9b\n8Sx94kREzpNhGBz4+gTpWXl8+pmDRpdBgJ+ZifGxpCbGMcAWjklTs4qXKOBFRM5RRXU9H+fkk55l\np+B4FQC26BBSEm0kj4glJMjfyxWKKOBFRM6KYRgcyislLTOP/xxw0NDows9iYvzwnqQmxnFh7+4a\nrUuHooAXETmDqpp6MnILScvMI89ZCUDPiG6kJMYxMT6WsOAAL1cocnoKeBGR/2IYBkfyy0nLymPn\nvkLqGlxYzCYuHhpDaqKNof0iNFqXDk8BLyLyjeraBnbsaxqtf11UAUB09yBSEm1MSrDRPUSjdfEd\nCngR6fK+KignPSuPjH2F1NY1YjaZGDXYSmqijYv6R2LWaF18kAJeRLqk2rpGdu4vJC3LzpH8MgAi\nwwOZPa4vkxNsRIQFerlCkdZRwItIl3LMUUF6pp3tuflU1zZiAhIGRpGaFEfCgCjMZo3WpXNQwItI\np1db38j2vfmkZdk5dKwUgO6hAUwb3YdLRtqI6h7k5QpF2p4CXkQ6rfziStKz7GzfW0BFdT0Aw/tH\nkpoYx8hBUfhZzF6uUKT9KOBFpFOpb3Cx+3MH6Vl5HPj6BNA0Wp89vi8pI23ERAR7uUIRz1DAi0in\nUFRSRXqWnY9y8imvahqtD+3bg9SkOGYkD+BESaWXKxTxLAW8iPishkYX2YecpGXmkftlCQAhQX7M\nuLgPKYk2ekWFAODvp03x0vUo4EXE5zhLq/kw28627HxKK+sAuLB3d1IT4xgz1Iq/n8XLFYp4nwJe\nRHxCo8vFnsPFpGfZyTlcjAF0C/Tj0tG9SU20EWcN9XaJIh2KAl5EOrSS8lo+zLbzYbadkvJaAAbY\nwklJtDF2WE8C/TVaFzkdjwZ8ZWUld911F6WlpdTX17N06VKsViurVq0CYMiQIaxevRqAF198kc2b\nN2Mymbj55ptJSUnxZKki4kUuwyD3yHHSMvPIPlSMyzAIDLCQmhRHaqKNvj3DvF2iSIfn0YD/29/+\nRv/+/bnzzjspLCzk+uuvx2q1smLFChISErjzzjtJT09nwIABvPfee7z++utUVFSwYMECJk2ahMWi\nb+oinVlpZR0f7bGTnmXHWVoDQN+eoaQmxTFuWE+6BWqjo8jZ8uhfS0REBJ999hkAZWVl9OjRg7y8\nPBISEgCYMmUKGRkZOBwOJk+eTEBAAJGRkcTFxXHo0CGGDBniyXJFxANchsGBr0pIy7KT+bmDRpdB\ngL+ZyQm9SE2K44LYME3NKnIePBrwc+bMYdOmTUyfPp2ysjKeffZZHnjgAff9UVFROBwOevToQWRk\npHt5ZGQkDodDAS/SiZRX1fFxTgFpWXkUlVQDEGcNITUxjgnDYwkO0mhdpDU8+hf0zjvvYLPZ+POf\n/8yBAwdYunQpYWHf7kszDOO0z2tu+X+LiAjGr41Pj7Fata+vtdTD1ussPTQMg9wvitmc8RUf77HT\n0OjC38/MlNG9mT2hP0MviGi30Xpn6aG3qY+t56keejTgd+/ezaRJkwAYOnQotbW1NDQ0uO8vLCwk\nJiaGmJgYjhw58r3lLSkpqWrTeq3WMByO8jZ9za5GPWy9ztDDypp6tn8zWs8vbvo7jY0MJjUpjuQR\nsYR28wfA6axol/V3hh52BOpj67V1D8/0ZcGjAd+vXz+ys7OZOXMmeXl5hISEEBcXx65duxgzZgxb\nt27luuuu44ILLmDdunXccsstlJSUUFRUxKBBgzxZqoi0kmEYHLaXkZ6Zx84DRdQ3uLCYTYwdFsOU\npDgG9+mhfesi7cijAX/VVVexYsUKFi5cSENDA6tWrcJqtXLffffhcrkYOXIkycnJAMyfP5+FCxdi\nMplYtWoVZrMuNSniC6prG8jILSAt084xR9OIPKZHN1ISbUyM70V4SICXKxTpGkzG2e7g9gFtvelI\nm6NaTz1sPV/p4ZcFZaRl5rFjXxG19Y1YzCYSL4wmNSmOYf0iMHtxtO4rPezo1MfW67Sb6EWkc6mp\na2DHvkLSsux8VdD0n1ZUeBCXTejH5IRe9AgN9HKFIl2XAl5EztnXheWkZ9nJyC2gpq4RkwkSB0WT\nmmRjRP8ozGbtWxfxNgW8iJyVuvpG/nOgiLTMPA7bywCICAtkxsV9uGSkjcjwIC9XKCLfpYAXkTPK\nc1aSnpnH9r0FVNU2YALiB0SRmmgjYVAUFh0AK9IhKeBF5HvqG1x8+lnTaP3zY6UAhIcEMGdUP1JG\n2oju0c3LFYpISxTwIuJWeLyK9Cw7H+XkU1FdD8BFF0SQmhhH4oXR+Fk0WhfxFQp4kS6uodFF5kEn\naZl57P+qBIDQbv7MGteXlJE2ekYGe7lCETkfCniRLspxopoPs+1sy7ZTVtU0Wh/cpwepSTZGD47B\n30+jdRFfpoAX6UIaXS6yDxWTlpVH7hfHMYDgQD+mjelNamIctugQb5coIm1EAS/SBRwvq+HDbDsf\nZts5UVEHwMC4cFIT47h4aAwB/m07C6OIeJ8CXqSTcrkMcr4oJj3LTvZhJ4YBQQEWpoyKIzUxjj4x\nod4uUUTakQJepJM5UVHLtm9G68VltQBcEBtGalIcY4fFEBSgP3uRrkB/6SKdgMsw2P9lCWmZeWQe\ndOIyDAL9LVwy0kZqko0LYsO9XaKIeJgCXsSHlVXW8VFOPulZeThO1ADQJyaU1EQb44fH0i1Qf+Ii\nXZX++kV8jGEYfPb1CdKy8vj0MweNLgN/PzMT42NJTYxjgC0ckxenZhWRjqHFgD927BiFhYWMHj2a\nN954g6ysLBYtWsTAgQM9UZ+IfKOiup7tOfmkZdkpOF4FgC06hJREG8kjYgkJ8vdyhSLSkbQY8Hff\nfTe/+c1v2LdvH2+++SY333wzDz30EOvWrfNEfSJdmmEY7DtSzNsfHOQ/Bxw0NLrws5gYP7wnqYlx\nXNi7u0brInJaLQa8yWQiISGBp556imuvvZaUlBSFu0g7q6qpJyO3kLTMPPKclQD0jOhGSmIcE+Nj\nCQsO8HKFItLRtRjwVVVV7Nmzhy1btvDXv/6Vuro6ysrKPFGbSJdiGAZH8stJy8pj575C6hpcWMwm\nJo20MWFYDEP7RWi0LiJnrcWA//nPf869997LVVddRWRkJE888QQ//OEPPVGbSJdQXdvAjn1No/Wv\niyoAiO4eREqijUkJNgZdEIXDUe7lKkXE17QY8IZh8M4777hv/+pXv2LDhg3tWpRIV/BVQTnpWXlk\n7Cuktq4Rs8lE0oXRTEmK46L+kZg1WheRVmg24Pft20dubi4vvfQS1dXV7uUNDQ0888wzXHPNNR4p\nUKQzqa1rZOf+QtKy7BzJb9rVFRkeyOxxfZmcYCMiLNDLFYpIZ9FswAcGBlJcXEx5eTmffvqpe7nJ\nZGLZsmUeKU6kszjmqCA908723HyqaxsxAQkDo0hNiiNhQBRms0brItK2mg34gQMHMnDgQMaPH09i\nYqInaxLpFGrrGtn1WRHp2XYOHSsFoHtoAJeO7sMlI3sR3b2blysUkc6sxX3w1dXV3HTTTZSXl2MY\nhnv5q6++2q6Fifgil8tg/9clZOwt4NPPHNTWNwIwvH8kqYlxjBwUhZ/F7OUqRaQraDHgV69ezZIl\nS4iNjfVEPSI+Kc9RwfbcAj7JLaSkvGkGt+juQcwY3oeJ8bHERAR7uUIR6WpaDPi4uDguv/xyT9Qi\n4lPKKuvYsa+Q7XsL+Kqw6TS2boEWLhnZi+QRvRjUu7uOhBcRr2k24I8ePQrAmDFj2LBhA2PHjsXP\n79uH9+nTp/2rE+lg6hsayTydriG1AAAgAElEQVToZPveAvZ+cRyXYWA2mUgYGEXyiFgSB0UT4G/x\ndpkiIs0H/PXXX4/JZHLvd3/++efd95lMJv7973+3f3UiHYDLMDh0rJTte/P5zwEH1bUNAPTrGUby\niFjGXdST8BBdOlZEOpZmA/7999/3ZB0iHU7h8Sq27y0gI7cAZ2nTXOsRYYGkJtlIHh5LnDXUyxWK\niDSvxX3wpzvn3WKx0L9/f6699lpCQkLapTARb6ioruc/+wvZnlvA4bymC9EE+ltIHhFL8ohYhvaN\n0DnrIuITWgx4q9VKdnY2l156KWazmffff59hw4ZRVFTEXXfdxR/+8AdP1CnSbhoaXeQcLmb73gKy\nDztpaDQwAcMviGDCiFhGDbYSFNDin4qISIfS4v9aBw4c4OWXX3YfYHfttddy880389xzz7Fw4cJ2\nL1CkPZycuW373nx27i+ioroegLjoEJJHxDJ+eKwuGysiPq3FgHc6nbhcrlOW5efnU19fT0VFRbsV\nJtIenKXVZOQ2ndpWeLwKgPBgf6aP6UPyiFj69gzVlKwi0im0GPCzZs1ixowZJCQkYDKZyM3NZerU\nqbz99ttMnTrVEzWKtEp1bQO7DhSxfW8Bnx09AYC/n5mxw2JIHhHL8P6RWMy6upyIdC4tBvxNN93E\nZZddxoEDB3C5XCxZsoQhQ4bQ2NiIxaLzfaVjanS5yD1Swva9+WQedFLf0LQVanCfHiSPiGXMkBiC\ng7RfXUQ6r2b/h0tPTyclJYWNGzeesjwnJ4ecnBzmzZvX7sWJnAvDMDhaVMH2vQV8sq+Qsso6AHpG\ndCN5RCwThscS3UMTvIhI19BswH/22WekpKScMlXsdyngpaMoKa/95pKx+RxzVAIQEuTHlFFxJI+I\nZUCvcO1XF5Eup9mAv+GGGwB45JFHcLlcFBcXY7VaPVaYyJnU1jWy+6CD7XsL2PflcQwDLGYTowZb\nmTA8loSBUfj7ab+6iHRdLe6EzMjI4J577iEgIIDNmzfz8MMPk5ycTGpqqgfKE/mWyzD47KsStu8t\nYNfnDmrrmqZiHWgLJ3lELBcP60loN38vVyki0jG0GPC/+93veOONN/jVr34FwOLFi1m8eLECXjwm\nz1lJxjeXjD1lKtYxfZgwIpbYSE3FKiLy31oM+ODgYKKjo923IyMj8fc//1HSu+++y4svvoifnx+3\n3norQ4YMYdmyZTQ2NmK1WnnssccICAjg3XffZf369ZjNZubPn8+VV1553usU31NW9Z2pWAtOnYp1\nwvBYLuzTQ1OxioicQYsBHxQUxM6dOwEoLS3lH//4B4GB53eFr5KSEp555hneeustqqqqePrpp9my\nZQsLFixg9uzZrF27lo0bNzJ37lyeeeYZNm7ciL+/P/PmzWP69On06NHjvNYrvqG+oZGsQ8Vk7C0g\n54tiGl2ailVE5Hy1GPD3338/q1atIicnhxkzZjBq1CgeeOCB81pZRkYGEyZMIDQ0lNDQUB588EGm\nTp3K6tWrAZgyZQovvfQS/fv3Jz4+nrCwMABGjRrF7t27dWGdTsgwDA4eKyUjt4Cd+4vcU7H27RlK\n8ohejLuoJ901FauIyDlrMeBzcnJ44oknCA1t/dSYx44do6amhsWLF1NWVsYtt9xCdXU1AQFN/4FH\nRUXhcDhwOp1ERka6nxcZGYnD4Wj1+qXjKCr5dipWx4nvTMWaaGPCiFh6aypWEZFWaTHgP/74Y556\n6inCw8OZOHEikydPdl+29nycOHGCP/zhD9jtdn76059iGIb7vu/+/F3NLf9vERHB+Pm17SZcqzWs\nTV+vKzrZw4qqOrZl2/lg11H2f3kcgMAAC1NG92bqmD7ED7Ji0VSsp6XPYeuph21DfWw9T/WwxYA/\nufm8qKiIHTt28Oyzz5KVlcUnn3xyziuLiooiKSkJPz8/+vbtS0hICBaLhZqaGoKCgigsLCQmJoaY\nmBicTqf7eUVFRSQmJrb4+iUlVedc05lYrWE4HOVt+ppdTY+IED7Y+WXTVKyHvp2K9aILIkj+r6lY\njxdr8qLT0eew9dTDtqE+tl5b9/BMXxZaDPj8/Hx27tzJzp07OXz4MDExMSxZsuS8Cpk0aRLLly/n\nl7/8JaWlpVRVVTFp0iS2bNnCj370I7Zu3crkyZMZOXIkK1eupKysDIvFwu7du1mxYsV5rVM8zzAM\nviwoZ3tOATsPFFFe1XTJWFt0CBNHxDLuop5Ehgd5uUoRkc6txYCfOnUqkyZNYtGiRYwfP75VK+vZ\nsyczZ85k/vz5AKxcuZL4+HjuuusuNmzYgM1mY+7cufj7+3PnnXeyaNEiTCYTS5cudR9wJx1XcWkN\nGblN+9Xzi5u2pvQIDdRUrCIiXmAyWtjB/fnnn7Nz50527NiBw+Fg8ODBjBs3jjlz5niqxrPW1puO\ntDmqZdW1Dez6rIiMvQUc+LppKlY/i5lRg6OZMDyW1LH9KDle6eUqfZs+h62nHrYN9bH1OtQm+sGD\nBzN48GCuuOIKPv30U1577TVWrFjRIQNePKPR5WLflyVk7C1g9+cO6r43FauV4KCmiyH5WXQ9eBER\nb2gx4B999FE+/fRTamtrGT9+PFdffTVr1671RG3SwTRNxZrPJ7mFlH5nKtYJ30zFatVUrCIiHcZZ\njeD/53/+h549e3qiHulgTlTU8klu0yVjjzmajnIPCfJjStI3U7HaNBWriEhH1GLA//jHP/ZEHdKB\n1NY3kvl501Ssud+ZijXpwmiSR/TSVKwiIj6gxYCXrsFlGHz29Qm2781n12ffTsU64JupWMdqKlYR\nEZ+igO/i7M5K96ltx8uapmKNCg9i+pg+TBjek15RIV6uUEREzkeLAX/48GFWr15NTk4OZrOZxMRE\n7rvvPvr16+eJ+qQdlFXVsXNfIRm5BRzJbzpdIyjAwuSEXiSP0FSsIiKdQYsB/+CDD/Lzn/+csWPH\nYhgG27dvZ9WqVaxbt84T9UkbqW9wkX3IyfbTTMU6YXgsSRdqKlYRkc6kxYA3DIPU1FT37enTp/PK\nK6+0Z03ShopOVJOelcdHe/Ipr6oHvpmKdXjTJWO7hwZ6uUIREWkPLQZ8fX09ubm5DB8+HIA9e/bQ\n2NjY7oXJ+Wt0ucg+VExaZh57jzTN2hbazZ8ZF/dhUnwvesdoKlYRkc6uxYC/6667uPPOOzl+vCko\nrFYrv/3tb9u9MDl3x8tq+DDbzofZdk5UNF2I5sLe3UlNimPMECv+bTyVroiIdFwtBnyvXr3YvHkz\n5eXlmEwmQkM1+utIXC6DvUeOk5aZR/ZhJ4YB3QItXDqqNylJNnpb9fsSEemKWgz4X//61/zlL3/R\nbG4dTGllHR/tsZOeZcdZWgNA/15hpCTGMW5YTwIDNFoXEenKWgz4Cy64gGXLlpGUlIS//7cXOpk3\nb167FibfZxgGB74q4YMsO5mfO2h0GQT4m7lkpI3UJBsXxIZ7u0QREekgzuogO4vFwp49e05ZroD3\nnIrqej7OyScty07h8aZ51ntbQ0hNimP8RbEEB+l6RSIicqoWk+GRRx7B5XJRXFyM1Wr1RE1C02j9\ncF4ZH2Tm8Z8DRTQ0uvCzmEkeEUtqYhwD4zTJi4iINK/FgM/IyOCee+4hICCAzZs38/DDD5OcnHzK\nufHSdqpqGvhkXwFpmXkcc1QC0DMymCmJNpLje+l68CIiclZaDPjf/e53vPHGG/zqV78CYPHixSxe\nvFgB38a+LCgjLTOPT/YVUlfvwmI2cfHQGFKT4hjat4dG6yIick5aDPjg4GCio6PdtyMjI0852E7O\nX21dIzv2F5KWmceXBU3XhI/uHkRKoo1J8b10lTkRETlvLQZ8UFAQO3fuBKC0tJR//OMfBAYqeFrj\nmKOC9Ew723Pzqa5txGSCpAujSUmMY0T/SMxmjdZFRKR1Wgz4+++/n1WrVpGTk8OMGTMYNWoUDzzw\ngCdq61TqGxrZdcDBB1l5HDpWCkCP0ACmj+nDJSNtRIYHeblCERHpTM7qSnbPP//8KcsqKiraraDO\npqHRxTsfHSE9y05FddNkLyP6R5KaFMfIQVFYzGYvVygiIp1Ri+ly2WWX8fHHH5+ybMmSJe1WUGdS\nWVPP2g1Z/CPjK0wmuGx8Px5dPIE7rkpk1GCrwl1ERNpNiyN4i8XCs88+S05ODosXLwaaztGWM3Oc\nqObJN7PJL65i1GArv/zhRQRqvnUREfGQFoeQERERvPzyyxQVFbFkyRIqKip0ylYLDttLeegvu8gv\nrmLW2L4suWKEwl1ERDyqxRG8YRj4+flx33338fbbb3PttddSU1Pjidp80q4DRbzw9300NLq4bsZg\npozq7e2SRESkC2ox4H/4wx+6f547dy5DhgzhiSeeaNeifJFhGGzZeZQ3PzhEQICF265IIGFgdMtP\nFBERaQctbqKfP3++++fXX3+dYcOG8eKLL7ZrUb6m0eXilS2f8cYHh+gRFsjd145SuIuIiFed02Hc\n7733XnvV4bOqaxt4auMe0rLs9IkJ5Z7rRtO3Z5i3yxIRkS7unOYZ1dHzpzpeVsOTb+7hmKOChIFR\n3Hj5cLoFaupWERHxvnNKo8cff7y96vA5XxWU89TGbE5U1DFlVBwLpl2o89pFRKTDaDHgDx48yJtv\nvklpaekpI/g1a9a0a2EdWfYhJ8+9k0tdfSNXTR3EjIv76NRBERHpUFoM+Ntvv53Zs2czbNgwT9TT\n4dmdlTz9Vg5+FhNLrohn9BCrt0sSERH5nhYDPjo6mptvvtkTtfiEf+06issw+PmcixTuIiLSYbW4\n0/iSSy7ho48+oq6uDpfL5f7XFVVU17N9bwHR3YMYMyTG2+WIiIg0q8UR/LPPPvu92eNMJhP79+9v\nt6I6qvSsPOoaXEwb3VtztouISIfWYsDv2rXLE3V0eA2NLt7fnUdggIVJCTZvlyMiInJGLQZ8ZWUl\nL7/8Mjk5OZhMJpKSkvjpT39KUFCQJ+rrMD79zEFJeS3TRvcmOEjnuouISMfW4j74e++9l4qKCq6+\n+mrmz5+Pw+Fg5cqVnqitQ/nnrqOYgEvHaPIYERHp+FocijqdTtauXeu+PWXKFK677rp2LaqjOZxX\nyhf2MhIHRdMzItjb5YiIiLSoxRF8dXU11dXV7ttVVVXU1ta2aqU1NTVMmzaNTZs2kZ+fz3XXXceC\nBQu47bbbqKurA+Ddd9/lJz/5CVdeeSVvvvlmq9bXWv/cdRSA6Rq9i4iIj2hxBH/VVVcxe/ZsRowY\ngWEY7Nu3j9tuu61VK3322Wfp3r07AL///e9ZsGABs2fPZu3atWzcuJG5c+fyzDPPsHHjRvz9/Zk3\nbx7Tp0+nR48erVrv+TheVsOuAw56W0MY2i/C4+sXERE5Hy0G/Lx585g4cSK5ubmYTCbuu+8+evbs\ned4rPHz4MIcOHSI1NRWAHTt2sHr1aqBp8/9LL71E//79iY+PJyysaVa2UaNGsXv3bqZOnXre6z1f\nGbkFuAyDaWN0OVoREfEdLW6ir62tJTc3l9LSUkpKSti2bRsbN2487xX+9re/Zfny5e7b1dXVBAQE\nABAVFYXD4cDpdBIZGel+TGRkJA6H47zX2Rp2ZxUAQ/p6fuuBiIjI+WpxBL9o0SLMZjNxcXGnLJ83\nb945r+ztt98mMTGRPn36nPb+5qajPdtpaiMigvHzs5xzXWdyvKIWi9nEsIFWLBbNFnc+rNYwb5fg\n89TD1lMP24b62Hqe6mGLAd/Q0MDrr7/eJitLS0vj6NGjpKWlUVBQQEBAAMHBwdTU1BAUFERhYSEx\nMTHExMTgdDrdzysqKiIxMbHF1y8pqWqTOk+yWsPIK6ogunsQx49XtulrdxVWaxgOR7m3y/Bp6mHr\nqYdtQ31svbbu4Zm+LLQ4JB00aBAlJSVtUsiTTz7JW2+9xRtvvMGVV17JkiVLSE5OZsuWLQBs3bqV\nyZMnM3LkSHJycigrK6OyspLdu3czZsyYNqnhXFRU11NRXU+MTo0TEREf0+IIvqCggBkzZjBw4EAs\nlm83f7/66qttUsAtt9zCXXfdxYYNG7DZbMydOxd/f3/uvPNOFi1ahMlkYunSpe4D7jzJ7mi6Bn/P\niG4eX7eIiEhrtBjwN9xwQ7us+JZbbnH/vG7duu/dP2vWLGbNmtUu6z5b+c6mzfIxCngREfExLQb8\n2LFjPVFHh2T/JuB7RmoTvYiI+BYdFn4Gdqc20YuIiG9SwJ9BvrMSi9lEVPeuNXOeiIj4PgX8Gdgd\nlUR3D8JiVptERMS3KLmaUVvXSHlVHdE9tHleRER8jwK+GfWNLgAC/dv2yngiIiKeoIBvRuM3AW8x\na4IZERHxPQr4ZjS6mq5/b7Eo4EVExPco4JvRoBG8iIj4MAV8M06O4P00g5yIiPggpVczGhu/2USv\nEbyIiPggBXwz3PvgdQ68iIj4IKVXMxpc3+yD10F2IiLigxTwzdAmehER8WUK+GboPHgREfFlCvhm\n6Ch6ERHxZUqvZjToQjciIuLDFPDN+HYfvFokIiK+R+nVjEaX9sGLiIjvUsA3Q9eiFxERX6aAb4ZO\nkxMREV+mgG/GyQvd6Ch6ERHxRUqvZmgELyIivkwB3wxdi15ERHyZ0qsZjboWvYiI+DAFfDNObqL3\n0yZ6ERHxQQr4Zny7iV4BLyIivkcB34yGk5PN6Ch6ERHxQUqvZuhCNyIi4ssU8M34dh+8WiQiIr5H\n6dUMXYteRER8mQK+GdpELyIivkwB34yTAW/WCF5ERHyQAr45TfmO4l1ERHyRAr4ZhjvhFfEiIuJ7\nFPDNML7JdzVIRER8kfKrGScDXtvoRUTEFyngm9WU8CYlvIiI+CAFfDPcA3jlu4iI+CAFfHOMlh8i\nIiLSUSngm/HtCF5DeBER8T1+nl7hmjVr+PTTT2loaODGG28kPj6eZcuW0djYiNVq5bHHHiMgIIB3\n332X9evXYzabmT9/PldeeaVH6zQMDeFFRMR3eTTgP/nkEw4ePMiGDRsoKSnhiiuuYMKECSxYsIDZ\ns2ezdu1aNm7cyNy5c3nmmWfYuHEj/v7+zJs3j+nTp9OjRw9PlgtoH7yIiPgmj26iv/jii3nqqacA\nCA8Pp7q6mh07dnDppZcCMGXKFDIyMsjOziY+Pp6wsDCCgoIYNWoUu3fv9mSp7tPklO8iIuKLPBrw\nFouF4OBgADZu3Mgll1xCdXU1AQEBAERFReFwOHA6nURGRrqfFxkZicPh8GSp3x5jpyG8iIj4II/v\ngwf417/+xcaNG3nppZeYMWOGe3lz+73Pdn94REQwfn6WNqkxMKCpNdFRofQIC2yT1+yqrNYwb5fg\n89TD1lMP24b62Hqe6qHHA37btm0899xzvPjii4SFhREcHExNTQ1BQUEUFhYSExNDTEwMTqfT/Zyi\noiISExNbfO2Skqo2q7Omth6A48crqK+pa7PX7Wqs1jAcjnJvl+HT1MPWUw/bhvrYem3dwzN9WfDo\nJvry8nLWrFnD888/7z5gLjk5mS1btgCwdetWJk+ezMiRI8nJyaGsrIzKykp2797NmDFjPFnqt/vg\ntYleRER8kEdH8O+99x4lJSXcfvvt7mWPPvooK1euZMOGDdhsNubOnYu/vz933nknixYtwmQysXTp\nUsLCPLtZSKfJiYiIL/NowF911VVcddVV31u+bt267y2bNWsWs2bN8kRZZ6QBvIiI+CJdya4ZOk1O\nRER8mQK+RYp4ERHxPQr4ZpzcB69N9CIi4osU8M3QIXYiIuLLFPAt0AheRER8kQK+Gd8eZKeEFxER\n36OAb4aBDqMXERHfpYBvjvJdRER8mAK+Gd8eRa+IFxER36OAb8bJo+iV7yIi4osU8M3QpehFRMSX\nKeBboBG8iIj4IgV8M9z74HWYnYiI+CAFfEuU7yIi4oMU8M3QbHIiIuLLFPDN+PYoekW8iIj4HgV8\nc3QYvYiI+DAFfDMMdAS9iIj4LgV8Mwy0/11ERHyXAr45hva/i4iI71LAN8MwDG2iFxERn6WAb8bJ\ny9yIiIj4IgV8MwxDB9mJiIjvUsA3y9D4XUREfJYCvhmGzpMTEREfpoBvhvJdRER8mQK+OYYOsRMR\nEd+lgG+GgU6TExER36WAb44BGsOLiIivUsA3Q/vgRUTElyngm2FoH7yIiPgwBXwzmvbBK+JFRMQ3\nKeCbo8lmRETEhyngm+HSZDMiIuLDFPBnYFbCi4iIj1LAN8NloKPsRETEZyngm2MYmBXwIiLioxTw\nzTB0oRsREfFhCvhmGGgELyIivksB3wxNFysiIr5MAd8Mw0AjeBER8Vl+3i7gTB5++GGys7MxmUys\nWLGChIQEj63bwACTvv+IiIhv6rABv3PnTr766is2bNjA4cOHWbFiBRs2bPDY+ntFBhMU5O+x9YmI\niLSlDhvwGRkZTJs2DYCBAwdSWlpKRUUFoaGhHln/r+YnYrWGUVxc4ZH1iYiItKUOG/BOp5Phw4e7\nb0dGRuJwOM4Y8BERwfj5Wdq0Dqs1rE1frytSD1tPPWw99bBtqI+t56kedtiA/29G04npZ1RSUtWm\n67Raw3A4ytv0Nbsa9bD11MPWUw/bhvrYem3dwzN9WeiwR5HFxMTgdDrdt4uKirBarV6sSERExHd0\n2ICfOHEiW7ZsASA3N5eYmBiP7X8XERHxdR12E/2oUaMYPnw4V199NSaTifvvv9/bJYmIiPiMDhvw\nAL/+9a+9XYKIiIhP6rCb6EVEROT8KeBFREQ6IQW8iIhIJ6SAFxER6YQU8CIiIp2QAl5ERKQTMhln\ncw1YERER8SkawYuIiHRCCngREZFOSAEvIiLSCSngRUREOiEFvIiISCekgBcREemEOvRsct708MMP\nk52djclkYsWKFSQkJHi7pA5lzZo1fPrppzQ0NHDjjTcSHx/PsmXLaGxsxGq18thjjxEQEMC7777L\n+vXrMZvNzJ8/nyuvvJL6+nqWL1+O3W7HYrHwyCOP0KdPH2+/Ja+oqanhBz/4AUuWLGHChAnq4Xl4\n9913efHFF/Hz8+PWW29lyJAh6uM5qKys5K677qK0tJT6+nqWLl2K1Wpl1apVAAwZMoTVq1cD8OKL\nL7J582ZMJhM333wzKSkplJeXc+edd1JeXk5wcDBPPPEEPXr08OI78pzPP/+cJUuW8LOf/YyFCxeS\nn5/f6s/egQMHTtv782LI9+zYscO44YYbDMMwjEOHDhnz58/3ckUdS0ZGhvGLX/zCMAzDOH78uJGS\nkmIsX77ceO+99wzDMIwnnnjCePXVV43KykpjxowZRllZmVFdXW3MmTPHKCkpMTZt2mSsWrXKMAzD\n2LZtm3Hbbbd57b1429q1a40f//jHxltvvaUenofjx48bM2bMMMrLy43CwkJj5cqV6uM5euWVV4zH\nH3/cMAzDKCgoMGbOnGksXLjQyM7ONgzDMO644w4jLS3N+Prrr40rrrjCqK2tNYqLi42ZM2caDQ0N\nxtNPP2288MILhmEYxuuvv26sWbPGa+/FkyorK42FCxcaK1euNF555RXDMIw2+eydrvfnS5voTyMj\nI4Np06YBMHDgQEpLS6moqPByVR3HxRdfzFNPPQVAeHg41dXV7Nixg0svvRSAKVOmkJGRQXZ2NvHx\n8YSFhREUFMSoUaPYvXs3GRkZTJ8+HYDk5GR2797ttffiTYcPH+bQoUOkpqYCqIfnISMjgwkTJhAa\nGkpMTAwPPvig+niOIiIiOHHiBABlZWX06NGDvLw891bLkz3csWMHkydPJiAggMjISOLi4jh06NAp\nPTz52K4gICCAF154gZiYGPey1n726urqTtv786WAPw2n00lERIT7dmRkJA6Hw4sVdSwWi4Xg4GAA\nNm7cyCWXXEJ1dTUBAQEAREVF4XA4cDqdREZGup93so/fXW42mzGZTNTV1Xn+jXjZb3/7W5YvX+6+\nrR6eu2PHjlFTU8PixYtZsGABGRkZ6uM5mjNnDna7nenTp7Nw4UKWLVtGeHi4+/5z6WFUVBRFRUUe\nfw/e4OfnR1BQ0CnLWvvZczqdp+39edd43s/sQgxdzfe0/vWvf7Fx40ZeeuklZsyY4V7eXL/OdXln\n9vbbb5OYmNjs/l718OydOHGCP/zhD9jtdn7605+e0gv1sWXvvPMONpuNP//5zxw4cIClS5cSFhbm\nvv9cetUV+9ectvjstbafGsGfRkxMDE6n0327qKgIq9XqxYo6nm3btvHcc8/xwgsvEBYWRnBwMDU1\nNQAUFhYSExNz2j6eXH7yW2l9fT2GYbi/9XYVaWlp/Pvf/2b+/Pm8+eab/PGPf1QPz0NUVBRJSUn4\n+fnRt29fQkJCCAkJUR/Pwe7du5k0aRIAQ4cOpba2lpKSEvf9zfXwu8tP9vDksq6qtX/DVqvVvbvk\nu69xvhTwpzFx4kS2bNkCQG5uLjExMYSGhnq5qo6jvLycNWvW8Pzzz7uPlk1OTnb3bOvWrUyePJmR\nI0eSk5NDWVkZlZWV7N69mzFjxjBx4kQ2b94MwAcffMC4ceO89l685cknn+Stt97ijTfe4Morr2TJ\nkiXq4XmYNGkSn3zyCS6Xi5KSEqqqqtTHc9SvXz+ys7MByMvLIyQkhIEDB7Jr1y7g2x6OHz+etLQ0\n6urqKCwspKioiEGDBp3Sw5OP7apa+9nz9/dnwIAB3+v9+dJscs14/PHH2bVrFyaTifvvv5+hQ4d6\nu6QOY8OGDTz99NP079/fvezRRx9l5cqV1NbWYrPZeOSRR/D392fz5s38+c9/xmQysXDhQi6//HIa\nGxtZuXIlX375JQEBATz66KP06tXLi+/Iu55++mni4uKYNGkSd911l3p4jl5//XU2btwIwE033UR8\nfLz6eA4qKytZsWIFxcXFNDQ0cNttt2G1WrnvvvtwuVyMHDmSu+++G4BXXnmF//f//h8mk4nbb7+d\nCRMmUFn5/9u7f5TGojAOwz8xYBHQdHaW6bS0yCpS2aRLZxGshOwgGF1BNpAduIWUIZAmJm0gWFlk\n7MQpBoUZh4FkiiuH5zJHaeUAAAFOSURBVClP9XG58HLP/fcjt7e3eXl5yfHxce7v73/b4i/VfD7P\n3d1d1ut1arVaTk9P8/DwkH6//1/n3mq1+uux34fAA0CBbNEDQIEEHgAKJPAAUCCBB4ACCTwAFMiX\n7IC9PD8/Zzgc5unpKfV6PUnS6/XSarUqngxIvCYH7OH9/T1XV1dpt9vpdDpJksVikW63m/F4nLOz\ns4onBFzBAzubTCY5ODj4jHvy69/Vj4+POTk5qXAy4IN78MDOlstlzs/Pv6yLO3wfAg/s7PDwMG9v\nb1WPAfyDwAM7azabmU6nX9YXi0VeX18rmAj4k8ADO7u8vEy9Xs9oNPpcWy6Xub6+zmazqXAy4IOn\n6IG9bLfbDAaDzGazNBqNHB0d5ebmJhcXF1WPBkTgAaBItugBoEACDwAFEngAKJDAA0CBBB4ACiTw\nAFAggQeAAgk8ABToJ0dm1xJS6fwpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f72920959e8>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(C,weights1)\n",
    "ymin, ymax = plt.ylim()\n",
    "plt.xlabel('C')\n",
    "plt.ylabel('non-zero weights')\n",
    "plt.title('Sparcity in l1 regularization')\n",
    "plt.axis('tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aB6YQqW-krKg"
   },
   "source": [
    "#From the above plot we can clearly see that L1 regularization  creates sparsity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 376
    },
    "colab_type": "code",
    "id": "_6HroVIYy1yn",
    "outputId": "bf7ba60e-c5a0-417d-f6e0-4f22e8a818b8"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf0AAAFnCAYAAACsMZCtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X9Y1fXdx/Hn4VeEQgodlJzb0hKZ\nCmpODSWDRIx2e/dDDRRraTbDShdqhEZaM0il3Mw0V/6Y2piaK+9W6LbQbJKN2AxNKlfdM0Q4B0ER\nEBC+9x9enluWetRzDpbf1+O6ui7Ol+/3c96ft3S9zufzPRwshmEYiIiIyBXP63IXICIiIm1DoS8i\nImISCn0RERGTUOiLiIiYhEJfRETEJBT6IiIiJqHQF3HB3r17uf/++xk5ciQJCQnce++9FBYWtslz\n33///ezbtw+ADRs2XNS169atY/HixRd1zYQJE3jrrbcA+PjjjxkzZgy33347d999N3//+98vaix3\nCQ8P5/Dhwxd1zaXM/UzvvPMOx48fB2DWrFm89957lzyWSJszROSStLS0GEOGDDHy8/Mdx7Zu3WoM\nHDjQqKura7M6Tp48adx0000ef56UlBTjzTffNBoaGoyBAwcaBQUFhmEYxvbt242hQ4d6/PnPpkeP\nHkZZWVmbPmdCQkKbP6eIu2ilL3KJqqqqsNlsREVFOY6NGDGCt956i6uvvprNmzczefJkZs6cyfDh\nw/nZz37G119/DYDdbmfSpEmMHDmSuLg4Vq1a5RgjLi6Ol156iYSEBA4dOsTBgwcZP3488fHx3HPP\nPY7VfVxcHIWFhTzwwAPU1NQwcuRI1q9fzy9+8QvHWC0tLURHR7N///5WtS9ZsoTZs2cDp1bwq1at\nIjk5mZiYGB5//HGM83xmV1NTE88++yyDBw8G4KabbqKiooJjx45969z/nMvhw4eZMmUKCQkJJCQk\nsGPHDse5y5cv5+abb+aee+5h/fr1xMXFAZCens7LL7/sOO8/H5+2dOlSEhISGD58OL/4xS8c9SxZ\nsoQ5c+YwevRoVq9e7Zj7oUOHGDlypOO/gQMH8uijjwLw5ZdfkpyczO233058fDxvv/02AE8++SRf\nffUVEyZMoLCwsNXux+7du7nrrrsYOXIkY8aMobi4GIDNmzfz2GOPkZGRQUJCAomJiXzxxRfn7K+I\nJyn0RS5Rx44d6dOnD/fddx8bN27k4MGDAHTu3Nlxzq5duxg/fjx/+ctfuO2221i4cCEAy5Yt4wc/\n+AF5eXmsWbOGnJwcysrKHNeVl5ezdetWrrvuOp566inuuOMO/vznP/Pwww8za9asVnU899xzeHt7\nk5eXx8iRI/nwww+pqqoCoKioiKCgICIiIs47l/fee49Vq1axdetWPvzwQ4qKis55brt27RgxYoTj\n8fvvv8+Pf/xjgoKCznr+mXN54okn6NmzJ1u3bmXFihXMmjWLqqoqvvjiC1599VXeeustXn/9dfLy\n8s5b73/au3cv69ev54033mDbtm00Njaybt06x/d37NjBihUr+PnPf+44dt1115GXl0deXh6vv/46\n7dq148EHHwRgwYIFxMbG8u677/Lcc88xe/ZsmpqayMrKAmDt2rUMGDDAMVZtbS3Tpk1jzpw55OXl\n8eCDDzJjxgxaWlocPRo3bhxbt25l0KBBrFmz5qLmJ+IuCn2RS2SxWFi1ahXx8fH87ne/Y/jw4dxx\nxx1s27bNcU737t3p27cvAAkJCfzjH/8AYM6cOTz11FMAdO3aFavVyjfffOO47tZbbwWgoaGB3bt3\n87Of/QyA22677bz370NCQhgwYABbt24F4M9//jOJiYlO5zJy5Ej8/f0JCAjgxz/+casXIOdTUlLC\nc889xzPPPHPOc07Ppa6ujt27dzuC90c/+hE33XQTO3bs4O9//zsDBw4kNDSUq666invuueeCnv+0\n3r17s337dtq3b4+Xlxf9+vVzvAgDiIqKIjg4+JzXZ2RkkJyc7Ni1efnll5k0aRJwaiejoaEBm812\nzus/+eQTOnfuzE033QSc+reuqqqitLQUOPVz0Lt3bwB+8pOfXHB/RdzN53IXIPJ9FhgYyGOPPcZj\njz2G3W5n8+bNPP74444t32uuucZxblBQkGPLubi42LG69/LywmazOVaFZ15XXV1NS0sLgYGBwKkX\nGu3atTtvTXfccQebN28mKSmJv/71ryxfvtzpPNq3b+/42tvbm+bmZqfXFBUVMX36dObPn8+gQYPO\ned7pudTU1GAYBklJSY7v1dXVMXjwYOrq6lr1qlOnTk6f/0z19fVkZWWxe/duAI4ePep4sXFmDWez\nZs0aTpw44VjlA+zcuZNly5ZRVVWFxWLBMIxW/z7/6ciRI9/a6QgMDKSystLx9WkX2l8RT1Doi1yi\nw4cP88033zi2ea+99loeeugh8vLyHPdsq6urHecfPXrUET4zZ87k/vvvJzk5GYvFQkxMzFmfo2PH\njlgsFqqqqggODsYwDP7973/zwx/+8Jx1xcfH88wzz7Bjxw6uvvpqbrjhBndN2aGkpIRp06bx4osv\nttrmPp+QkBC8vb154403vvXCZd26ddTV1TkeV1RUOL728vJqFbhHjx791thr1qzh66+/ZvPmzbRr\n144XX3yR8vJypzXt37+flStXsnHjRry8Tm18NjU1MX36dBYvXsywYcNobGwkMjLS6dzO/Lc2DIOj\nR48SEhLCl19+6bQOkbai7X2RS1RWVsbUqVPZu3ev49gnn3zCoUOH6NOnDwBfffUVn376KQBbt251\nbP9WVlbSu3dvLBYLf/zjH6mvr28Veqf5+fkxZMgQ/vjHPwKnVqAPPfQQFovFcY6vry8tLS2OXyML\nDAwkJiaGefPmcfvtt7t93oZhkJ6eztNPP33BgQ/g4+PDsGHDyM3NBU6tzp988knKysqIjIxk9+7d\nHDlyhMbGRt58803HdVarlZKSEgAOHjx41vcbVFZW0q1bN9q1a0dpaSk7duw4az/PVFdXx+OPP868\nefMIDQ11HD/9b3F6O37NmjX4+vo6xvPx8fnWmxYjIyOx2+2O2zd/+tOf6Ny5Mz/4wQ8uuD8ibUEr\nfZFL1K9fP5599lnmzp1LTU0NLS0tXHvttbz44ot06dLFcc7q1aspLCwkICCAZcuWATBt2jSmTp1K\nhw4dSEpK4t577+Wpp57i9ddf/9bzzJ8/nxkzZvD6669zzTXXsGjRolbft1qt3HTTTcTGxvLKK6/Q\nv39/x3sLLuR+/sX65z//yWeffcaiRYta1ZKTk0OvXr3Oe+3cuXN5+umn2bhxIwCjRo0iLCyMsLAw\n7rrrLu666y7CwsJITExk9erVAIwdO5ZHHnmEESNG8JOf/ISEhIRvjZuUlMRjjz1GQkIC4eHhpKen\n8+ijjzrGOJtt27Zx8OBBsrOzyc7OBiA4OJjXX3+dBx98kDvvvJOQkBAefvhhhg8fzpQpU3j77bcZ\nOXIkSUlJ/OpXv3KMFRAQwOLFi3n22Wepq6sjODiYF154odWLM5HvAotxvt/NEZFLtnnzZrZs2XLe\n4PGUTz75hGeeeYZNmza1+XNfKsMwHCG5fft2Fi9e3GrFLyKu0/a+yBXm5MmTLF26lAkTJlzuUi7Y\nkSNHGDx4MKWlpRiGwbvvvuv4rQcRcR+FvsgV5NNPPyU+Pp7Q0FBGjRp1ucu5YMHBwUyfPp2f//zn\nJCQkcPToUccH5YiI+2h7X0RExCS00hcRETEJhb6IiIhJXPG/smez1bh9zI4dA6iqOv/vAMv5qYeu\nUw9dpx66Tj10nSd6aLUGnvW4VvqXwMfH+3KX8L2nHrpOPXSdeug69dB1bdlDhb6IiIhJKPRFRERM\nQqEvIiJiEgp9ERERk1Doi4iImIRCX0RExCQU+iIiIiah0BcRETEJhb6IiIhJKPRFRERMQqEvIiJi\nEgp9ERERk1Doi4iImIRCX0RExCQU+iIiIiah0BcRETEJhb6IiIhJKPRFRERMQqEvIiJiEgp9ERER\nk1Doi4iImIRCX0RExCQU+iIiIiah0BcRETEJH08NXF9fT3p6OpWVlTQ0NJCamsrQoUNJT0/nf//3\nf2nXrh2/+c1vuOaaa9iyZQtr1qzBy8uLsWPHMmbMGJqamkhPT+fQoUN4e3uTlZVF165dKSkpYe7c\nuQCEh4czb948T01BRETkiuKxlX5+fj69e/dm3bp1LF68mOzsbDZs2EDHjh3ZtGkTiYmJFBYWUldX\nx9KlS1m9ejVr165lzZo1VFdX8/bbbxMUFMTvf/97pkyZQk5ODgDz588nIyOD3Nxcjh8/zo4dOzw1\nBRERkSuKx0I/MTGRyZMnA1BWVkanTp3Iz89n1KhRANx7773cdttt7Nmzhz59+hAYGIi/vz/9+/en\nqKiIgoIC4uPjAYiOjqaoqIjGxkZKS0uJjIwEIDY2loKCAk9NQURE5Iri8Xv6SUlJzJgxg4yMDEpL\nS3n//feZMGECv/zlL6mursZutxMcHOw4Pzg4GJvN1uq4l5cXFosFu91OUFCQ49yQkBBsNpunpyAi\nInJF8Ng9/dNyc3PZv38/M2fOpKWlheuvv55HHnmEl19+mVdeeYWf/OQnrc43DOOs45zt+LnOPVPH\njgH4+HhfWvHnYbUGun1Ms1EPXaceuk49dJ166Lq26qHHQn/v3r2EhIQQFhZGREQEzc3NeHl58dOf\n/hSAoUOHsmTJEm699VbsdrvjuoqKCvr27UtoaCg2m42ePXvS1NSEYRhYrVaqq6sd55aXlxMaGnre\nOqqq6tw+N6s1EJutxu3jmol66Dr10HXqoevUQ9d5oofnehHhse39wsJCVq5cCYDdbqeuro7//u//\nZufOnQDs27eP66+/nqioKIqLizl27Bi1tbUUFRUxYMAAhgwZQl5eHnDqTYGDBg3C19eXbt26UVhY\nCMC2bduIiYnx1BRERESuKBbjQvbIL8GJEyeYPXs2ZWVlnDhxgkceeYSbb76ZJ554ApvNRkBAAM8/\n/zzXXnsteXl5vPbaa1gsFlJSUhg1ahTNzc3MmTOHr7/+Gj8/P7KzswkLC+PAgQNkZmbS0tJCVFQU\nTz755Hnr8MQrUL2ydZ166Dr10HXqoevUQ9e15UrfY6H/XaHQ/25SD12nHrpOPXSdeui6K2J7X0RE\nRL5bFPoiIiImodAXERExCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoi\nIiImodAXERExCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAX\nERExCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+\niIiISSj0RURETEKhLyIiYhI+nhq4vr6e9PR0KisraWhoIDU1la1bt7Jv3z46dOgAwKRJk7j11lvp\n1asX/fv3d1y7evVqWlpaSE9P59ChQ3h7e5OVlUXXrl0pKSlh7ty5AISHhzNv3jxPTUFEROSK4rHQ\nz8/Pp3fv3kyePJnS0lImTpxIv379ePzxx4mNjW11bvv27Vm7dm2rY1u2bCEoKIicnBw++OADcnJy\nWLx4MfPnzycjI4PIyEjS0tLYsWMHw4YN89Q0RERErhge295PTExk8uTJAJSVldGpU6eLur6goID4\n+HgAoqOjKSoqorGxkdLSUiIjIwGIjY2loKDAvYWLiIhcoTx+Tz8pKYkZM2aQkZEBwLp167jvvvv4\n5S9/yZEjRwBobGwkLS2NpKQkVq1aBYDdbic4OPhUkV5eWCwW7HY7QUFBjrFDQkKw2WyenoKIiMgV\nwWPb+6fl5uayf/9+Zs6cSUZGBh06dCAiIoIVK1bw0ksvkZmZyaxZsxg1ahQWi4WUlBQGDBjwrXEM\nw7igY/+pY8cAfHy83TKXM1mtgW4f02zUQ9eph65TD12nHrqurXrosdDfu3cvISEhhIWFERERQXNz\nMz169CAkJASAuLg4xxvykpOTHdcNHjyYzz//nNDQUGw2Gz179qSpqQnDMLBarVRXVzvOLS8vJzQ0\n9Lx1VFXVuX1uVmsgNluN28c1E/XQdeqh69RD16mHrvNED8/1IsJj2/uFhYWsXLkSOLVVX1dXR2Zm\nJgcPHgRg9+7d3HjjjXz55ZekpaVhGAYnT56kqKiIG2+8kSFDhpCXlwecelPgoEGD8PX1pVu3bhQW\nFgKwbds2YmJiPDUFERGRK4rHVvpJSUnMnj2bcePGceLECTIzMwkICGD69OlcffXVBAQEkJWVRUhI\nCJ07d2b06NF4eXkRFxdHZGQkvXr1YteuXSQnJ+Pn50d2djYAGRkZZGZm0tLSQlRUFNHR0Z6agoiI\nyBXFYlzIjfHvMU9sO2k7y3XqoevUQ9eph65TD113RWzvi4iIyHeLQl9ERMQkFPoiIiImodAXEREx\nCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiI\nSSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiISSj0RURE\nTEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiISSj0RURETEKhLyIi\nYhI+nhq4vr6e9PR0KisraWhoIDU1la1bt7Jv3z46dOgAwKRJk7j11lvZsmULa9aswcvLi7FjxzJm\nzBiamppIT0/n0KFDeHt7k5WVRdeuXSkpKWHu3LkAhIeHM2/ePE9NQURE5IrisdDPz8+nd+/eTJ48\nmdLSUiZOnEi/fv14/PHHiY2NdZxXV1fH0qVL2bRpE76+vowePZr4+Hjy8/MJCgoiJyeHDz74gJyc\nHBYvXsz8+fPJyMggMjKStLQ0duzYwbBhwzw1DRERkSuGx7b3ExMTmTx5MgBlZWV06tTprOft2bOH\nPn36EBgYiL+/P/3796eoqIiCggLi4+MBiI6OpqioiMbGRkpLS4mMjAQgNjaWgoICT01BRETkiuKx\nlf5pSUlJHD58mOXLl7N69WrWrVvHqlWrCAkJ4amnnsJutxMcHOw4Pzg4GJvN1uq4l5cXFosFu91O\nUFCQ49yQkBBsNtt5n79jxwB8fLzdPi+rNdDtY5qNeug69dB16qHr1EPXtVUPnYb+N998Q3l5OTfd\ndBMbNmzgn//8J5MmTaJ79+4X9AS5ubns37+fmTNnkpGRQYcOHYiIiGDFihW89NJL9OvXr9X5hmGc\ndZyzHT/XuWeqqqq7oDovhtUaiM1W4/ZxzUQ9dJ166Dr10HXqoes80cNzvYhwur3/5JNP4uvry6ef\nfsrGjRtJSEjgV7/6ldMn3Lt3L2VlZQBERETQ3NxMjx49iIiIACAuLo7PP/+c0NBQ7Ha747qKigpC\nQ0MJDQ11rOKbmpowDAOr1Up1dbXj3PLyckJDQ53WIiIiIhcQ+haLhcjISP785z8zfvx4hg0bdkEr\n7MLCQlauXAmA3W6nrq6OzMxMDh48CMDu3bu58cYbiYqKori4mGPHjlFbW0tRUREDBgxgyJAh5OXl\nAafeFDho0CB8fX3p1q0bhYWFAGzbto2YmJhLnryIiIiZON3er6ur45NPPmHr1q2sW7eOxsZGjh07\n5nTgpKQkZs+ezbhx4zhx4gSZmZkEBAQwffp0rr76agICAsjKysLf35+0tDQmTZqExWJh6tSpBAYG\nkpiYyK5du0hOTsbPz4/s7GwAMjIyyMzMpKWlhaioKKKjo13vgoiIiAlYDCfL9nfeeYdXXnmFu+++\nm/vvv5+cnByCg4N54IEH2qpGl3jiXpPuYblOPXSdeug69dB16qHr2vKevtOVvmEYvPXWW47Hv/zl\nL/nDH/7gvspERESkTZwz9D/99FP27dvHypUrqa+vdxw/efIkS5cuJTk5uU0KFBEREfc4Z+hfddVV\nVFZWUlNTw8cff+w4brFYmDVrVpsUJyIiIu5zztDv3r073bt3Z/DgwfTt27ctaxIREREPcHpPv76+\nnocffpiamppWv6q3fv16jxYmIiIi7uU09OfNm0dqaiqdO3dui3pERETEQ5yGfpcuXRg1alRb1CIi\nIiIedM7QP/3JeQMGDOAPf/gDAwcOxMfn/0/v2rWr56sTERERtzln6N9///1YLBbHffxXXnnF8T2L\nxcJf//pXz1cnIiIibnPO0H/vvffasg4RERHxMKf39M/2O/ne3t5cf/31jB8/nnbt2nmkMBEREXEv\np39lz2q1cujQISIiIujVqxfl5eVcc801VFRU8MQTT7RFjSIiIuIGTlf6JSUlrF692vEmvvHjx/PI\nI4+wfPlyUlJSPF6giIiIuIfTlb7dbqelpaXVsbKyMpqamjh+/LjHChMRERH3crrSHzlyJCNGjCAy\nMhKLxcK+ffuIi4vjzTffJC4uri1qFBERETdwGvoPP/wwiYmJlJSU0NLSQmpqKuHh4TQ3N+Pt7d0W\nNYqIiIgbnHN7f8eOHQBs2rSJv//979TU1FBbW0txcTGbNm1S4IuIiHzPnHOl/9lnnzFs2LBWf1b3\nTKNHj/ZYUSIiIuJ+5wz9hx56CICsrCxaWlqorKzEarW2WWEiIiLiXk7fvV9QUMDw4cOZMGECAM89\n9xzbt2/3dF0iIiLiZk5D/8UXX2TDhg2OVf6UKVN4+eWXPV6YiIiIuJfT0A8ICODaa691PA4ODsbX\n19ejRYmIiIj7Of2VPX9/fz766CMAjh49yp/+9CeuuuoqjxcmIiIi7uV0pf/000/z2muvUVxczIgR\nI9i5cyfPPPNMW9QmIiIibuR0pV9cXExOTg7t27dvi3pERETEQ5yG/t/+9jd+/etfExQUxJAhQ4iJ\niXF8JK+IiIh8fzgN/Xnz5gFQUVHB7t27WbZsGf/85z/58MMPPV6ciIiIuI/T0C8rK+Ojjz7io48+\n4l//+hehoaGkpqa2RW0iIiLiRk5DPy4ujqFDhzJp0iQGDx7cFjWJiIiIBzgN/bfeeouPPvqI9evX\ns3jxYnr06MGgQYO444472qI+ERERcROnod+jRw969OjBXXfdxccff8zrr79ORkaG09Cvr68nPT2d\nyspKGhoaSE1NJTY2FoCdO3fy4IMP8tlnnwHQq1cv+vfv77h29erVtLS0kJ6ezqFDh/D29iYrK4uu\nXbtSUlLC3LlzAQgPD3e850BERETOz2noZ2dn8/HHH9PQ0MDgwYNJSkrihRdecDpwfn4+vXv3ZvLk\nyZSWljJx4kRiY2NpaGhgxYoVrf54T/v27Vm7dm2r67ds2UJQUBA5OTl88MEH5OTksHjxYubPn09G\nRgaRkZGkpaWxY8cOhg0bdglTFxERMZcLWuk/8MADdOrU6aIGTkxMdHxdVlbmuH758uWMGzeOhQsX\nnvf6goIC7rzzTgCio6PJyMigsbGR0tJSIiMjAYiNjaWgoEChLyIicgGcfiLf3XfffdGBf6akpCRm\nzJhBRkYGX331FSUlJdx+++2tzmlsbCQtLY2kpCRWrVoFgN1uJzg4+FSRXl5YLBbsdjtBQUGO60JC\nQrDZbJdcm4iIiJk4Xem7Kjc3l/379zNz5kzCwsKYM2fOt86ZNWsWo0aNwmKxkJKSwoABA751jmEY\nF3TsP3XsGICPj/elFX8WK/9nH3/bU+q28URExNyGRHVh4n/1apPn8ljo7927l5CQEMLCwoiIiKC2\ntpYDBw4wY8YM4NSH/aSkpLBu3TqSk5Md1w0ePJjPP/+c0NBQbDYbPXv2pKmpCcMwsFqtVFdXO84t\nLy8nNDT0vHVUVdW5dV71dY0ANDc7f8Eh5+btbVEPXaQeuk49dJ166B42W41bx7NaA8963Gno/+tf\n/2LevHkUFxfj5eVF3759yczM5Ec/+tF5ryssLKS0tJTZs2djt9tpaWnhvffew8vr1B2FuLg41q1b\nx5dffsnSpUtZtGgRzc3NFBUVMXLkSPz8/MjLyyMmJob8/HwGDRqEr68v3bp1o7CwkAEDBrBt2zYm\nTJhwCe24dGPjbmDqvf3c/g9kNlZroHroIvXQdeqh69RD17VlD52G/rPPPsvEiRMZOHAghmGwa9cu\n5s6d67j3fi5JSUnMnj2bcePGceLECTIzMx2Bf6Zu3brRuXNnRo8ejZeXF3FxcURGRtKrVy927dpF\ncnIyfn5+ZGdnA5CRkUFmZiYtLS1ERUURHR19iVMXERExF4vh5Mb4/fffz5o1a1odu++++/jd737n\n0cLcxROvnvTK1nXqoevUQ9eph65TD13niR6ea3vf6bv3m5qa2Ldvn+PxJ598QnNzs/sqExERkTbh\ndHv/iSeeIC0tjSNHjgBgtVp5/vnnPV6YiIiIuJfT0A8LCyMvL4+amhosFgvt27dvi7pERETEzZxu\n75/+FbvAwEAFvoiIyPeY05X+j3/8Y2bNmkW/fv3w9fV1HB89erRHCxMRERH3chr6TU1NeHt788kn\nn7Q6rtAXERH5fnEa+llZWbS0tFBZWdnqL+OJiIjI94vTe/oFBQUMHz7c8cl3zz33HNu3b/d0XSIi\nIuJmTkP/xRdfZMOGDY5V/pQpU3j55Zc9XpiIiIi4l9PQDwgI4Nprr3U8Dg4ObvWGPhEREfl+cHpP\n39/fn48++giAo0eP8qc//YmrrrrK44WJiIiIezld6T/99NO89tprFBcXM2LECHbu3MkzzzzTFrWJ\niIiIG13QJ/K98sorrY4dP37cYwWJiIiIZzhd6ScmJvK3v/2t1bHU1FSPFSQiIiKe4TT0vb29WbZs\nGcuXL3ccc/LXeEVEROQ7yGnod+zYkdWrV1NRUUFqairHjx/HYrG0RW0iIiLiRk7v6RuGgY+PD5mZ\nmbz55puMHz+eEydOtEVtIiIi4kZOV/r/9V//5fj6zjvvJDs7m65du3q0KBEREXE/p6E/duxYx9e5\nublERETw6quverQoERERcT+noX+md955x1N1iIiIiIddVOjrXfsiIiLfXxcV+osWLfJUHSIiIuJh\nTt+9/8UXX7Bx40aOHj3aaqW/YMECjxYmIiIi7uU09KdPn87tt99OREREW9QjIiIiHuI09K+99loe\neeSRtqhFREREPMjpPf1bbrk0thC9AAAUP0lEQVSFDz74gMbGRlpaWhz/iYiIyPeL05X+smXLvvVX\n9SwWC/v37/dYUSIiIuJ+TkO/sLCwLeoQERERD3Ma+rW1taxevZri4mIsFgv9+vXjvvvuw9/fvy3q\nExERETdxek//qaee4vjx4yQlJTF27FhsNhtz5sxpi9pERETEjZyu9O12Oy+88ILjcWxsLBMmTPBo\nUSIiIuJ+Tlf69fX11NfXOx7X1dXR0NDgdOD6+nqmTZtGSkoKY8aMIT8/3/G9nTt3Eh4e7ni8ZcsW\n7rnnHsaMGcPGjRsBaGpqIi0tjeTkZFJSUjh48CAAJSUlJCUlkZSUxNNPP33hMxURETE5pyv9e++9\nl9tvv53evXtjGAaffvop06ZNczpwfn4+vXv3ZvLkyZSWljJx4kRiY2NpaGhgxYoVWK1W4NSLiKVL\nl7Jp0yZ8fX0ZPXo08fHx5OfnExQURE5ODh988AE5OTksXryY+fPnk5GRQWRkJGlpaezYsYNhw4a5\n3gkREZErnNPQHz16NEOGDGHfvn1YLBYyMzPp1KmT04ETExMdX5eVlTmuWb58OePGjWPhwoUA7Nmz\nhz59+hAYGAhA//79KSoqoqCggDvvvBOA6OhoMjIyaGxspLS0lMjISODUrYaCggKFvoiIyAVwGvoN\nDQ3s27fP8dn7O3fuBE69GLgQSUlJHD58mOXLl/PVV19RUlLCtGnTHKFvt9sJDg52nB8cHIzNZmt1\n3MvLC4vFgt1uJygoyHFuSEgINpvtvM/fsWMAPj7eF1TrxbBaA90+ptmoh65TD12nHrpOPXRdW/XQ\naehPmjQJLy8vunTp0ur4hYZ+bm4u+/fvZ+bMmYSFhTl95/+5/nzv2Y5fyJ/6raqqu6A6L4bVGojN\nVuP2cc1EPXSdeug69dB16qHrPNHDc72IcBr6J0+eJDc396KfcO/evYSEhBAWFkZERAS1tbUcOHCA\nGTNmAFBRUUFKSgqPPvoodrvdcV1FRQV9+/YlNDQUm81Gz549aWpqwjAMrFYr1dXVjnPLy8sJDQ29\n6NpERETMyOm792+44QaqqqoueuDCwkJWrlwJnNrCb2lp4S9/+QsbNmxgw4YNhIaGsm7dOqKioigu\nLubYsWPU1tZSVFTEgAEDGDJkCHl5ecCpNwUOGjQIX19funXr5viUwG3bthETE3PRtYmIiJiR05X+\n4cOHGTFiBN27d8fb+//vja9fv/681yUlJTF79mzGjRvHiRMnyMzMxMvr268x/P39SUtLY9KkSVgs\nFqZOnUpgYCCJiYns2rWL5ORk/Pz8yM7OBiAjI4PMzExaWlqIiooiOjr6YucsIiJiShbDyY3xjz76\n6KzHBw4c6JGC3M0T95p0D8t16qHr1EPXqYeuUw9d9526p/99CXcRERE5P6f39EVEROTKoNAXEREx\nCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiI\nSSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiISSj0RURE\nTEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiISSj0RURETEKhLyIi\nYhI+nhq4vr6e9PR0KisraWhoIDU1lQ4dOrBgwQJ8fHzw8/Nj4cKFBAcH06tXL/r37++4dvXq1bS0\ntJCens6hQ4fw9vYmKyuLrl27UlJSwty5cwEIDw9n3rx5npqCiIjIFcVjoZ+fn0/v3r2ZPHkypaWl\nTJw4kfDwcBYsWEDXrl156aWX2LBhA1OmTKF9+/asXbu21fVbtmwhKCiInJwcPvjgA3Jycli8eDHz\n588nIyODyMhI0tLS2LFjB8OGDfPUNERERK4YHtveT0xMZPLkyQCUlZXRqVMnfvOb39C1a1cMw6C8\nvJzOnTuf8/qCggLi4+MBiI6OpqioiMbGRkpLS4mMjAQgNjaWgoICT01BRETkiuKxlf5pSUlJHD58\nmOXLlwPw/vvvM3/+fLp168aoUaMAaGxsJC0tjdLSUhISEnjggQew2+0EBwcD4OXlhcViwW63ExQU\n5Bg7JCQEm83m6SmIiIhcETwe+rm5uezfv5+ZM2eyZcsWbrnlFmJiYli0aBErVqxgypQpzJo1i1Gj\nRmGxWEhJSWHAgAHfGscwjAs69p86dgzAx8fbLXM5k9Ua6PYxzUY9dJ166Dr10HXqoevaqoceC/29\ne/cSEhJCWFgYERERNDc38+6775KYmIjFYiEhIYElS5YAkJyc7Lhu8ODBfP7554SGhmKz2ejZsydN\nTU0YhoHVaqW6utpxbnl5OaGhoeeto6qqzu1zs1oDsdlq3D6umaiHrlMPXaceuk49dJ0neniuFxEe\nu6dfWFjIypUrAbDb7dTV1bFs2TL2798PwJ49e7j++uv58ssvSUtLwzAMTp48SVFRETfeeCNDhgwh\nLy8POPWmwEGDBuHr60u3bt0oLCwEYNu2bcTExHhqCiIiIlcUj630k5KSmD17NuPGjePEiRNkZmZi\ntVqZN28e3t7e+Pv7s2DBAkJCQujcuTOjR4/Gy8uLuLg4IiMj6dWrF7t27SI5ORk/Pz+ys7MByMjI\nIDMzk5aWFqKiooiOjvbUFERERK4oFuNCbox/j3li20nbWa5TD12nHrpOPXSdeui6K2J7X0RERL5b\nFPoiIiImodAXERExCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiIm\nodAXERExCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXEREx\nCYW+iIiISSj0RURETEKhLyIiYhIKfREREZNQ6IuIiJiEQl9ERMQkFPoiIiImodAXERExCYW+iIiI\nSSj0RURETEKhLyIiYhI+nhq4vr6e9PR0KisraWhoIDU1lQ4dOrBgwQJ8fHzw8/Nj4cKFBAcHs2XL\nFtasWYOXlxdjx45lzJgxNDU1kZ6ezqFDh/D29iYrK4uuXbtSUlLC3LlzAQgPD2fevHmemoKIiMgV\nxWMr/fz8fHr37s26detYvHgx2dnZrFq1igULFrB27Vr69evHhg0bqKurY+nSpaxevZq1a9eyZs0a\nqqurefvttwkKCuL3v/89U6ZMIScnB4D58+eTkZFBbm4ux48fZ8eOHZ6agoiIyBXFY6GfmJjI5MmT\nASgrK6NTp0785je/oWvXrhiGQXl5OZ07d2bPnj306dOHwMBA/P396d+/P0VFRRQUFBAfHw9AdHQ0\nRUVFNDY2UlpaSmRkJACxsbEUFBR4agoiIiJXFI/f009KSmLGjBlkZGQA8P777zNy5EjsdjujRo3C\nbrcTHBzsOD84OBibzdbquJeXFxaLBbvdTlBQkOPckJAQbDabp6cgIiJyRfDYPf3TcnNz2b9/PzNn\nzmTLli3ccsstxMTEsGjRIlasWEGXLl1anW8YxlnHOdvxc517po4dA/Dx8b604s/Dag10+5hmox66\nTj10nXroOvXQdW3VQ4+F/t69ewkJCSEsLIyIiAiam5t59913SUxMxGKxkJCQwJIlS+jXrx92u91x\nXUVFBX379iU0NBSbzUbPnj1pamrCMAysVivV1dWOc8vLywkNDT1vHVVVdW6fm9UaiM1W4/ZxzUQ9\ndJ166Dr10HXqoes80cNzvYjw2PZ+YWEhK1euBMBut1NXV8eyZcvYv38/AHv27OH6668nKiqK4uJi\njh07Rm1tLUVFRQwYMIAhQ4aQl5cHnHpT4KBBg/D19aVbt24UFhYCsG3bNmJiYjw1BRERkSuKx1b6\nSUlJzJ49m3HjxnHixAkyMzOxWq3MmzcPb29v/P39WbBgAf7+/qSlpTFp0iQsFgtTp04lMDCQxMRE\ndu3aRXJyMn5+fmRnZwOQkZFBZmYmLS0tREVFER0d7akpiIiIXFEsxoXcGP8e88S2k7azXKceuk49\ndJ166Dr10HVXxPa+iIiIfLco9EVERExCoS8iImISCn0RERGTUOiLiIiYhEJfRETEJBT6IiIiJqHQ\nFxERMQmFvoiIiEko9EVERExCoS8iImISCn0RERGTUOiLiIiYhEJfRETEJBT6IiIiJqHQFxERMQmF\nvoiIiEko9EVERExCoS8iImISCn0RERGTUOiLiIiYhEJfRETEJBT6IiIiJqHQFxERMQmFvoiIiEko\n9EVERExCoS8iImISCn0RERGTsBiGYVzuIkRERMTztNIXERExCYW+iIiISSj0RURETEKhLyIiYhIK\nfREREZNQ6IuIiJiEz+Uu4PvkueeeY8+ePVgsFjIyMoiMjLzcJX3nLFiwgI8//piTJ0/yi1/8gj59\n+jBr1iyam5uxWq0sXLgQPz8/tmzZwpo1a/Dy8mLs2LGMGTOGpqYm0tPTOXToEN7e3mRlZdG1a9fL\nPaXL4sSJE/zsZz8jNTWVm2++WT28SFu2bOHVV1/Fx8eHxx57jPDwcPXwItTW1vLEE09w9OhRmpqa\nmDp1Klarlblz5wIQHh7OvHnzAHj11VfJy8vDYrHwyCOPMGzYMGpqakhLS6OmpoaAgABycnLo0KHD\nZZxR2/r8889JTU3l5z//OSkpKZSVlbn881dSUnLW/l80Qy7I7t27jYceesgwDMM4cOCAMXbs2Mtc\n0XdPQUGB8eCDDxqGYRhHjhwxhg0bZqSnpxvvvPOOYRiGkZOTY6xfv96ora01RowYYRw7dsyor683\n7rjjDqOqqsrYvHmzMXfuXMMwDGPnzp3GtGnTLttcLrcXXnjBuPvuu4033nhDPbxIR44cMUaMGGHU\n1NQY5eXlxpw5c9TDi7R27Vpj0aJFhmEYxuHDh42EhAQjJSXF2LNnj2EYhvH4448b27dvN/79738b\nd911l9HQ0GBUVlYaCQkJxsmTJ40lS5YYv/3tbw3DMIzc3FxjwYIFl20uba22ttZISUkx5syZY6xd\nu9YwDMMtP39n6/+l0Pb+BSooKGD48OEAdO/enaNHj3L8+PHLXNV3y09/+lN+/etfAxAUFER9fT27\nd+/mtttuAyA2NpaCggL27NlDnz59CAwMxN/fn/79+1NUVERBQQHx8fEAREdHU1RUdNnmcjn961//\n4sCBA9x6660A6uFFKigo4Oabb6Z9+/aEhoby7LPPqocXqWPHjlRXVwNw7NgxOnToQGlpqWN383QP\nd+/eTUxMDH5+fgQHB9OlSxcOHDjQqoenzzULPz8/fvvb3xIaGuo45urPX2Nj41n7fykU+hfIbrfT\nsWNHx+Pg4GBsNttlrOi7x9vbm4CAAAA2bdrELbfcQn19PX5+fgCEhIRgs9mw2+0EBwc7rjvdyzOP\ne3l5YbFYaGxsbPuJXGbPP/886enpjsfq4cX55ptvOHHiBFOmTGHcuHEUFBSohxfpjjvu4NChQ8TH\nx5OSksKsWbMICgpyfP9iehgSEkJFRUWbz+Fy8fHxwd/fv9UxV3/+7Hb7Wft/SfVd0lWCoU8vPqe/\n/OUvbNq0iZUrVzJixAjH8XP17GKPX8nefPNN+vbte857yOrhhamuruall17i0KFD3Hfffa36oB46\n99Zbb3Hdddfx2muvUVJSwtSpUwkMDHR8/2J6Zcb+nY87fv5c6alW+hcoNDQUu93ueFxRUYHVar2M\nFX037dy5k+XLl/Pb3/6WwMBAAgICOHHiBADl5eWEhoaetZenj59+9drU1IRhGI5Xx2axfft2/vrX\nvzJ27Fg2btzIyy+/rB5epJCQEPr164ePjw8//OEPadeuHe3atVMPL0JRURFDhw4FoGfPnjQ0NFBV\nVeX4/rl6eObx0z08fczMXP1/2Gq1Om63nDnGpVDoX6AhQ4awdetWAPbt20doaCjt27e/zFV9t9TU\n1LBgwQJeeeUVxzt1o6OjHX3btm0bMTExREVFUVxczLFjx6itraWoqIgBAwYwZMgQ8vLyAMjPz2fQ\noEGXbS6Xy+LFi3njjTfYsGEDY8aMITU1VT28SEOHDuXDDz+kpaWFqqoq6urq1MOL9KMf/Yg9e/YA\nUFpaSrt27ejevTuFhYXA//dw8ODBbN++ncbGRsrLy6moqOCGG25o1cPT55qZqz9/vr6+dOvW7Vv9\nvxT6K3sXYdGiRRQWFmKxWHj66afp2bPn5S7pO+UPf/gDS5Ys4frrr3ccy87OZs6cOTQ0NHDdddeR\nlZWFr68veXl5vPbaa1gsFlJSUhg1ahTNzc3MmTOHr7/+Gj8/P7KzswkLC7uMM7q8lixZQpcuXRg6\ndChPPPGEengRcnNz2bRpEwAPP/wwffr0UQ8vQm1tLRkZGVRWVnLy5EmmTZuG1WolMzOTlpYWoqKi\nePLJJwFYu3Yt//M//4PFYmH69OncfPPN1NbWMnPmTKqrqwkKCmLhwoWtbg9cyfbu3cvzzz9PaWkp\nPj4+dOrUiUWLFpGenu7Sz9+BAwfO2v+LpdAXERExCW3vi4iImIRCX0RExCQU+iIiIiah0BcRETEJ\nhb6IiIhJ6BP5RMStKioqWLBgAZ9//jnt2rUD4NFHHyU6OvoyVyYi+pU9EXEbwzAYO3Ysd955J+PH\njwfgs88+Y+LEifz+97/nhz/84WWuUMTctNIXEbcpKCjAYrE4Ah9O/e3vd955h2uuueYyViYioHv6\nIuJGX3zxBX369PnWcQW+yHeDQl9E3Mbb25vm5ubLXYaInINCX0TcpkePHvzjH//41vHPPvuMurq6\ny1CRiJxJoS8ibjNw4EDatWvHihUrHMe++OILHn74YQ4fPnwZKxMR0Lv3RcTNjh8/TlZWFnv27KFD\nhw5cddVVTJs2jcjIyMtdmojpKfRFRERMQtv7IiIiJqHQFxERMQmFvoiIiEko9EVERExCoS8iImIS\nCn0RERGTUOiLiIiYhEJfRETEJP4PDmbmDqs90p0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f72937c1390>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(C,weights)\n",
    "ymin, ymax = plt.ylim()\n",
    "plt.xlabel('C')\n",
    "plt.ylabel('non-zero weights')\n",
    "plt.title('Sparcity in l2 regularization')\n",
    "plt.axis('tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mouwoyg_6B_M"
   },
   "source": [
    "Checking Multicollinearity\n",
    "-----------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 347
    },
    "colab_type": "code",
    "id": "x63stfPPQ6CE",
    "outputId": "ad0a70c8-d5e7-46fb-89da-2e07876f3441"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAFKCAYAAABRtSXvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XuQVOX95/FP3+fWwwwwoBhBNLoq\nRo1VyppEVMr4Eze/pPxZBDQXs1q56MZoKslKNJpsEiwhV9EkZBVjSv0JiJaa0o1IBGNcxIV4w6hc\nVMARYYYZmB56+nbO2T/6wgzd59LD9HQ3vF//DHT3TH8fBubD9znP8xyfZVmWAADAiPNXuwAAAA5X\nhCwAABVCyAIAUCGELAAAFULIAgBQIYQsAAAVEhzpL9jVFXN9TXt7k3p74yP91lXFmOoDY6oPjKk+\nMKasjo6o7XNV6WSDwUA13raiGFN9YEz1gTHVB8bkjuliAAAqhJAFAKBCCFkAACqEkAUAoEIIWQAA\nKoSQBQCgQghZAAAqhJAFAKBCCFkAACqEkAUAoEIIWQAAKoSQBQAMy99f+UD/+8k3ZVlWtUupWYQs\nAGBYnv9np1761y71D6SrXUrNImQBAMOSTGckSSaNrC1CFgAwLKm0KUkySVlbhCwAYFiSaUOSuCbr\ngJAFAAxLMpUNWTpZe4QsAGBYUplcyNLJ2iJkAQDDkipMF1e5kBpGyAIAhqUwXUzK2iJkAQBlsyyr\n0MlyTdYeIQsAKJthWoX9sWSsPUIWAFC2/B5ZiU7WCSELAChbfmWxxDVZJ4QsAKBsqcygTpaQtUXI\nAgDKlk4f6GQt0+GFRzhCFgBQNjpZbwhZAEDZUoM7WULWFiELAChbOsPqYi8IWQBA2ZKDt/CQsbYI\nWQBA2dJs4fGEkAUAlC3FdLEnhCwAoGyDFz7RydojZAEAZRu68KmKhdQ4QhYAULbB08Vs4bFHyAIA\nysZ0sTeELACgbJz45A0hCwAo2+AtPJxdbI+QBQCUbcj9ZOlkbRGyAICysU/WG0IWAFA2btruTdDt\nBevWrdMNN9ygE088UZJ00kkn6dZbb614YQCA2pXi7GJPXENWks455xwtWrSo0rUAAOrEkIVPdLK2\nmC4GAJSNa7LeeArZLVu26Fvf+pauuOIKvfjii5WuCQBQ49JMF3vis1z6/F27dmnDhg2aNWuWduzY\noa9+9atauXKlwuFwyddnMoaCwUBFigUA1Iar/tdf1dOXlCRd/e/TdNkFH69yRbXJ9ZrsxIkTdeml\nl0qSJk+erPHjx2vXrl069thjS76+tzfu+qYdHVF1dcXKLLW2Mab6wJjqA2OqfYnkgWuysVjisBnb\ncL5PHR1R2+dcp4uffPJJLVmyRJLU1dWlPXv2aOLEiWUVAAA4vHCsojeunezMmTP1/e9/X3/729+U\nTqf1k5/8xHaqGABw+DNNSxmDhU9euIZsS0uLFi9ePBq1AADqQP5esqGgX+mMycInB2zhAQCUJX/a\nU2Mk26exT9YeIQsAKEv+tKeGXMhyTdYeIQsAKEuhkw1nt2ua3OrOFiELAChL/posnaw7QhYAUJb8\ndHFjOBeyrHyyRcgCAMqSny5uiOSmi+lkbRGyAICypA6aLra4JmuLkAUAlCWVHrqFh07WHiELAChL\nfuFT/pos+2TtEbIAgLIcPF1MJ2uPkAUAlKVouphrsrYIWQBAWQqdbJjVxW4IWQBAWdKFLTxMF7sh\nZAEAZckfRtEU4TAKN4QsAKAsBxY+5aeLq1lNbSNkAQBlSecWPjXkt/CQsrYIWQBAWfKdLIdRuCNk\nAQBlOXgLDxlrj5AFAJSFwyi8I2QBAGVJZ0wF/D6FgtkIYXWxPUIWAFCWVNpQKOiX35f9PZ2sPUIW\nAFCWVMZUOOiX35dNWTpZe4QsAKAs6YyhcCggf66VJWPtEbIAgLIk06ZCQb98Pp98YrrYCSELAChL\nOmMqHMye9uT3+7ifrANCFgDgmWVZSmUMhULZ+PD5fNzqzgEhCwDwLGNYsiwpktu+4/czXeyEkAUA\neJa/zV0oP13s83F2sQNCFgDgWf60p/Dg6WI6WVuELADAs/y5xYWFTz628DghZAEAnuU72fzCJ7/f\nx2EUDghZAIBn6fx0cX7hk48tPE4IWQCAZ0XTxX6uyTohZAEAnh288MnvE/tkHRCyAADPUuncNdlc\nJ8vqYmeELADAs1Run2x48MInQtYWIQsA8OzghU8+DqNwRMgCADxjn2x5CFkAgGf5TjYUZJ+sF4Qs\nAMCzZL6TDQ06u1iErB1CFgDgWanDKNjCY89TyCYSCV100UV67LHHKl0PAKCGpYqmi7nVnRNPIfuH\nP/xBY8aMqXQtAIAal1/4FBk0Xcw1WXuuIbt161Zt2bJFF1xwwSiUAwCoZQcvfPKxT9aRa8guWLBA\n8+bNG41aAAA17sCxirlOVpJliZsE2Ag6Pfn444/rzDPP1LHHHuv5C7a3NymY2z/lpKMj6vlr1gvG\nVB8YU31gTDXK55MkTTo6ewkxEglJksaNjyrg91WtrJE0kt8nx5Bds2aNduzYoTVr1uijjz5SOBzW\nUUcdpU996lO2n9PbG3d9046OqLq6YuVXW8MYU31gTPWBMdWu/nhKkrSvd78mTGhVJnfM4u7dfQoG\n6n/DynC+T06h7Biyv/3tbwu/vuuuu3TMMcc4BiwA4PCWzhgKBf3y5Tpaf657Zbq4tPr/bwcAYNSk\n0mZhj6yUXV0scbs7O46d7GDXX399JesAANSBVMYoLHqSsmcXS+yVtUMnCwDwzDCtIQuc8tPFhGxp\nhCwAwDPDOChkC9PFhGwphCwAwDPDtArdq1TY0cPt7mwQsgAAz0zTUsA/aOGTn07WCSELAPDMsEpP\nF7OFpzRCFgDgmVk0XczCJyeELADAM7NodXHucTK2JEIWAOCJZVlFC58K08WkbEmELADAk/yUMPtk\nvSNkAQCe5FcQl+pkWV1cGiELAPDEMIs7WfbJOiNkAQCemCVClk7WGSELAPDEKDVdzDVZR4QsAMAT\np06WjC2NkAUAeFKqk/UV9smSsqUQsgAATwoLn3xck/WKkAUAeFKYLg5wdrFXhCwAwJMD08Xchccr\nQhYA4IlZYrqYfbLOCFkAgCclt/BwFx5HhCwAwJNSJz4xXeyMkAUAeOJ0djGNbGmELADAE8M0JR0c\nstmPTBeXRsgCADzJd7LBIYdRMF3shJAFAHhiWCx8KhchCwDwpOTZxdwgwBEhCwDwpOTZxblfWmY1\nKqp9hCwAwBPDYLq4XIQsAMCTfJBy03bvCFkAgCelD6PIfiRiSyNkAQCeOB1GQSdbGiELAPDE8VhF\nrsmWFKx2AQCA+pDvVt/ZvlepjKloS4Pe2tabfWzHXvn9Pl1w5jHVLLHm0MkCADzJd7K+Ult46GRL\nImQBAJ4U9sn6ip8jY0sjZAEAnuSni32+UnfhIWVLIWQBAJ4U7sLjKzVdXI2Kah8hCwDw5EAne+Cx\nfFdLxpZGyAIAPDlwTZaFT14RsgAAT/J7YX2DksOn/DXZalRU+whZAIAnhRsE0Ml6RsgCADwxSqwu\nLlyTJWNLcj3xaWBgQPPmzdOePXuUTCZ13XXX6cILLxyN2gAANcQssU+20Mmy9Kkk15BdvXq1Tjvt\nNH39619XZ2enrr76akIWAI5AhuU0XVyNimqfa8heeumlhV/v3LlTEydOrGhBAIDaVOowChY+OfN8\ng4C5c+fqo48+0uLFix1f197epGAw4Pr1OjqiXt+6bjCm+sCY6gNjqj3hcDYyoi0RRVsikqSW5uzH\nYCigaEtD3Y9RGtnvk+eQXbp0qd566y394Ac/0JNPPjnkfzKD9fbGXb9WR0dUXV0x71XWAcZUHxhT\nfWBMtal/f0qSFB9IyS9L0ZYGxQeyj6VSGcX6E3U/xuF8n5xC2XV18caNG7Vz505J0imnnCLDMNTT\n01NWAQCA+mdaDgufmC8uyTVk169fr/vuu0+S1N3drXg8rvb29ooXBgCoLaW28OQXQZlkbEmuITt3\n7lz19PToyiuv1De+8Q3ddttt8vvZXgsARxqTYxXL5npNtqGhQb/61a9GoxYAQA0rrC4u0WeRsaXR\nkgIAPCl9gwDuJ+uEkAUAeGLm7ic79Jps9iMRWxohCwDwxCh5rCKHUTghZAEAnpS+QUD2I9PFpRGy\nAABPTNMacj1W4lhFN4QsAMATw7R08A5OOllnhCwAwBPTtIqO1C1ck61GQXWAkAUAeGJYlg4+tp5O\n1hkhCwDwxDBKXJPlfrKOCFkAgCclp4tZ+OSIkAUAeGKY1pA9shLTxW4IWQCAJ6Zlye9n4VM5CFkA\ngCdGieliKdvN0smWRsgCADwxS0wXS9lulowtjZAFAHhi18n66WRtEbIAAE8M0yzZyUqSScaWRMgC\nADwptYVHyk8Xk7KlELIAAE+yZxeXXviE0ghZAIAr07JkWaUD1c/CJ1uELADAlVm4YXvpTtYkZUsi\nZAEArswSN2zP84lO1g4hCwBwZRQ62eLnOIzCHiELAHBlOHWyPh/HKtogZAEArkw62WEhZAEArgqd\nbKktPOJWd3YIWQCAK+fVxSx8skPIAgBcGVb+mmzxcz6fZHFVtiRCFgDgik52eAhZAIArwzAl2a0u\nZuGTHUIWAODKeZ8snawdQhYA4Cp/bGLJGwSITtYOIQsAcOV8GAVbeOwQsgAAV86HUWRPfKKbLUbI\nAgBcOd4ggPvJ2iJkAQCuMi6drMSUcSmELADAlfOt7rKYLi5GyAIAXDmdXZw/oMIkY4sQsgAAV253\n4ZE4WrEUQhYA4MrtWEWJa7KlELIAAFdu+2QlrsmWQsgCAFw5ThfnPpKxxYJeXrRw4UJt2LBBmUxG\n3/zmN3XxxRdXui4AQA3JmE43CGC62I5ryL700kvavHmzli1bpt7eXl122WWELAAcYQqdbIn5TxY+\n2XMN2bPPPlunn366JKm1tVUDAwMyDEOBQKDixQEAaoPziU90snZcr8kGAgE1NTVJklasWKEZM2YQ\nsABwhDEcVxdnP7LwqZina7KStGrVKq1YsUL33Xef4+va25sUDLqHcEdH1Otb1w3GVB8YU31gTLWl\nsSkiSWpqCiva0lB4PNrSoHAoGyVNjZG6HmPeSI7BU8i+8MILWrx4se69915Fo85v3tsbd/16HR1R\ndXXFvFVYJxhTfWBM9YEx1Z6+2IAkKZlIK9afkJQN2Fh/QpmMIUnq35+o6zFKw/s+OYWya8jGYjEt\nXLhQ999/v9ra2sp6YwDA4cEwvOyTHc2K6oNryD799NPq7e3VjTfeWHhswYIFmjRpUkULAwDUDufV\nxbmFT6wuLuIasnPmzNGcOXNGoxYAQI3yduLTaFZUHzjxCQDgyrQcVheLLTx2CFkAgCvDy114SNki\nhCwAwBWHUQwPIQsAcGU4LHzKd7cmC5+KELIAAFfOC5/oZO0QsgAAV2buLjylFj7lcU22GCELAHB1\noJMtfs7PFh5bhCwAwJXpeIMApovtELIAAFeeDqNg4VMRQhYA4MrTsYpkbBFCFgDgytuxiqTswQhZ\nAIArx5u2c6yiLUIWAODKdFhdTCdrj5AFALhy7GTZwmOLkAUAuHLuZPP3k8XBCFkAgCvDsuT3+Vj4\nVCZCFgDgyjQt+Uvd504sfHJCyAIAXBmGpYBdyNLJ2iJkAQCuDKdOlsMobBGyAABXpuXeyXI/2WKE\nLADAlWHah6yfTtYWIQsAcGWapu10cR7XZIsRsgAAV6ZDJ1vY1UPGFiFkAQCuMg4Ln/LTxeZoFlQn\nCFkAgCsvnSzTxcUIWQCAK8fDKFj4ZIuQBQC4clpdTCdrj5AFALhynC7mWEVbhCwAwJXziU/Zj3Sy\nxQhZAIAr07QUKHWfO3E/WSeELADAkWlasiT3hU+jWFO9IGQBAI6M3A3bWfhUPkIWAODIzIWs3186\nMlj4ZI+QBQA4opMdPkIWAODItNxClmuydghZAIAjozBdTCdbLkIWAODIMLJH/7vdT9YkY4sQsgAA\nR6ZLJ5tHJ1uMkAUAODIsb9PFXJQtRsgCABzlO9kgC5/KRsgCABy5LXzKP2wyXVyEkAUAOHK7Jsv9\nZO15CtlNmzbpoosu0oMPPljpegAANYbDKIbPNWTj8bh+9rOf6dxzzx2NegAANcZ1nyzHKtpyDdlw\nOKx77rlHEyZMGI16AAA1xqSTHbag6wuCQQWDri8raG9vUjAYcH1dR0fU89esF4ypPjCm+sCYasfO\nfQlJUrSlQdGWhiHPRVsalEobkqRAwF+3YxxsJMfgPT096u2Nu76moyOqrq7YSL91VTGm+sCY6gNj\nqi09Pdmf64lEWrFB85/RlgbF+hPK5E6ESmfMuh1j3nC+T06hzOpiAIAjFj4NHyELAHBU2MLjY+FT\nuVynizdu3KgFCxaos7NTwWBQzzzzjO666y61tbWNRn0AgCozTOcbBNDJ2nMN2dNOO00PPPDAaNQC\nAKhB7re641hFO0wXAwAcuW3hkbLdLJ1sMUIWAODIbeGTlO1mydhihCwAwJGX+8n6RCdbCiELAHCU\nv5+s63TxaBVURwhZAIAjT50s08UlEbIAAEeGwcKn4SJkAQCO3LbwSNmDKsjYYoQsAMCR6fGarEnK\nFiFkAQCODmzhsY8Mn+hkSyFkAQCOvC184ppsKYQsAMCR58MoRqugOkLIAgAc5W8Q4N7JjlZF9YOQ\nBQA48nR2sZguLoWQBQA4MlzuJysVH0bx3s4+ZQyz0qXVPEIWAOCo0MkG3I5VzL5u8wd79bM/r9cL\nr+8clfpqGSELAHDk7VZ3BzrZD7v3D/l4JCNkAQCOvJz4NHgLT28sKUnam/t4JCNkAQCOCp2sx2uy\ne/uz4drbT8gSsgAAR546WQ3uZFO5j4QsIQsAcOTtMIoD95PNh+u+/lShCz5SEbIAAEfl3k82P11s\nWpb64qmK11fLCFkAgCNPNwjI5W8qbah/IF14/EifMiZkAQCOvN3qLvtcz0GheqSvMCZkAQCOvC58\nkqQ9fQlJ0vgxDZJYYUzIAgAcGbnjEZ062fyRiz37siF7/KRWSUwXE7IAAEf5hU8O22QLz+U72alH\nZ0OW6WIAABwYlqWA31e47lpSvpPty4ZqPmSZLgYAwIFpWo5TxZLkP6iT7WhrVHNDkOniahcAAKht\nhmk5LnqSDix86okl5ff5NKY5rPZopLBn9khFyAIAHHnpZAtbePoSGtMSlt/vU1s0ooGkoUQqMxpl\n1iRCFgDgyFMnm3s6nTHV1hKRJLXnPh7JU8aELADAkbeQPfB8ezQy5OPgFcbxRFrxxJHT2RKyAABH\n3qaLD/w638G25UI2v8LYsizd/uA/9atlr1Sm0BpEyAIAhkhnTD354nvqya0UNkyrcNiEncGdbFs0\nLKl4uvijnrg+7N6v93bGjpgpZEIWADDEun/t0uMvvKe//N/3JeU62YBzXAyO4OLp4uydeN7a1lt4\nzb/e7xm5gmsYIQsAGOLVLd2SpNe2dMuyLBkjNF381vsHQnZw4B7OgtUuAABQO9IZQ2++l+0y9/an\ntG1XTGbZ08XZcI02hhQM+NQbS8g0Lb21rVfjWiNKZUz96/0eWZblfIrUYYBOFgBQ8Na2vUqmDU0c\n2yRJenVzd/mdbC5kfT6f2loi6o0ltW1XTPFkRqceN1anTGnX3v6UPuqJFz5nb39Sz67foUzuZgSH\nC0IWAFCQnyqeO/PjCgZ8hZB1P/Ep+3xjJKCG8IFJ0rZoRPv2p7Qx1x2fcly7Tj1urCTpX4Omjx94\n5h09vGqz1rzSOaLjqTZCFgAgKbvF5rUt3WpuCOq048fq5Mnt2r67XxnD9NzJ5g+iyGtviciysoup\nJOmUKWN16pR2SQcWP23fFdMrm7Ph/vRL25TOGEV11StCFgAgSdq+q1+9saROP2GcAn6/zjxxfOE5\nr8cq5qeK8/K//7B7vz7W0awxzWGNb2tUR1uD3t6+V6Zp6S8vvi9J+vgxY7S3P6UXXt9Z+Pyde/br\nf/5hrf5z1aa6DFtPIXv77bdrzpw5mjt3rl5//fVK1wQAGAWptFHYCytJr2zukiSdeWJH9uPHD4Ss\n23Rx/un2gzrZwZ3tKVPGDvn1QDKjf7yxUxs2dWnq0a36H//xCYWDfj21dpvSGVPxREZ3PfqG9vQl\ntGr9B1o9aCrZsiyt/ucHemjlJg0ka/cEKdfVxS+//LK2bdumZcuWaevWrbr55pu1bNmy0ahtiIxh\nav9AWmMO+gYCAMr39rZe3ff0W9rTl9C/nTNZl503Va9u6VbA79NpU7NhOLa1QZMntGj77n7XTjY/\nX9xm08lK0qnHtQ/59d9f+1APPbtJkvSFzxynMc1hXfDJY7Ty/+3QC69/qI3v9uijnrg+/Ymj9PrW\nPXp41WYdM75Zx08aoweeeUf/eCPb8b6zo1ffufx0jW9rVCpt6K/rtmv1q50699Sj9O+fPk6NkWzU\npTOGNrzTpQntTTp+Uuuh/QF65Bqya9eu1UUXXSRJOuGEE7Rv3z719/erpaWl4sWZpqV3tvdq3Vu7\nteGd3dqfyOjYCS0655QJOm3qOG3fHdOb7/Vo8wf7NH5Mg6ZNHatpx42VYVra2rlPWzr3KZ0xdfyk\nVn38mDEaN6ZBH3bHtWN3TF17E5rQ3qjJE1o0qaNZ/QNp7eyO66OeuMJBv44a16SjxzUrHPRr994B\nde0dUDyR0fgxDepob1Rbc0R98ZT27Etob39SR02IKiRLY1sblDFM7e1PaW8sKdOy1B6NqK0looZw\nQLF4Wvv2JxVPZNTSGFJrc1jRprASqYxi8bRi8bSCAZ+iTWG1NoXk8/nUP5BW/0Ba6YyplsaQmhuD\naggHlUhltH8go3gyo0jIr+bGkJobQjIMU/sT2cclqSkSVHNDUIGAXwPJjAaSGaUyphrCATVGgmoI\nB5TKmErkngsG/GqIBOULBdXTl1AiZSiRMmSYphrCQTWGAwoF/Uqms4+n0qZCQb8aIgE1hAIyTEuJ\nlKFkOntdpSEcUCQUUMDvUyJtKJkylDZMRULZ14dCAaUz2cdTmey1n0jucyxLSuY+x7Ss7OeEAwoE\n/Eqls++RzpgKBfwK5z7HMEwl09m6fD4pEgooHArI75MyPr925q4xhUMBRUJ+hYMBpTOmUpns5wQC\nvuxzQb9MS0rlapOkcCigcNCvQMCvdNpQMmMqk8mOPxzyKxTMvn8qYyqVNuT3+RQO+RUOBeSTCnUZ\npqVwyK9IKKBgwF9473Rm0PuHAjJNqzDO/PtHQgH5/b7C4/sShvr7E4oE/QqFAspkcuPPZN8/P36f\nT0rmvi+Gmf2zjIQCCgazY0nk/iyDAb8iudoM0yp8n22/l5ns9zISzr5POvf6RNpQwO9TQzi7EMay\nsn8vBlIZWaYUCQfUGM6+fzL3eDJlKBwKKCWf4v1JpTOmBpIZxRMZ+XxSYySopoagAn5/9gzc3N/l\nxnD273gkHFAiZWh/7nzccNCvpoaQmhuCMk1L/YnsvyXTlJobg2ppDCkcDKh/IK1YPKV4IqPGhqBa\nm8JqaQwV/l327U8p4Pdl/702h+X3Sfv6U9q3P6VkylC0OaS25oiaG0OKxVPqjSXVtz+lxkhQbS0R\ntUXDSlrSlm096u078HNhbDSiSDig3lhSe/oSisXTGtMc1rjWBrW1RLQ/kVZPLKnevoSCQb/GtTZo\nXGuDAgGf9u1Pqa8/pWTGUGtTWGNyP0vyHadpWurbn1Jvf1J7Y0mFgn61RyNqbQ7ruQ2d+ts/P5DP\nJ41pDuuv67brlU1d2tU7oGlTxxZCSZLOPHG8tu/u93yrO7vp4oDfp5OObSs8fnLuumw6Y+q4o6L6\nxPHjJEmzpk/W6lc69fCqzTJMS9OOa9fXZp2sLR/s0y+XvqrfP75RE9ubtKVzn447KqopR0X1/Ksf\n6qd/Xq/PnTtFqzZ8oO59Cfkk/fXl7Vr75kf6jxnHa09fQqtf6VQsntbpJ4zTjbPPKCeOhs01ZLu7\nuzVt2rTC78eOHauurq5RCdmHnt1UmB4Y0xzWqce1653te/Xo8+/q0effLbwu2hTSls592vzBPj3+\nwntFXye/qg0AkHX0uCZd899O1THjm7Xi+a3624YPJA2dIpayIfvki+8r6HLiU77TtQvZqUe3Dgnv\n1qZwoUv+/GemFq7pjmmJ6PwzJ2nV+g/U0dagb37hNAX8fv2Xye264qIT9eDKTYrF9+mcUybov196\niiKhgCZPjOo/n92kpc9tUcDv0yXnTNYl0ydrzSudeuqlbfrT/3lbUrbhuPS/TtHFZx97CH9y5fFZ\nLleSb731Vp1//vmFbvaKK67Q7bffrqlTp45KgQAA1CvXhU8TJkxQd3d34fe7d+9WR0dHRYsCAOBw\n4Bqyn/70p/XMM89Ikt58801NmDBhVKaKAQCod67XZM866yxNmzZNc+fOlc/n049//OPRqAsAgLrn\nek0WAAAMDyc+AQBQIYQsAAAVMqoh+/LLL+vcc8/V6tWri55bunSpZs6cOZrljIhSY3r77bd15ZVX\n6stf/rKuu+46DQwMVLHC8tmNae7cuZo7d25dX5fftWuXrrnmGn3lK1/Rl770JW3cuLHaJY2IJUuW\n6Atf+IIuv/zyw+ro0+7ubp199tlat25dtUs5ZJlMRjfddJOuuOIKffGLX9T69eurXdKwHY5H7S5c\nuFBz5szR5ZdfrpUrV47Y1x21kN2+fbv+9Kc/6ayzzip6bs+ePXr22WdHq5QRYzemn//855o3b54e\nfPBBTZkyRY899liVKiyf3Zjmz5+vm2++WUuXLlV/f7+ef/75KlV4aO6//3599rOf1QMPPKDvfe97\n+s1vflPtkg7Z5s2b9dRTT+nRRx/VT3/6U61Zs6baJY2YhQsX6thjR+/ggEp64okn1NjYqIcffljz\n58/XHXfcUe2ShmXwUbvz58/X/Pnzq13SIXvppZe0efNmLVu2TPfee69uv/32EfvaoxayHR0duvvu\nuxWNRoue+8UvfqHvfOc7o1XKiLEb0+LFi3X66adLyp6QtXfv3mqUNyylxpRKpdTZ2VkY04UXXqi1\na9dWq8RD0t7eXvh+9PX1qb293eUzat/q1as1a9YsBYNBTZs2rS7/LZWydu1aNTc366STTqp2KSPi\n85//vH74wx9Kqr+fC4PZHbVyDTRVAAAD90lEQVRbz84++2zdeeedkqTW1lYNDAzIMAyXz/Jm1EK2\nsbFRgUCg6PF169YpEonojDNG5xzJkWQ3pvw+4ng8rieeeEKXXHLJaJc2bKXG1Nvbq9bWA4dpjxs3\nTl1dXaNd2oj42te+pqefflqXXHKJfvSjH+mGG26odkmHrLOzUzt37tQ111yjq666Sm+//Xa1Szpk\nqVRKv/vd7/Td73632qWMmFAopEgke8Tgn//8Z33uc5+rckXD093dPeQ/p/mjdutZIBBQU1OTJGnF\nihWaMWNGyZ/tw+G6T3Y4HnnkET3yyCNDHrv++ut13nnnDXkslUpp0aJF+v3vf1+JMkaU1zHlxeNx\nXXvttbr66qt1wgknjEaJZSt3THn1suur1PhmzJihWbNm6dprr9Xq1au1YMEC3X333VWqsHylxtTd\n3a3zzjtP9957rzZs2KBbbrlFjz76aJUqLJ/d92n27NlD/nNXT5z+bT300EN68803tXjx4ipVN7Lq\n5eeBF6tWrdKKFSt03333jdwXtUbZTTfdZD333HOWZVnWq6++al188cXW7NmzrdmzZ1vTpk2zbrzx\nxtEu6ZANHpNlWVY6nbauuuoqa/ny5VWs6tAMHlMqlbLOP//8wnOPPfaYdccdd1SpskNzzTXXWG+8\n8YZlWZaVTCaHjKte3XnnndZf/vKXwu+nT59exWpGxpw5cwo/F6ZPn27NmjXL2rRpU7XLOmTLly+3\nrr76aiuRSFS7lGFbtGiR9fDDDxd+P3PmTCsWi1WxopHx97//3br88sut3t7eEf26Vd3Cc8YZZ+iZ\nZ57R8uXLtXz5ck2YMOGwWIhyzz336JxzztHs2bOrXcqICIVCOv744wurIVeuXOna7daqKVOm6LXX\nXpMkvf7665oyZUqVKzp0M2bM0D/+8Q9J0tatW3X00UdXuaJDt3Tp0sLPhQsuuEA//vGPdeKJJ1a7\nrEOyY8cOLV26VHfffXdh2rgeHY5H7cZiMS1cuFB//OMf1dbW5v4JZRi1E5/WrFmjJUuW6N1339XY\nsWPV0dFR1JLPnDlTzz333GiUMyLsxvSZz3xGH/vYxxQKhSRJ06dP17e//e0qV+uN3Zi2bNmi2267\nTaZp6owzzigs4Kg3u3fv1i233KJEIiFJuuWWW3TyySdXuapDt2jRIr344ouSpHnz5umTn/xklSsa\nOfPmzdNll12m6dOnV7uUQ/LrX/9aTz31lCZNmlR4bMmSJQqHw1Wsanh++ctfav369YWjduv939Cy\nZct01113Dbm73IIFC4Z8r4aLYxUBAKgQTnwCAKBCCFkAACqEkAUAoEIIWQAAKoSQBQCgQghZAAAq\nhJAFAKBCCFkAACrk/wP4TmMfOclhnwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f72946ad0f0>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "import seaborn as ssb\n",
    "noise = np.random.normal(loc=0.1,scale=0.1,size=100)\n",
    "noise = noise.reshape(100,1)\n",
    "noise_counts = w[0:100] + noise\n",
    "l1 = LogisticRegression(C=7,penalty='l1')\n",
    "l1.fit(w[0:100],y_train[0:100])\n",
    "l2 = LogisticRegression(C=7,penalty='l1')\n",
    "l2.fit(noise_counts,y_train[0:100])\n",
    "weights = np.array(l1.coef_)\n",
    "weight_noise = np.array(l2.coef_)\n",
    "weight = np.subtract(weights,weight_noise)\n",
    "ssb.distplot(weight)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SjwPImq56gs_"
   },
   "source": [
    "**since the weights did not differ much i.e almost all the values are 0 we can use these features for modeling**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "id": "JVGKo6QrK15q",
    "outputId": "f3bdcdcf-e633-4813-92c4-b36133821681"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on test set:  89.60000000000001 %\n",
      "confusion matrix\n",
      "[[ 53  46]\n",
      " [ 32 619]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "model = LogisticRegression(C=7,penalty='l1')\n",
    "model.fit(w,y_train)\n",
    "pred = model.predict(w2)\n",
    "acc = accuracy_score(y_test,pred)\n",
    "confusion = confusion_matrix(y_test,pred)\n",
    "print('Accuracy on test set: ',acc*100,\"%\")\n",
    "print('confusion matrix')\n",
    "print(confusion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "0V_pUl58QIkP",
    "outputId": "ddadfcd4-f096-4a17-db4d-ce263cec90b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training set 97.2 %\n"
     ]
    }
   ],
   "source": [
    "model = LogisticRegression(C=7,penalty='l1')\n",
    "model.fit(w,y_train)\n",
    "pred = model.predict(w)\n",
    "acc = accuracy_score(y_train,pred)\n",
    "print('Accuracy on training set',acc*100,'%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 233
    },
    "colab_type": "code",
    "id": "RqPU3GeYQXnh",
    "outputId": "6caf50a8-c748-4174-bb07-d1b5e994764a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Top 10 Feat \n",
      "            words       coefficients\n",
      "29298        home  9.683420949353382\n",
      "10424       chick  9.673354348199254\n",
      "69572  youll find  9.446962758281025\n",
      "29656  hot chocol  9.431074957993324\n",
      "16249       didnt  9.332920914441534\n",
      "32949       label  9.293406480505263\n",
      "60309   tast like     9.287782308518\n",
      "69174       write  9.218771142954218\n",
      "68841       would   9.19104504377822\n",
      "67391        well  9.116858827505174\n"
     ]
    }
   ],
   "source": [
    "feat = tfidf_vect.get_feature_names()\n",
    "feat = np.array(feat).reshape(1,69808)\n",
    "features = np.vstack((feat,np.absolute(model.coef_)))\n",
    "features= pd.DataFrame(data=features.T,columns=('words','coefficients'))\n",
    "top_feat= features.sort_values('coefficients',axis=0,inplace=False,ascending=False,kind='quicksort')\n",
    "print('  Top 10 Feat ')\n",
    "print(top_feat[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h3kpYFDemCZn"
   },
   "source": [
    "Summary\n",
    "======\n",
    "\n",
    "\n",
    "1.  Using tfidf featurization  to get text vectors\n",
    "\n",
    "2.  Obtaining optimal parameters using hyperparameter tuning using grid search and random search .\n",
    "\n",
    "3.  Obtaining best penality (L1,L2) \n",
    "2.  Observing sparcity in  L1 and L2 regularization.\n",
    "\n",
    "\n",
    "5.   Checking collinearity in features ,since featues are  not collinear we can use these features for modeling .\n",
    "2. Obtaining  various performance measures of the model.\n",
    "\n",
    "7.     Accuracy on training set is 97 and on test set is  89.\n",
    "\n",
    "9.    Obtaining top most important features .\n",
    "\n",
    "  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UsqHBh45qnvv"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "Amazon_Grid_search_Random_search_final.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}