Naïve Bayes Text Classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "psychological-virginia",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Text classification with Naïve Bayes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "internal-potato",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import f1_score, accuracy_score\n",
    "import re\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, HashingVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "revolutionary-decision",
   "metadata": {},
   "source": [
    "### The `sklearn` news dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "entire-hypothesis",
   "metadata": {},
   "outputs": [],
   "source": [
    "news = datasets.fetch_20newsgroups(subset=\"all\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "moderate-oriental",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".. _20newsgroups_dataset:\n",
      "\n",
      "The 20 newsgroups text dataset\n",
      "------------------------------\n",
      "\n",
      "The 20 newsgroups dataset comprises around 18000 newsgroups posts on\n",
      "20 topics split in two subsets: one for training (or development)\n",
      "and the other one for testing (or for performance evaluation). The split\n",
      "between the train and test set is based upon a messages posted before\n",
      "and after a specific date.\n",
      "\n",
      "This module contains two loaders. The first one,\n",
      ":func:`sklearn.datasets.fetch_20newsgroups`,\n",
      "returns a list of the raw texts that can be fed to text feature\n",
      "extractors such as :class:`~sklearn.feature_extraction.text.CountVectorizer`\n",
      "with custom parameters so as to extract feature vectors.\n",
      "The second one, :func:`sklearn.datasets.fetch_20newsgroups_vectorized`,\n",
      "returns ready-to-use features, i.e., it is not necessary to use a feature\n",
      "extractor.\n",
      "\n",
      "**Data Set Characteristics:**\n",
      "\n",
      "    =================   ==========\n",
      "    Classes                     20\n",
      "    Samples total            18846\n",
      "    Dimensionality               1\n",
      "    Features                  text\n",
      "    =================   ==========\n",
      "\n",
      "Usage\n",
      "~~~~~\n",
      "\n",
      "The :func:`sklearn.datasets.fetch_20newsgroups` function is a data\n",
      "fetching / caching functions that downloads the data archive from\n",
      "the original `20 newsgroups website`_, extracts the archive contents\n",
      "in the ``~/scikit_learn_data/20news_home`` folder and calls the\n",
      ":func:`sklearn.datasets.load_files` on either the training or\n",
      "testing set folder, or both of them::\n",
      "\n",
      "  >>> from sklearn.datasets import fetch_20newsgroups\n",
      "  >>> newsgroups_train = fetch_20newsgroups(subset='train')\n",
      "\n",
      "  >>> from pprint import pprint\n",
      "  >>> pprint(list(newsgroups_train.target_names))\n",
      "  ['alt.atheism',\n",
      "   'comp.graphics',\n",
      "   'comp.os.ms-windows.misc',\n",
      "   'comp.sys.ibm.pc.hardware',\n",
      "   'comp.sys.mac.hardware',\n",
      "   'comp.windows.x',\n",
      "   'misc.forsale',\n",
      "   'rec.autos',\n",
      "   'rec.motorcycles',\n",
      "   'rec.sport.baseball',\n",
      "   'rec.sport.hockey',\n",
      "   'sci.crypt',\n",
      "   'sci.electronics',\n",
      "   'sci.med',\n",
      "   'sci.space',\n",
      "   'soc.religion.christian',\n",
      "   'talk.politics.guns',\n",
      "   'talk.politics.mideast',\n",
      "   'talk.politics.misc',\n",
      "   'talk.religion.misc']\n",
      "\n",
      "The real data lies in the ``filenames`` and ``target`` attributes. The target\n",
      "attribute is the integer index of the category::\n",
      "\n",
      "  >>> newsgroups_train.filenames.shape\n",
      "  (11314,)\n",
      "  >>> newsgroups_train.target.shape\n",
      "  (11314,)\n",
      "  >>> newsgroups_train.target[:10]\n",
      "  array([ 7,  4,  4,  1, 14, 16, 13,  3,  2,  4])\n",
      "\n",
      "It is possible to load only a sub-selection of the categories by passing the\n",
      "list of the categories to load to the\n",
      ":func:`sklearn.datasets.fetch_20newsgroups` function::\n",
      "\n",
      "  >>> cats = ['alt.atheism', 'sci.space']\n",
      "  >>> newsgroups_train = fetch_20newsgroups(subset='train', categories=cats)\n",
      "\n",
      "  >>> list(newsgroups_train.target_names)\n",
      "  ['alt.atheism', 'sci.space']\n",
      "  >>> newsgroups_train.filenames.shape\n",
      "  (1073,)\n",
      "  >>> newsgroups_train.target.shape\n",
      "  (1073,)\n",
      "  >>> newsgroups_train.target[:10]\n",
      "  array([0, 1, 1, 1, 0, 1, 1, 0, 0, 0])\n",
      "\n",
      "Converting text to vectors\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "\n",
      "In order to feed predictive or clustering models with the text data,\n",
      "one first need to turn the text into vectors of numerical values suitable\n",
      "for statistical analysis. This can be achieved with the utilities of the\n",
      "``sklearn.feature_extraction.text`` as demonstrated in the following\n",
      "example that extract `TF-IDF`_ vectors of unigram tokens\n",
      "from a subset of 20news::\n",
      "\n",
      "  >>> from sklearn.feature_extraction.text import TfidfVectorizer\n",
      "  >>> categories = ['alt.atheism', 'talk.religion.misc',\n",
      "  ...               'comp.graphics', 'sci.space']\n",
      "  >>> newsgroups_train = fetch_20newsgroups(subset='train',\n",
      "  ...                                       categories=categories)\n",
      "  >>> vectorizer = TfidfVectorizer()\n",
      "  >>> vectors = vectorizer.fit_transform(newsgroups_train.data)\n",
      "  >>> vectors.shape\n",
      "  (2034, 34118)\n",
      "\n",
      "The extracted TF-IDF vectors are very sparse, with an average of 159 non-zero\n",
      "components by sample in a more than 30000-dimensional space\n",
      "(less than .5% non-zero features)::\n",
      "\n",
      "  >>> vectors.nnz / float(vectors.shape[0])\n",
      "  159.01327...\n",
      "\n",
      ":func:`sklearn.datasets.fetch_20newsgroups_vectorized` is a function which \n",
      "returns ready-to-use token counts features instead of file names.\n",
      "\n",
      ".. _`20 newsgroups website`: http://people.csail.mit.edu/jrennie/20Newsgroups/\n",
      ".. _`TF-IDF`: https://en.wikipedia.org/wiki/Tf-idf\n",
      "\n",
      "\n",
      "Filtering text for more realistic training\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "\n",
      "It is easy for a classifier to overfit on particular things that appear in the\n",
      "20 Newsgroups data, such as newsgroup headers. Many classifiers achieve very\n",
      "high F-scores, but their results would not generalize to other documents that\n",
      "aren't from this window of time.\n",
      "\n",
      "For example, let's look at the results of a multinomial Naive Bayes classifier,\n",
      "which is fast to train and achieves a decent F-score::\n",
      "\n",
      "  >>> from sklearn.naive_bayes import MultinomialNB\n",
      "  >>> from sklearn import metrics\n",
      "  >>> newsgroups_test = fetch_20newsgroups(subset='test',\n",
      "  ...                                      categories=categories)\n",
      "  >>> vectors_test = vectorizer.transform(newsgroups_test.data)\n",
      "  >>> clf = MultinomialNB(alpha=.01)\n",
      "  >>> clf.fit(vectors, newsgroups_train.target)\n",
      "  MultinomialNB(alpha=0.01, class_prior=None, fit_prior=True)\n",
      "\n",
      "  >>> pred = clf.predict(vectors_test)\n",
      "  >>> metrics.f1_score(newsgroups_test.target, pred, average='macro')\n",
      "  0.88213...\n",
      "\n",
      "(The example :ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py` shuffles\n",
      "the training and test data, instead of segmenting by time, and in that case\n",
      "multinomial Naive Bayes gets a much higher F-score of 0.88. Are you suspicious\n",
      "yet of what's going on inside this classifier?)\n",
      "\n",
      "Let's take a look at what the most informative features are:\n",
      "\n",
      "  >>> import numpy as np\n",
      "  >>> def show_top10(classifier, vectorizer, categories):\n",
      "  ...     feature_names = np.asarray(vectorizer.get_feature_names())\n",
      "  ...     for i, category in enumerate(categories):\n",
      "  ...         top10 = np.argsort(classifier.coef_[i])[-10:]\n",
      "  ...         print(\"%s: %s\" % (category, \" \".join(feature_names[top10])))\n",
      "  ...\n",
      "  >>> show_top10(clf, vectorizer, newsgroups_train.target_names)\n",
      "  alt.atheism: edu it and in you that is of to the\n",
      "  comp.graphics: edu in graphics it is for and of to the\n",
      "  sci.space: edu it that is in and space to of the\n",
      "  talk.religion.misc: not it you in is that and to of the\n",
      "\n",
      "\n",
      "You can now see many things that these features have overfit to:\n",
      "\n",
      "- Almost every group is distinguished by whether headers such as\n",
      "  ``NNTP-Posting-Host:`` and ``Distribution:`` appear more or less often.\n",
      "- Another significant feature involves whether the sender is affiliated with\n",
      "  a university, as indicated either by their headers or their signature.\n",
      "- The word \"article\" is a significant feature, based on how often people quote\n",
      "  previous posts like this: \"In article [article ID], [name] <[e-mail address]>\n",
      "  wrote:\"\n",
      "- Other features match the names and e-mail addresses of particular people who\n",
      "  were posting at the time.\n",
      "\n",
      "With such an abundance of clues that distinguish newsgroups, the classifiers\n",
      "barely have to identify topics from text at all, and they all perform at the\n",
      "same high level.\n",
      "\n",
      "For this reason, the functions that load 20 Newsgroups data provide a\n",
      "parameter called **remove**, telling it what kinds of information to strip out\n",
      "of each file. **remove** should be a tuple containing any subset of\n",
      "``('headers', 'footers', 'quotes')``, telling it to remove headers, signature\n",
      "blocks, and quotation blocks respectively.\n",
      "\n",
      "  >>> newsgroups_test = fetch_20newsgroups(subset='test',\n",
      "  ...                                      remove=('headers', 'footers', 'quotes'),\n",
      "  ...                                      categories=categories)\n",
      "  >>> vectors_test = vectorizer.transform(newsgroups_test.data)\n",
      "  >>> pred = clf.predict(vectors_test)\n",
      "  >>> metrics.f1_score(pred, newsgroups_test.target, average='macro')\n",
      "  0.77310...\n",
      "\n",
      "This classifier lost over a lot of its F-score, just because we removed\n",
      "metadata that has little to do with topic classification.\n",
      "It loses even more if we also strip this metadata from the training data:\n",
      "\n",
      "  >>> newsgroups_train = fetch_20newsgroups(subset='train',\n",
      "  ...                                       remove=('headers', 'footers', 'quotes'),\n",
      "  ...                                       categories=categories)\n",
      "  >>> vectors = vectorizer.fit_transform(newsgroups_train.data)\n",
      "  >>> clf = MultinomialNB(alpha=.01)\n",
      "  >>> clf.fit(vectors, newsgroups_train.target)\n",
      "  MultinomialNB(alpha=0.01, class_prior=None, fit_prior=True)\n",
      "\n",
      "  >>> vectors_test = vectorizer.transform(newsgroups_test.data)\n",
      "  >>> pred = clf.predict(vectors_test)\n",
      "  >>> metrics.f1_score(newsgroups_test.target, pred, average='macro')\n",
      "  0.76995...\n",
      "\n",
      "Some other classifiers cope better with this harder version of the task. Try\n",
      "running :ref:`sphx_glr_auto_examples_model_selection_grid_search_text_feature_extraction.py` with and without\n",
      "the ``--filter`` option to compare the results.\n",
      "\n",
      ".. topic:: Recommendation\n",
      "\n",
      "  When evaluating text classifiers on the 20 Newsgroups data, you\n",
      "  should strip newsgroup-related metadata. In scikit-learn, you can do this by\n",
      "  setting ``remove=('headers', 'footers', 'quotes')``. The F-score will be\n",
      "  lower because it is more realistic.\n",
      "\n",
      ".. topic:: Examples\n",
      "\n",
      "   * :ref:`sphx_glr_auto_examples_model_selection_grid_search_text_feature_extraction.py`\n",
      "\n",
      "   * :ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py`\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(news.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "subjective-isolation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"From: Mamatha Devineni Ratnam <mr47+@andrew.cmu.edu>\\nSubject: Pens fans reactions\\nOrganization: Post Office, Carnegie Mellon, Pittsburgh, PA\\nLines: 12\\nNNTP-Posting-Host: po4.andrew.cmu.edu\\n\\n\\n\\nI am sure some bashers of Pens fans are pretty confused about the lack\\nof any kind of posts about the recent Pens massacre of the Devils. Actually,\\nI am  bit puzzled too and a bit relieved. However, I am going to put an end\\nto non-PIttsburghers' relief with a bit of praise for the Pens. Man, they\\nare killing those Devils worse than I thought. Jagr just showed you why\\nhe is much better than his regular season stats. He is also a lot\\nfo fun to watch in the playoffs. Bowman should let JAgr have a lot of\\nfun in the next couple of games since the Pens are going to beat the pulp out of Jersey anyway. I was very disappointed not to see the Islanders lose the final\\nregular season game.          PENS RULE!!!\\n\\n\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "portable-aging",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10,  3, 17, ...,  3,  1,  7])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "anticipated-terrorism",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism',\n",
       " 'comp.graphics',\n",
       " 'comp.os.ms-windows.misc',\n",
       " 'comp.sys.ibm.pc.hardware',\n",
       " 'comp.sys.mac.hardware',\n",
       " 'comp.windows.x',\n",
       " 'misc.forsale',\n",
       " 'rec.autos',\n",
       " 'rec.motorcycles',\n",
       " 'rec.sport.baseball',\n",
       " 'rec.sport.hockey',\n",
       " 'sci.crypt',\n",
       " 'sci.electronics',\n",
       " 'sci.med',\n",
       " 'sci.space',\n",
       " 'soc.religion.christian',\n",
       " 'talk.politics.guns',\n",
       " 'talk.politics.mideast',\n",
       " 'talk.politics.misc',\n",
       " 'talk.religion.misc']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "news.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "considered-techno",
   "metadata": {},
   "source": [
    "### Split data into testing and training\n",
    "* We will use `75%` of the data for training and the rest for testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "respective-nowhere",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(18846, 18846)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = news.data, news.target\n",
    "len(X),len(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "respective-nashville",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.25, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "valuable-religious",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14134, 4712, 4712, 14134)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_train), len(X_test),len( y_test), len(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coral-smoke",
   "metadata": {},
   "source": [
    "### Bag of words\n",
    "* Machine learning algo works on numerical data so we want to convert the text to numeric\n",
    "* From `sklearn.feature_extraction.text` we have three classes that transforms text into numeric which are:\n",
    "    1. **`CountVectorizer`** -sically creates a dictionary of words from the text corpus. Then, each instance is converted to a vector of numeric features where each element will be the count of the number of times a particular word appears in the document.\n",
    "    2. **`HashingVectorizer`** -instead of constricting and maintaining the dictionary in memory, implements a hashing function that maps tokens into feature indexes, and then computes the count as in CountVectorizer\n",
    "    3. **`TfidfVectorizer`** - works like the CountVectorizer, but with a more advanced calculation called **Term Frequency Inverse Document Frequency (TF-IDF)**. This is a statistic for measuring the importance of a word in a document or corpus. Intuitively, it looks for words that are more frequent in the current document, compared with their frequency in the whole corpus of documents. You can see this as a way to normalize the results and a."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "arctic-coaching",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "silent-dallas",
   "metadata": {},
   "source": [
    "> We are going to create 3 classifiers with three different vectorizers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fuzzy-satisfaction",
   "metadata": {},
   "source": [
    "#### `CountVectorizer()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "sunrise-stanley",
   "metadata": {},
   "outputs": [],
   "source": [
    "countvectorizer_pipe = Pipeline([\n",
    "    (\"vect\", CountVectorizer() ),\n",
    "    (\"clf\", MultinomialNB())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conservative-insert",
   "metadata": {},
   "source": [
    "#### `TfidfVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "expanded-tunisia",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_pipe = Pipeline([\n",
    "    (\"vect\", TfidfVectorizer()),\n",
    "    (\"clf\", MultinomialNB())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "straight-clinton",
   "metadata": {},
   "source": [
    "#### `HashingVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "portable-marks",
   "metadata": {},
   "outputs": [],
   "source": [
    "hashingvectorizer_pipe = Pipeline([\n",
    "    (\"vect\", HashingVectorizer(alternate_sign=False) ),\n",
    "    (\"clf\", MultinomialNB())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "addressed-jason",
   "metadata": {},
   "source": [
    "#### Comparing the vectorising methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pleasant-signal",
   "metadata": {},
   "source": [
    "#### `countvectorizer_pipe`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "challenging-limitation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('vect', CountVectorizer(stop_words='english')),\n",
       "                ('clf', MultinomialNB())])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countvectorizer_pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "academic-springfield",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([4]), 4)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countvectorizer_pipe.predict([X_test[10]]), y_test[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "still-brunei",
   "metadata": {},
   "source": [
    "#### scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "statewide-modem",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9533040894297439"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countvectorizer_pipe.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reverse-theology",
   "metadata": {},
   "source": [
    "> The count vectoctorizer is `93%` accurate on the train dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "reduced-vampire",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8771222410865874"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countvectorizer_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "short-concentration",
   "metadata": {},
   "source": [
    "> The count vectorizer is `86%` accurate on the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "undefined-disorder",
   "metadata": {},
   "source": [
    "#### `tfidf_pipe`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "solid-village",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('vect', TfidfVectorizer()), ('clf', MultinomialNB())])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "looking-briefs",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([4]), 4)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_pipe.predict([X_test[10]]), y_test[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "desperate-duration",
   "metadata": {},
   "source": [
    "#### Scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "interim-variation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9231640016980331"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_pipe.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suitable-technique",
   "metadata": {},
   "source": [
    "> The tfdif is `92%` accurate on the train dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "logical-resident",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8752122241086587"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "korean-software",
   "metadata": {},
   "source": [
    "> The tfdif is `84%` accurate on the test dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appointed-funds",
   "metadata": {},
   "source": [
    "#### `hashingvectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "serial-lawrence",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('vect', HashingVectorizer(alternate_sign=False)),\n",
       "                ('clf', MultinomialNB())])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hashingvectorizer_pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "apart-medium",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([4]), 4)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hashingvectorizer_pipe.predict([X_test[10]]), y_test[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "honey-howard",
   "metadata": {},
   "source": [
    "#### Scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "catholic-mercy",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8243950757039762"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hashingvectorizer_pipe.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "comic-sacrifice",
   "metadata": {},
   "source": [
    "> The hashingvectorizer yeilds `82%` accuracy on the train dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "final-midwest",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7521222410865874"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hashingvectorizer_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "terminal-soldier",
   "metadata": {},
   "source": [
    "> The hashingvectorizer yeilds `75%` accuracy on the test dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "precise-behalf",
   "metadata": {},
   "source": [
    "### Imporving our models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "understanding-lingerie",
   "metadata": {},
   "outputs": [],
   "source": [
    "> We can remove stopword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "particular-listening",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('vect',\n",
       "                 CountVectorizer(binary=True, ngram_range=(1, 2),\n",
       "                                 stop_words='english')),\n",
       "                ('clf', MultinomialNB())])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countvectorizer_pipe = Pipeline([\n",
    "    (\"vect\", CountVectorizer(stop_words='english', ngram_range=(1, 2), binary=True ) ),\n",
    "    (\"clf\", MultinomialNB())\n",
    "])\n",
    "countvectorizer_pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "generic-burton",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9941983868685439, 0.907258064516129)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countvectorizer_pipe.score(X_train, y_train), countvectorizer_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "false-causing",
   "metadata": {},
   "source": [
    "> The `countvectorizer` has increased to `99%` accuracy on train datasets and `91%` accuracy on test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "daily-memphis",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9721239564171501, 0.8788200339558574)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_pipe = Pipeline([\n",
    "    (\"vect\", TfidfVectorizer(max_df=0.5, stop_words='english', use_idf=True, ngram_range=(1, 2)) ),\n",
    "    (\"clf\", MultinomialNB())\n",
    "])\n",
    "tfidf_pipe.fit(X_train, y_train)\n",
    "tfidf_pipe.score(X_train, y_train), tfidf_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "imperial-techno",
   "metadata": {},
   "source": [
    "> The `tfidf_pipe` has increased to `97%` on the train and `88%` on the test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "bibliographic-clock",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([14]), 14)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_pipe.predict([X_train[19]]), y_train[19]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "worst-quilt",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}