FakeNewsDetection-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FakeNewsDetection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In this notebook, we will be solving a use case of 'Fake News Detection' - Predict if a news published in an article is 'Real' or 'Fake' for a sample data using ML Algorithms!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Update variables under **Variables** section if required before running the notebook. To run notebook cell by cell, click on a cell and click **Run** button below the Menu bar. Or to run all cells, select **Cell --> Run** All from Menu bar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Library imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\SONU\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\SONU\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "#Import necessary libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import nltk #Import NLTK ---> Natural Language Toolkit\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize, RegexpTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import PorterStemmer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample dataset we use is **fakedata.csv** which is available in the current directory. This can be replaced with any dataset containing a text column with 'News' and a label column citing is the news is 'Real' or 'Fake'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attributes of **fakedata.csv** dataset:\n",
    "   1. **text**: text of the article\n",
    "   2. **label**: a label that marks the article as potentially unreliable\n",
    "        \n",
    "        1: fake \n",
    "        0: real"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Specify the input filename\n",
    "INPUTFILE = r'fake_news_train.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Specify the ratio of test data for prediction\n",
    "test_data_ratio = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Loading & Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the file for analysis. Change the path of the file if required\n",
    "df = pd.read_csv(INPUTFILE, encoding='utf-8',header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>news_text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Get the latest from TODAY Sign up for our news...</td>\n",
       "      <td>fake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2d  Conan On The Funeral Trump Will Be Invited...</td>\n",
       "      <td>fake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>It’s safe to say that Instagram Stories has fa...</td>\n",
       "      <td>real</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Much like a certain Amazon goddess with a lass...</td>\n",
       "      <td>real</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>At a time when the perfect outfit is just one ...</td>\n",
       "      <td>real</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           news_text label\n",
       "0  Get the latest from TODAY Sign up for our news...  fake\n",
       "1  2d  Conan On The Funeral Trump Will Be Invited...  fake\n",
       "2  It’s safe to say that Instagram Stories has fa...  real\n",
       "3  Much like a certain Amazon goddess with a lass...  real\n",
       "4  At a time when the perfect outfit is just one ...  real"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['label'] = np.where(df['label'] == 'fake', 1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4998, 2)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data has 2 columns. 'news_text' column which contains the text which has the News data and 'label' column which contains the label if the news is Real or Fake **{fake, real}**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.patches.Wedge at 0x1516e970>,\n",
       "  <matplotlib.patches.Wedge at 0x1516ee68>],\n",
       " [Text(-1.1464648034448126, 0.3544269381159497, 'fake'),\n",
       "  Text(1.1464648034448124, -0.35442693811595033, 'real')],\n",
       " [Text(-0.6687711353428074, 0.20674904723430398, '40.5%'),\n",
       "  Text(0.6687711353428072, -0.20674904723430432, '59.5%')])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAADnCAYAAAA+T+sCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAewklEQVR4nO3deXyV5Zn/8c+VlRACCQTCEjygIssBFG1ZRBGXuta627p0XI/aRUdntJ32p7+pnandaP3VdWrsWLfWeWnH2p8zLli12mo7LoAYhAJCIAESdgJkP/f8cT8hgYYsnOfkfp7zXO/X60gI59znEvI9z3Y/1y3GGJRS0ZLlugClVP/T4CsVQRp8pSJIg69UBGnwlYogDb5SEaTBVyqCNPhKRZAGX6kI0uArFUEafKUiSIOvVARp8JWKIA2+UhGkwVcqgjT4SkWQBl+pCNLgKxVBGnylIkiDr1QEafCViiANvlIRpMFXKoI0+EpFUI7rAtShWyCSDwz3HqXeowTIxf7b7rzdmEfdVaiCSoMfcAtESoDJwBTv18nAUUAZMKiHl68CNPjqb2jwA2SByABgFjAPONHAdLEBV8pXGnyHFogUACcC84wN+0yB/PY/F2eVqUynwe9nC0QGAucYuAQ4W6AQNOSqf2nw+8ECkUHAuUm4VOBMgQFhCbokJA4cCbR0ejQCG4EaU2FaHZanDpEGP40WiBzdBrdkweUCA8J07VRikgPkcRpXkcUdB3lamyRkA7DuII+1psLs6peCVZ9o8H22QCQ3CZe0wT/mwrHZrgs6dFcDc6nlWEYd9DnZwFjvMberJ0hClgNvtz9MhVnrd6Gq7zT4PlkgMrwVbhO4IRuGhWnrfhBDgR1kk+oWe5L3SABIQqrp9EEAVJoKY1J8D9VHGvwULRApaYDv5MMNOTDAdT0hUA5c5j0AtklC/gA8A/zOVJhGZ5VFiAb/EC0QKdoLd+XD1wugwHU9ITYUuMB77JSEPAs8iT0s0D2BNNHg99ECkQF74dt5cNvAnmfOqb4ZAlzvPdZKQp4CnjAVZqXbsjJPBhyK9p9vi1zSDFUD4a4cDX26jQPuBP4qCXlXEvJVScgQxzVlDN3i98IdIrEB8NRQOMF1LRE123vcIwl5ELjXVJgtjmsKNd3id+NrIlnfEvmXYbBiiIY+CIYA3waqJCH3SkJGuy4orDT4B3GbyKwyWDUM7szpNH9eBcJA4FbgU+8DoNRxPaGjwT9AXCT7NpGfjIQ/DoTxrutR3cqn4wPgnyUhet6llzT4nVwuMu48WDQG/iFbz3+ESRHwHewHwPWOawkFDT4QF5FrRS6bAkuGwjTX9ahDNhyokIS8Kgk5zHUxQRb54MdF8ubDI5PgqQEw2HU9yhefAz6WhNwkCQnLjZD9KtLBP05kxOfgjfFwfVbE/y4yUBHwMPCaJETP1Rwgsj/sJ4lMORXeKYfjXdei0uoUYKkk5Ou69e8QyeCfJXLKPHhzBBzhuhbVLwqB+4E3JCH6b07Egh8XkfNFrjkeXhhiTwSpaDkJWCQJOcd1Ia5FJvhxkZwJ8I1Z8HCBzrOPsiLgBUnIra4LcSkSwY+L5MfguzPh7lydhads56B7JSEPS0IiOV8j44MfF8kbC985AW7L09Cr/d0EvCQJKXZdSH/L6ODHRfLK4c4T4e/ztDuO6tppwLtRO+mXscGPi+SOhm/Ng3/M1w45qnuTgL9IQk50XUh/ycjgx0VySuDWeXB7vr2TS6meDMNO9rmsx2dmgIwLflwkKxeunw+369l71Ud5wJOSkPNdF5JuGRd84JyT4fZiGOG6EBVK2cAzkpBTXReSThkV/LjIZ2fDd8bojDyVmnzgt5KQWa4LSZeMCX5cZPwkuGcSzHBdi8oIg7CX+jLyNu2MCH5cZFiZnaBzkujCs8o/JcCrmXipL/TBj4sMyIFbT4QzsyHXdT0q44zEnu0f47oQP4U6+HERAS6dC+cXgTZcVOkyDliYSU09Qx18YPrhcOV4mOq6EJXxJgPPSUJCvAByh9AGPy4ytBBumQ2z9aBe9ZOTgO+5LsIPoQx+XCQLuOYkmJ9vb7NUqr98QxLyBddFpCqUwQdOngyfL4PDXReiIkeAxyUhof7ZC13w4yJj8uDLM+BY17WoyCrGzu4L7b38oQq+t4t/5WyYmq+tsJVbnwX+2XURhypUwQeOLYVZ4+EY14UoBXxLEjLXdRGHIjTBj4sMBL48F47JsjdSKOVaNvCUJCR0e5+hCT5w1iSID9WFLFWwjAN+6LqIvgpF8OMiowXOOQaOdl2LUl1IhO1mnsAH35uW+6WpUF6g03JVMGUDP3VdRF8EPvjYa/XTp+jWXgXbaWGa2BPo4Htb+/OnwsiB2lFHBd8CSUgo7hANdPCxJ/KmxfXynQqHCcDNrovojcAG39vanzfFbu3LXNejVC/dFYbbdwMbfCAGHD0FprguRKk+KAa+67qIngQ5+F8YBQVFMNZ1IUr10Q2SkLjrIroTyODHRUqBGdPs5AilwiYbuMt1Ed3pMfgicouIfCIiTx/kz68WkQd8rmt2LkgZTPd5XKX6y4WSkNGuiziY3mzxvwp8zhhzRbqLAbv8FXD6NCjN0YUuVXjlAje6LuJgug2+iPwbdgLNSyLyTRF5V0QWicg7IjKxi+ef4z2nVERO977+UESeFZHeLmc1GSg6XLf2KvxulITkuS6iK90G3xhzE7ABOBl4GDjRGDMD+L/APZ2fKyIXAP8EnO19607gNGPMscD7wD/0sqZTyyBPT+qpDFAGXOK6iK70pYPIEOBxEZkAGPbvYX8K8BngdGPMLhH5PPYy3J/s5XjygHd7egPvpN70SRDYYyOl+uhmoMvzYy715az+vwBvGGOmAuey//H3amzTy6O83wuw0BhzjPeYYoy5rhfvMQOgzO7uK5UJZklCPuO6iAP1JfhDgBrv66sP+LMq4CLgCRGJA38G5orIkQAiUigiR9ENb6bevOFgCnWmnsosgZvG25fg/wj4vogsootDBGPMcuAK4FlsP7yrgV+LyEfY3fxJPYw/HCg/CjJqqSKlgC9KQoa7LqKzHo/xjTHjvC+30LErD/bkHcaYXwK/9L5eRMcU29XYhoS9NQUwZfu/h1KZIB+4CljgupB2QZq5N3sANBbZOfpKZZpzXRfQWSCCHxcpBCYcCcVZAalJKZ8dLwkpdl1Eu6CE7EhARui1e5W5coAzXBfRLijBnwS0FkO560KUSqNzXBfQLihLAE3LgvpBaTqjnwR+hr0eeS2wDXgK2Iv9pPkSf/sXsQ34MfZSA9gTDxcBrcBjwE7geO8B8BwwG/3kUt06SxKSZSpM0nUhzoMfFykARo2F1uz9ZwP65m1sw74m7/f/BczD9vP6DfA/dAS4s2H87TzjFdh+YKcAD3qv24D9cNHQqx6UAjOx81ycCsKufjnA6DTlZgewHJjl/d4Aq4D2JujHAZV9GC8baMEGvd0rBOjgTQVdIHb3gxD8wwAZmqYTe7/D/k2L9/u9QAEda3AVY3fbu7INuBd7d9Kn3vcmeN+/HzgB+6ExBnsYoVQvaPA9U4E9g9LQPnsZMIhD25UYDPwf4DbsBdhfAY3YD4wrvO9Pxx5GzMN+wDxB3/YeVCTNCEKDDqfB9+bnTwR25cNQv8dfiw3/PdiTeauAF4AGoM17zg663lrnAIXe1+XY4/3NBzznHeyhwjrsHUtXAm/5VbzKZCe5LsD1Fn8QMKAEBqTjxN7Z2HnF38aG8kjgcu/Xpd5zPgC66oq4m47j+K3Y+crDOv35XuATbPCb6TiUaPGvfJW5ZrguwPVZ/aFAsjQNW/vunI29Qfpl7PH5TO/7lUA19kTdp8Cr2E/GLOylvIGdxngNONX7s4nYrf9PsZf0lOrBMa4LCELwpaQfgn+E9wC75b6li+fE6dj6T6f73l+dF0nLBW5ItUAVJce4LsD1rn4pIIP6eYuvlGPDJSFObz93HfzDgMZCDb6Knmk9PyV9XAe/HGjI6ziBrlRUOO074Tr4I4DGbNuoQKkoiWbw4yJZ2BPlLbpwhoqgv1mXoj+53OLnY6fOk63BV9ETzS0+NuztwdddfRU15ZIQ6flp6eE8+AMhXzomvikVFe2Hus7e3JV8gAF2lR2loqjI1Ru73uIrFWWRDH4eIG3797RQ/jrwhkIVLJEMvgGMBj89jL3z+Ouu61DdimTw2wCSGvy02AJP3m7Mh67rUN2KZPCTAK0dPTGUT3bDhmq4yXUdqkfRDb7u6vurDdrq4LKnjWnq+dnKsUgGv837T9J4E3lU6urgkYeM8a8DWFZ6Wp4rIKLB37elb4U9DuvIGPXw6WI/12IvppihHOfbeOpAzmasugz+vvZ0zbbFnUpBK7TUwiW/NyblcyYSkwEIo5nKGWQ579KUyXa4emOXwd+NN1W3Geod1pER6uAn/+bfWfwLmMSJDGSUT+Oprm139caug58F0KRb/JTshGWv2GbCKZOYTGYol1HO0X6Mp7oVveBXGtOKDXyuBv/QtUBTHVxUaUzKJ0glJoPI4iamMgfZt9iQSp/oBd+zA8hr0F39Q1YLd//cmOWpjiMxEeCLxJnPAEp9KE31bIerN3Yd/K1Afv3Bl69T3dgO7y+EH/g03NEM5yJGum0CGTGR3eJvBfLqoM5xHaHTDHt83MUfQg43EWcOor0R+lFkg78RG/wdbXYlKtVLtXBHhTHrUh3H28W/kqmcSB7FqVememmPqTDOVlwLQvANwF7d6vfaVvjDz4x52KfhZjKK8xjOFJ/GU73jbGsP7oNfi3ctf7f9WvWgEXbVwaV+jCUxGUYeNzKJ2bqD3++c9kpwPStrG9AKZO+EWp0t0j0D1MJXf2FMyntHEpMs4GqmMZdcd3PG++wZ7GKFgt1snY89U/Qn7FzQQcDJdN3QravXAvwPsB67qOJ873srgSZgqu//B+2W9vyU9HEa/EpjknGR9UDJFt3i92gLvHi/MU/7NNw8xvJ5hrlt83xIzmH/xm1vA7OAUcAK4CPgM718bTN2DfSLgLewm6LB2OCf6WvVB1qS1tF74HpXH2A1ULgeNhm9Rfeg9sLWTXCFH2NJTEZSwHVM2LdCeLjtBEZ6X48B1vbx9Uns7lQbNhFLgSmkOx2L0zp6D4IQ/DVAXiM074Ya18UEURJMHVzzuDG7Uh1LYpIDXMc05pLjrr1zSl4Cngfapy2VAFXe12vo/l7PA1+bB4z1vlfg/b4OGOdrxV1xusV3fYwPUI23pd8Ka4rsP4PqZDP8xwPG/H+fhjuV8ZxJMeN9Gq9/nYtdYrUBG+IhwDzgXWAREOPgm7OuXjsKONp7gN3dPw77wVCDXcd5hs//D4Ya86jZ6vOofRKELX4N9jRKbg186rqYoNkDGzfAtX6MJTEZSyHXcDif9WM8J9rXVS7AhnwzUAycBVwAHIE9Ru/tazvb4v06BLvncCqwC//nlYrb3XwIQPArjUliT8cUfwrVbZ3u04+6JLTVwhVPG9OQ6lgSkzwgwXSOJzukS5a10DHNqwW7ySjBbsHBHqcvAib14bWdfYDd2rcf84O9AtDqQ+37c7qbD8HY1Qf7FzGzFdrqYV2x/dyOvDr4xUPGvOHTcGczgc9RFOJDqQbgNe/rJPanZCzwMbDM+/44Opaj3IM9439mN69ttxYopWOvYBjwG+yu/jBf/y9Ag7/Pvl38LbCmWINPPaxdDV/zYyyJyeEM5svEQt5GazBwYRffn0rX19sL6bgkd7DXthvH/if0ZnmP9FictpF7yfmuvqcOezQ1YFXH+dbIaoWWOrj0eduzICVeG60bmcYcbZwZAIZdwCrXZQQi+N4dZouAko2wNerz9uvgvoeNec+n4c5nIqdQqG20AkFYaCqM8/kqgQi+533shEo2QaXjWpzZCZ+8At/wYyyJySRKuIyxvl+QUofuv10XAMEK/krs+dbc5fZ0TeS0QFMtXOxd6UiJxKRQ22gF0kuuC4AABb/SmGbgz0BpLWyrtxN7IqUO/vURY5b1/MzueffYX8oU5lPAcB9KU34wfGQqzMZ0v42IrBWRbtunBSb4nr/g7e5X22v7kbEdPnwVvufTcNMYziWMYrpP4yk/CL/r80ss33MatOD/FXv1Nf9j+DiZjqkTAdQMe31sozWYHG5iCrO0jVbgPNebJ4nIOBFZISJPYA977xKR90TkIxG5u9PzfisiH4hIpYjc0JdCAhV8r+X2m8Dw3dBQF5Gtfi38U4Uxa1Mdx9vFv4I488hnaOqVKd8kWWMqTF8m7kwAHgJuw95zOBM4BjhOROZ5z7nWGHMc9ibkW0Sk11ONAhV8z1+AbEA+ssf8GW0rvP0aPODTcJ9lJBcwgrhP4ym/CM/08RVVxpg/A6d7j0XAh9gJyRO859wiIkuwORnb6fs9CsrMvX0qjamOi3wCHFYDm3fA6uIMncnXCLtq4VKfdvGHkscNTGaW7uAHkPBsH1/RfnOxAN83xvx8v+FE5gOnAXOMMXtF5E32bzHSrSBu8cFe6ywEWJGhW30D1MHN/27MplTH8tpoXcVUTiD3oPemKVeSrDAVZtEhvvoV4FoRGQQgImNEZAT2HsLtXugnAbP7MmhQg78Me9Nk0TJY1dBxw2TG2AIv32fMEz4NdwLlnEspE30aT/lJ+NGhvtQY8yrwK+BdEVmKPUFYBLwM5IjdO/4BfdxAig97mWkRFzkBuA6omgnHxm0bhYzQANtWwRGPG7Mj1bEkJmUM4Mccz4Xk7Lu3TAVFku1kMcpUmCbXpXQW1C0+2Cm8jUD+h7Ck0XEfcr8YMLVwnU+hzwauZTpzNfQBleThoIUeAhz8SmMasbszZa3QtgJed12THzbDcw8Y81ufhjuVcZxFMYf7NJ7yk6GFHP6f6zK6Etjge17H9k3J/xA+3mNX3gmtPbCpBq7yYyyJyRgGcg1HhLiNVqZr5VlTYZwunHEwgQ5+pTH12D4oIwE+goVuKzp0SUjWwZd9aqOVS0cbrV5fwlH9LJd7XJdwMIEOvudtbJOOwuWwZkdIG3LWwWMPGvNaz8/slTM5ktMZzGE+jaf81sofTYUJ7O3lgQ++d6z/DNi7zD6EhaajFWIo1MO61XCTH2NJTMZTxFWMC3kbrUyX7dsNV2kR+OB73sMe3xdXwaYN9veh0AatPrbRyke4genMIavL1eFUELSy0jxqXnZdRndCEXzv5p1fY3ueylvw+yb/u52nRS3c/7Axf/FpuPM4itMoZLRP46l0MP7s3aVTKILvWYpd13RUIzQvhhcd19OjnbDiFbjdj7EkJhMp4XJtoxVwDbxpfmkCf+k5NMH3bmT5Ffb4vmAZrArybbut0Fxr77H3o43WwH1ttLK0jVZgJWmhjWtcl9EboQk+QKUx24An8S7vvQUvt3S/RKIztXDPIyb1s7rePfaXMJmTKGCED6WpdNnLg+bXqfdV6A+hCr7nHWxXkpH10LA0IF1LO9sOi1+F7/o03FRKuYTR+5Z1VEHUwmay+KbrMnordMH3dp2fwPYSyFsCyzbYef2B0AwNtXChT/fYF5HNTcSZg4Tv3ypSGrnVPG2ae35iMITyh6nSmFrstf1yQF6Hl3fDBsdlAVAL33rUmDWpjrOvjdZUbaMVeA28Z54xv3JdRl+EMvie17HX88e0QNub8GyrvZvPma3wzmtwn0/DHUcZFzCiy1XhVFAkaaOVq12X0VehDb63y/8YsAMo2Qw7FsHzrqb0NUH9ZrsYhh+7+CXkcqO20QqBPTxonkl9LYT+FtrgA1QasxvbqLII25L7r+vgj/1dhwE2wd8/alJfLMFro/V3TOME8hiSenUqbXbzCXXc6rqMQxHq4ANU2rbUj2NbEMub8Po2WNGfNWyFV+435jGfhjueMXyBUib5NJ5Kh2Z2U8OF5vWAtrDqQeiD73nLe4xNgnkJnquH9f3xxg2wvRYu82MsickI8kkwkZl+jKfSJEmSjXzNvGVCu6R7RgTfO65+CnvL7uhmaH0Zfp3uJp1eG63EY8ak3BbsgDZag3woT6XLZp5kBU+6LiMVGRF82Hf77n3AVmDEbmhYCE82Q3263nMzPP+AMb/xabj5xDiLksxcQyBj7ORjlnCDqQrnLn67jAk+QKUxu4CfYtt1Dd0Ku/4AT6XjMt8eqNsAf+fHWBKT0QzkOo7UXfxAa2In1ZxrqsIzUedgMir4AJXGbAZ+AuQBRdVQ9yd42s/wJyFZC1c+ZUzK9wl4bbSu1zZaAZekjQ1ca94Jx1z8nmRc8AEqjVkP3AuUAAM/heq34PEW2OvH+HXwxEPG+NX/7wyO4EwGE/NpPOU3A2zkfvOa+U/XpfglI4MPUGnMCuBnQCkwqAo2vQ6PpXrMvxvWrYUb/ahRYjJO22iFQA0vUMkdrsvwU8YGH6DSmCXAAqAYGLwBtiyEx5rsbL8+a4PWWvjScyb1YzyJST6QYBpzyNY2WoG1gXdYxjWmKvXWaUGS0cEHqDRmGfBD7CKcxXWw/WX490O51FcLDz1szLs+lXYuEzmNQYzxaTzltzqW8DEXmarUL9cGTcYHH6DSmJXA94FcYNg2qH8RfrEDVvd2jF2wcgXc5kc9EpMJFHMFYznWj/FUGmzmEz7iC6Yq9dWMgygSwYd9U3u/DySBst3Q+AI8vd429uiW10br4v/yp41WAcKNTGMOWeSkOp5Kg818wmLOM2vMOtelpEtkgg9QaUw18K9AHXBYEngNFi6B/0zCQY/hauGHPzfGr/5+FzOZUyigzKfxlJ/qWM5iLjZrzUrXpaRTYJfJTqe4yADs5Ju5QDXQEoNRc+GL+ex/R9wO+Oh5OMan223jDOMHHMvZ2lEngOpYxhIuMWvDd5ttX0Xyh8+b3vsotmtvOfZy38YX4ZHtsKr9eV4brYt8bKP1FW2jFUCGJGt4h8XRCD1EdIvfWVwkDtwMtAGbAebAzAlw+gb45s+M+Umq7+G10bqe6dzMSKalOp7yUSuNfMLv2cgdpsp84rqc/hL54APERUYCXwEOA2qAUcNh2Wb4qU9b+2Mp43tM5wxEe+oERiPbWcxv2MV3TZXpl9u4g0KD74mL5AGfB84DtgN3eh1+UiIxKSaXHzGXS7WjToDsYB2LeZRm7jNVJhTLsflJg3+AuMjhQFOlMTWpjuXt4n+dGXyF4UxOvTrlixqWUsmPgWdMlWlxXY4Leh35AJXGfOrjcHMYzXka+oBI0spK/kgVdwN/CPs99anQ4KeJxKTUa6M1y3UtCtjDJpbxBtu521SZfu3JGEQa/DTwOuVew0hGk8NA1/VEWhtNrONDVvI6cJ+pMnWuSwoCDX56nARMpYp3aWANkzifATpTr9/tYg1L+YA9PA88b6pMg+uSgkKDnx5FgABF1LGJLTxCnBMYyTxEl7lOu1b2sJr3qOLPwC9MlVnV42siRs/qp4F3Nn8qcB32Q6AGSDKMUo7iNIqY6LTATLaV5XzMBzTxH8ArmdAfLx00+GkkMSkELgROA3YC2wAYyRiO4FQKGe+wvMzSSB0rWcxG3gIeN1Wm2nVJQabB7wcSk4nA5cA4bPjthJFyxjOeUyig3F11IddADWtZynrWY++9eNtUmTbXZQWdBr+feGf6jwa+BJRhOwDZmYHjmEiMU8hnhLsKQ2Yv61jDUmrYAryPnYyT1gVUMokGv59JTHKAzwBfxPYCrAMaEIQjmEo588ljqMsaA203a/iUj9nEVmzgXzRVpsp1WWGjwXdEYpIHHA9cjO0HuAloQhDGcgSjmMFgJupVAE89K1nNMurYju2a9N+mKvVp1VGlwXdMYlIAzAMuAPKxHYDtOYACChjPdEqZEcl5AC3sZBsfsZZN7GQ38DbwUqb2wetPGvyA8K4AzADOwDYHacX2B7CXo8oYRTkzKGZaRq+400YTu1hODcvZwF7schavA6+aKrPZcXUZQ4MfMN4cgHJgDnAydi9gL3YxUEMOOYxjEqXEKWRcRnwItNHILlZQx1+ppp42BHvycyHwgaky2xxXmHE0+AHmnQeYApyKnRCUxF4OtGv2CcJwyhjOeAYzjoHEyCbfWcG9laSFBmrYQzVbqaaG3STJBpqwu/PvAmujfPdcumnwQ0JiMgw4DvshMNz7dgv2nICdgy4IZYyilPEMZjwDOYwscl3Uu59mtrGXanaxnq1Us4U9GIqwU8YbsUF/H1gV1fvj+5sGP4QkJiXYyUBx4BjYd/mvGftBYFcGziKLoZQymGEUUkoBw8inlDxKyPb5rsEkzbRST4v3aGIHO6mmjlr2koO9cmGw9zBsAJYCi4HVmbY8VRho8EPOOycwFPtBMA07Sahzi68G7DmCBuyhgpVDDkUMppAhFDCYPAYiZHkdgGXf1wc+DEmaqaeJehqoZy/17KGeJpqwS5Pnw76tucHuvi8HKoEqoEbvknNPg59hvA+CUuwJwhHYD4Sx2NmCWXRsdbOwVw6asR8I7T8I5iAPsEuQ5cO+RT7bP0jEe9Rjz0Esx7YpXw9sMVWpr0Ck/KXBjwhvynAxMNh7DMGeKxiG3Tpn9/AA24R0C3a24U5s0Hd5v+7WOfLhocFXKoJ0RRelIkiDr1QEafCViiANvlIRpMFXKoI0+EpFkAZfqQjS4CsVQRp8pSJIg69UBGnwlYogDb5SEaTBVyqCNPhKRZAGX6kI0uArFUEafKUiSIOvVARp8JWKIA2+UhGkwVcqgjT4SkWQBl+pCPpfnd4ohzj3sicAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Plot the distribution of the label column in the data\n",
    "cat_target=df.label\n",
    "label_size = [cat_target.sum(),len(cat_target)-cat_target.sum()]\n",
    "plt.pie(label_size,explode=[0.1,0.1],colors=['darkred','darkgreen'],startangle=90,shadow=True,labels=['fake','real'],autopct='%1.1f%%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the input data is 'text', it is necessary to preprocess it. The steps in preprocessing include **Tokenization, Stop word removal, Stemming, Tf-Idf Vectorizer.** During this preprocessing phase, the data is cleansed,treated and converted to vectors for the machine/algorithm to understand!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a function to preprocess the data\n",
    "def preprocess_data(data):\n",
    "  \n",
    "  # 1. Tokenization\n",
    "    tk = RegexpTokenizer('\\s+', gaps = True)\n",
    "    text_data = [] # List for storing the tokenized data\n",
    "    for values in data.news_text:\n",
    "        tokenized_data = tk.tokenize(values) # Tokenize the news\n",
    "        text_data.append(tokenized_data) # append the tokenized data\n",
    "\n",
    "  # 2. Stopword Removal\n",
    "\n",
    "  # Extract the stopwords\n",
    "    sw = stopwords.words('english')\n",
    "    clean_data = [] # List for storing the clean text\n",
    "  # Remove the stopwords using stopwords\n",
    "    for data in text_data:\n",
    "        clean_text = [words.lower() for words in data if words.lower() not in sw]\n",
    "        clean_data.append(clean_text) # Appned the clean_text in the clean_data list\n",
    "  \n",
    "  # 3. Stemming\n",
    "\n",
    "  # Create a stemmer object\n",
    "    ps = PorterStemmer()\n",
    "    stemmed_data = [] # List for storing the stemmed data\n",
    "    for data in clean_data:\n",
    "        stemmed_text = [ps.stem(words) for words in data] # Stem the words\n",
    "        stemmed_data.append(stemmed_text) # Append the stemmed text\n",
    "  \n",
    "\n",
    "  # 4. tfidf vectorizer --> Term Frequency Inverse Document Frequency  \n",
    "  # Flatten the stemmed data\n",
    "\n",
    "    updated_data = []\n",
    "    for data in stemmed_data:\n",
    "        updated_data.append(\" \".join(data))\n",
    "\n",
    "  # TFID Vector object\n",
    "    tfidf = TfidfVectorizer()\n",
    "    tfidf_matrix = tfidf.fit_transform(updated_data)\n",
    "\n",
    "    return tfidf_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocess the text data\n",
    "preprocessed_data = preprocess_data(df.drop('label', axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this preprocessing step is a matrix of vectors representing the input data. We concatenate this with the target column for subsequent steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Input \u001b[1;32mIn [11]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[0m features_df \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mDataFrame(\u001b[43mpreprocessed_data\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtoarray\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[0;32m      2\u001b[0m datadf \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mconcat([features_df,df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlabel\u001b[39m\u001b[38;5;124m'\u001b[39m]],axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m)\n",
      "File \u001b[1;32mc:\\users\\sonu\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\scipy\\sparse\\_compressed.py:1051\u001b[0m, in \u001b[0;36m_cs_matrix.toarray\u001b[1;34m(self, order, out)\u001b[0m\n\u001b[0;32m   1049\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m out \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m order \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m   1050\u001b[0m     order \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_swap(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcf\u001b[39m\u001b[38;5;124m'\u001b[39m)[\u001b[38;5;241m0\u001b[39m]\n\u001b[1;32m-> 1051\u001b[0m out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_process_toarray_args\u001b[49m\u001b[43m(\u001b[49m\u001b[43morder\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1052\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (out\u001b[38;5;241m.\u001b[39mflags\u001b[38;5;241m.\u001b[39mc_contiguous \u001b[38;5;129;01mor\u001b[39;00m out\u001b[38;5;241m.\u001b[39mflags\u001b[38;5;241m.\u001b[39mf_contiguous):\n\u001b[0;32m   1053\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mOutput array must be C or F contiguous\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "File \u001b[1;32mc:\\users\\sonu\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\scipy\\sparse\\_base.py:1298\u001b[0m, in \u001b[0;36mspmatrix._process_toarray_args\u001b[1;34m(self, order, out)\u001b[0m\n\u001b[0;32m   1296\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m out\n\u001b[0;32m   1297\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m-> 1298\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mnp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mzeros\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mshape\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43morder\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43morder\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[1;31mValueError\u001b[0m: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size."
     ]
    }
   ],
   "source": [
    "features_df = pd.DataFrame(preprocessed_data.toarray())\n",
    "datadf = pd.concat([features_df,df['label']],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using **Scikit-learn** ML library for training and predicting. There are many algorithms available in Scikit-learn. We will be experimenting with **Logistic Regression & NaiveBayes** algorithms. Other algorithms available in the sklearn library https://scikit-learn.org/stable/supervised_learning.html that solves **Classification** problems can also be used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'datadf' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Input \u001b[1;32mIn [12]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# Split data into train & test in the ratio of 80:20\u001b[39;00m\n\u001b[0;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m train_test_split \n\u001b[1;32m----> 3\u001b[0m X_train, X_test, y_train, y_test \u001b[38;5;241m=\u001b[39m train_test_split(\u001b[43mdatadf\u001b[49m, datadf\u001b[38;5;241m.\u001b[39mlabel, test_size\u001b[38;5;241m=\u001b[39mtest_data_ratio, random_state \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m42\u001b[39m)\n",
      "\u001b[1;31mNameError\u001b[0m: name 'datadf' is not defined"
     ]
    }
   ],
   "source": [
    "# Split data into train & test in the ratio of 80:20\n",
    "from sklearn.model_selection import train_test_split \n",
    "X_train, X_test, y_train, y_test = train_test_split(datadf, datadf.label, test_size=test_data_ratio, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Code for Evaluation Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is a Classification problem, we write a function to get evaluation metrics like Confusion Matrix, Accuracy, Precision, Recall etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluation Metrics for Classification models\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "# Function for deriving a classification report & Confusion matrix for any algorithm\n",
    "def compute_metrics(data, y_true, model_obj, model):\n",
    "\n",
    "        # Make predictions\n",
    "    y_pred = model_obj.predict(data)\n",
    "    print(metrics.classification_report(y_true, y_pred,target_names = ['real', 'fake']))\n",
    "    \n",
    "    cm = confusion_matrix(y_true, y_pred, labels=model_obj.classes_)\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm,display_labels=['real', 'fake'])\n",
    "    disp.plot()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Logistic Regression Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import LogisticRegression model from sklearn library\n",
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "# Initialize the model \n",
    "lr_reg = LogisticRegressionCV(Cs=20, cv=3, random_state=42)\n",
    "\n",
    "# fit the model\n",
    "lr_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Compute the Evaluation Metrics for train set of Logistic Regression model\n",
    "lr_metrics_train =  compute_metrics(X_train, y_train, lr_reg, 'LogisticRegression')\n",
    "lr_metrics_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the Evaluation Metrics for test set for Logistic Regression model\n",
    "lr_metrics =  compute_metrics(X_test, y_test, lr_reg, 'LogisticRegression')\n",
    "lr_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Naive Bayes Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Naive Bayes model from sklearn library\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "# Initialize the model \n",
    "mnb = MultinomialNB(alpha=0.0)\n",
    "\n",
    "# Fit the model\n",
    "mnb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Compute the Evaluation Metrics for train set of Naive Bayes model\n",
    "mnb_metrics_train = compute_metrics(X_train, y_train, mnb, 'Naive Bayes')\n",
    "mnb_metrics_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Compute the Evaluation Metrics for test set of Naive Bayes model\n",
    "mnb_metrics = compute_metrics(X_test, y_test, mnb, 'Naive Bayes')\n",
    "mnb_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}