wiki-pre-processing and topic modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-processing wiki text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter by list of pages of interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get list of pages from catscan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/materials-pages', error_bad_lines = False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pageDict = {\" \".join(page.split('_')):0 for page in df['title']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pageDict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON pre-processed dump"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Search for articles in dump by title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#this works in python 2.7 but not in 3.6\n",
    "#try finding files by title\n",
    "import os\n",
    "import bz2\n",
    "import json\n",
    "import time\n",
    "\n",
    "t0 = time.time()\n",
    "\n",
    "if 0==0:\n",
    "    writefile = \"data/wiki_proc_filt.json\"\n",
    "    written = 0\n",
    "    with open(writefile, 'w') as fwrite:\n",
    "        dirs = os.listdir('data/wiki_processed')\n",
    "        for folder in dirs:\n",
    "            filelist = os.listdir(\"data//wiki_processed//\"+folder)\n",
    "            for filename in filelist: \n",
    "                filepath = \"data//wiki_processed//\" + \"//\".join([folder, filename])\n",
    "                with bz2.BZ2File(filepath, 'r') as fread:\n",
    "                    for line in fread:\n",
    "                        d = json.loads(line)\n",
    "                        if d['title'] in pageDict:\n",
    "                            written +=1\n",
    "                            fwrite.write(line)       \n",
    "\n",
    "\n",
    "    print(\"Searched for \" + str(len(pageDict)) + \" titles.\")\n",
    "    print(str(written) + \" titles found and written to \" + writefile)\n",
    "    \n",
    "t = time.time()\n",
    "print(\"time taken = \" + str(t-t0) + \"s.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Search for articles in dump by \"id\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our list of articles in the category 'materials' with depth = 5 had 197530 articles. Searching by title, only 164600 of those were found in the processed wikipedia dump. This might be because the catscan is more recent than the dump. \n",
    "\n",
    "Let's try to search instead by \"id\" rather than \"title\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this works in python 2.7 but not in 3.6\n",
    "import time\n",
    "\n",
    "t0 = time.time()\n",
    "\n",
    "#make the if statement true if you want to run this. May take a few minutes. \n",
    "\n",
    "if 1==1:\n",
    "\n",
    "    idDict = {id:0 for id in df[\"pageid\"]}\n",
    "    writefile = \"data/wiki_proc_filt_id.json\"\n",
    "    writtenId = 0\n",
    "    with open(writefile, 'w') as fwrite:\n",
    "        dirs = os.listdir('data/wiki_processed')\n",
    "        for folder in dirs:\n",
    "            filelist = os.listdir(\"data//wiki_processed//\"+folder)\n",
    "            for filename in filelist: \n",
    "                filepath = \"data//wiki_processed//\" + \"//\".join([folder, filename])\n",
    "                with bz2.BZ2File(filepath, 'r') as fread:\n",
    "                    for line in fread:\n",
    "                        d = json.loads(line)\n",
    "                        if int(d['id']) in idDict:\n",
    "                            writtenId +=1\n",
    "                            fwrite.write(line)\n",
    "                                \n",
    "    \n",
    "    print(\"Searched for \" + str(len(idDict)) + \" titles.\")\n",
    "    print(str(writtenId) + \" titles found and written to \" + writefile)\n",
    "    \n",
    "t = time.time()\n",
    "print(\"Time taken: \" + str(t-t0) + \"s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even after searching by id, about 27000 articles are missing. We will go ahead and process the articles we could find. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one complication in the dump I have processed. We need the dump with the links, but the one I have is without links. The idea behind preserving the links is that we should convert phrases within a link into a single entity - an n-gram. So below I obtained a different pre-processed dump. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing another pre-processed dump"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file is tab separated. Each section is written as a new entry. Also contains references. \n",
    "1. If there's a link, join words within the link by underscores to use it as a phrase\n",
    "2.  Write the output to a new file in json format\n",
    "3. Join sections so that each article is one entry.\n",
    "4. Add a new entry in the json file with section names for each article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def join_links(example):\n",
    "    pattern = re.compile(r'(<a href\\\")(.*?)(\\\">)(.*?)(</a>)')\n",
    "    matches = pattern.findall(example)\n",
    "    for match in matches:\n",
    "        #print(match)\n",
    "        t = ''.join(x for x in match)\n",
    "        example = example.replace(t, match[1].replace(\" \",\"_\"))\n",
    "    return example\n",
    "    #print(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "readfile = open(\"data//wiki_materials_rows_out.txt\", 'r')\n",
    "tempfile = open(\"data/link_joined.txt\", 'w')\n",
    "t0 = time.time()\n",
    "i=0\n",
    "for line in readfile:\n",
    "    line = join_links(line)\n",
    "    tempfile.write(line)\n",
    "    \n",
    "t = time.time()\n",
    "print(\"Time taken = \" + str(t-t0) + \"s.\")\n",
    "tempfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at how the file looks now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "readfile = open(\"data//link_joined.txt\", 'r')\n",
    "\n",
    "i=0\n",
    "for line in readfile:\n",
    "    if i<2:\n",
    "        example=line\n",
    "    elif i<5:\n",
    "        print(line)\n",
    "    else:   \n",
    "        break\n",
    "    i+=1\n",
    "    \n",
    "readfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ease of working, convert the tab separated .txt file into json file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t0 = time.time()\n",
    "lines_read = 0\n",
    "lines_write = 0\n",
    "with open(\"data/link_joined.txt\", 'r') as readfile:\n",
    "    with open(\"data/link_joined.json\", 'w') as writefile:\n",
    "        for line in readfile: \n",
    "            lines_read+=1\n",
    "            data = line.split('\\t')\n",
    "            if len(data)==6:\n",
    "                lines_write += 1\n",
    "                line = {\"id\" : data[0], \"title\" : data[1], \"section\" : data[2], \"text\" : data[5]} \n",
    "                d = json.dumps(line)\n",
    "                writefile.write(d+'\\n')\n",
    "print(str(lines_read) + \" lines read.\")\n",
    "print(str(lines_write) + \" lines written. \")\n",
    "                \n",
    "t = time.time()\n",
    "print(\"time taken = \" + str(t-t0) + \"s.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure we still have all the articles we started with ~166000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_dict = {}\n",
    "with open(\"data/link_joined.json\") as f_r: # | wc -l\n",
    "    i = 0\n",
    "    for line in f_r:\n",
    "        #if i>10:\n",
    "        #    break\n",
    "        d = json.loads(line)  \n",
    "        title_dict[d['title']] = 1\n",
    "        i+=1\n",
    "        \n",
    "print(len(title_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fuse text for same title together\n",
    "Assume that text from the same article is consecutive. Once an article ends, it never restarts again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#n_line = 0\n",
    "t0 = time.time()\n",
    "titles = 0\n",
    "n_line = 1\n",
    "with open(\"data/link_joined.json\", 'r') as readfile:\n",
    "    with open(\"data/titles_joined.json\", 'w') as writefile:\n",
    "        for line in readfile:\n",
    "            if 0==0:\n",
    "                d = json.loads(line)\n",
    "                n_line += 1\n",
    "                if n_line==1:\n",
    "                    id_prev = d[\"id\"]\n",
    "                    title_prev = d[\"title\"]\n",
    "                    text_prev = d[\"text\"]\n",
    "                    section_prev = d[\"section\"]\n",
    "                    sections = d[\"section\"] \n",
    "                    #new_title = 0\n",
    "                elif d[\"id\"] == id_prev:\n",
    "                    new_title = 0\n",
    "                    text_prev = text_prev + \"\\n\" + d[\"text\"]\n",
    "                    if d[\"section\"] != section_prev: \n",
    "                        sections = sections + \",\" + d[\"section\"]\n",
    "                else:\n",
    "                    titles+=1\n",
    "                    #new_title = 1\n",
    "                    #print(title_prev)\n",
    "                    writeline = {\"id\": id_prev, \"title\": title_prev, \"section\": sections, \"text\": text_prev}\n",
    "                    writeline = json.dumps(writeline)\n",
    "                    writefile.write(writeline + '\\n')\n",
    "                    id_prev = d[\"id\"]\n",
    "                    title_prev = d[\"title\"]\n",
    "                    text_prev = d[\"text\"]\n",
    "                    section_prev = d[\"section\"]\n",
    "                    sections = d[\"section\"] \n",
    "\n",
    "print(str(titles) + \" written.\")\n",
    "t = time.time()\n",
    "print(\"time taken = \" + str(t-t0) + \"s.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a list of titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "titles = {}\n",
    "with open('data/titles_joined.json', 'r') as readfile:\n",
    "    for line in readfile:\n",
    "        d = json.loads(line)\n",
    "        title = d['title'].replace(\" \", \"_\")\n",
    "        titles[title] = 0\n",
    "len(titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles.keys()[5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get categories for each title from dbpedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WikiExtractor does not import the category information for each article by default. However dbpedia stores a lot of structured information about each wikipedia article such as category names, templates, info-boxes etc. \n",
    "\n",
    "Let's get the categories info from dbpedia. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "import warnings\n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('error')\n",
    "t0 = time.time()\n",
    "titles_cat = defaultdict(list)\n",
    "with open('data/article_categories_en.ttl', 'r') as readfile:\n",
    "    n=0\n",
    "    for line in readfile:\n",
    "        if n==0 or len(line.split(\">\"))<2:\n",
    "            n+=1\n",
    "            continue\n",
    "        n=n+1\n",
    "        t_db = line.split(\"<http://dbpedia.org/resource/\")[1].split(\">\")[0]\n",
    "        t_db = t_db.decode('utf-8')\n",
    "\n",
    "        if t_db in titles: \n",
    "            cat = line.split(\"<http://dbpedia.org/resource/\")[2].split(\":\")[1].split(\">\")[0]\n",
    "            titles_cat[t_db].append(cat)            \n",
    "        if 0==1:\n",
    "            break\n",
    "t = time.time()\n",
    "print(\"Extracted categories for \" + str(len(titles_cat.keys())) + \" titles in \" + str(t-t0) + \"s.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "notfound = []\n",
    "for key in titles.keys():\n",
    "    if key in titles_cat:\n",
    "        continue\n",
    "    else:\n",
    "        notfound.append(key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in notfound:\n",
    "    print(x)\n",
    "    #print(x.encode('ascii', 'ignore'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/titles_cat.json', 'w') as writefile:\n",
    "    json.dump(titles_cat, writefile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t0 = time.time()\n",
    "titles = 0\n",
    "with open('data/wiki_final_processed.json', 'w') as writefile:\n",
    "    with open ('data/titles_joined.json', 'r') as readfile:\n",
    "        for line in readfile:\n",
    "            d = json.loads(line)\n",
    "            d['title'] = d['title'].replace(\" \", \"_\")\n",
    "            if d['title'] in titles_cat:\n",
    "                titles += 1\n",
    "                d['category'] = titles_cat[d['title']]\n",
    "                writeline = json.dumps(d)\n",
    "                writefile.write(writeline + '\\n')\n",
    "t = time.time()\n",
    "print(\"Wrote categories for \" + str(titles) + \" articles in \" + str(t-t0) + \"s.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a list of articles under each category. Let's also get a list of categories for each article, by storing in a data structure called default dict. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from collections import defaultdict\n",
    "ids = []\n",
    "categories = []\n",
    "n=0\n",
    "cat_titles = defaultdict(list)\n",
    "with open('data/wiki_final_processed.json','r') as readfile:\n",
    "    for line in readfile:\n",
    "        d = json.loads(line)\n",
    "        n+=1\n",
    "        for category in d['category']:\n",
    "            cat_titles[category].append(d['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store categories and article names in a pandas dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will store this in a pandas dataframe for easy filtering and querying. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cats = pd.DataFrame.from_dict(cat_items, orient = 'index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer2.stem('gardening')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>articles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Climate_forcing</td>\n",
       "      <td>[Albedo, Greenhouse_effect, Global_warming_pot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Climatology</td>\n",
       "      <td>[Albedo, Drought, Extreme_weather, Weathering,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Electromagnetic_radiation</td>\n",
       "      <td>[Albedo, Aberration_of_light, Beer–Lambert_law...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Land_surface_effects_on_climate</td>\n",
       "      <td>[Albedo, Rain_shadow, Land_surface_effects_on_...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Radiometry</td>\n",
       "      <td>[Albedo, Crookes_radiometer, Radiometry, Pyrom...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          category  \\\n",
       "0                  Climate_forcing   \n",
       "1                      Climatology   \n",
       "2        Electromagnetic_radiation   \n",
       "3  Land_surface_effects_on_climate   \n",
       "4                       Radiometry   \n",
       "\n",
       "                                            articles  \n",
       "0  [Albedo, Greenhouse_effect, Global_warming_pot...  \n",
       "1  [Albedo, Drought, Extreme_weather, Weathering,...  \n",
       "2  [Albedo, Aberration_of_light, Beer–Lambert_law...  \n",
       "3  [Albedo, Rain_shadow, Land_surface_effects_on_...  \n",
       "4  [Albedo, Crookes_radiometer, Radiometry, Pyrom...  "
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cats_df = pd.DataFrame(list(cat_titles.items()), columns=['category', 'articles'])\n",
    "cats_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort the dataframe based on the number of articles in each category, so that we can focus on categories with the most articles. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "cats_df['n_articles'] = [len(cats_df.iloc[i][1]) for i in range(len(cats_df))]\n",
    "cats_df.sort_values('n_articles', axis=0, ascending=False, inplace=True, na_position='last')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>articles</th>\n",
       "      <th>n_articles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Living_people</td>\n",
       "      <td>['Anatoly_Karpov', 'Anita_Hill', 'Dale_Chihuly...</td>\n",
       "      <td>3536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>American_films</td>\n",
       "      <td>['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...</td>\n",
       "      <td>3438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>American_black-and-white_films</td>\n",
       "      <td>['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...</td>\n",
       "      <td>3036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Enzymes_of_unknown_structure</td>\n",
       "      <td>['L-lactate_dehydrogenase_(cytochrome)', '1,4-...</td>\n",
       "      <td>2087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>English-language_films</td>\n",
       "      <td>['King_Kong_(1933_film)', 'U-571_(film)', 'Cro...</td>\n",
       "      <td>1964</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         category  \\\n",
       "0                   Living_people   \n",
       "1                  American_films   \n",
       "2  American_black-and-white_films   \n",
       "3    Enzymes_of_unknown_structure   \n",
       "4          English-language_films   \n",
       "\n",
       "                                            articles  n_articles  \n",
       "0  ['Anatoly_Karpov', 'Anita_Hill', 'Dale_Chihuly...        3536  \n",
       "1  ['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...        3438  \n",
       "2  ['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...        3036  \n",
       "3  ['L-lactate_dehydrogenase_(cytochrome)', '1,4-...        2087  \n",
       "4  ['King_Kong_(1933_film)', 'U-571_(film)', 'Cro...        1964  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cats_df.to_csv('data\\cats_articles', index=False, header = True)\n",
    "#cats_df = pd.read_csv('data\\cats_articles')\n",
    "cats_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "cats_df.iloc[:300, [0,2]].to_csv('data\\cats_articles300', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize with Spacy and save to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "import time\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en')\n",
    "from spacy.tokenizer import Tokenizer\n",
    "tokenizer = Tokenizer(nlp.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lucky Jim is a novel by Kingsl ['Lucky', 'Jim', 'is', 'a']\n",
      "Tokenized 10000 articles\n",
      "A hydraulic press is a device  ['A', 'hydraulic', 'press', 'is']\n",
      "Tokenized 20000 articles\n",
      "Inertron is a fictional metal  ['Inertron', 'is', 'a', 'fictional']\n",
      "Tokenized 30000 articles\n",
      "A focusing screen is a flat tr ['A', 'focusing', 'screen', 'is']\n",
      "Tokenized 40000 articles\n",
      "Ray-Ban Wayfarer sunglasses ha ['Ray-Ban', 'Wayfarer', 'sunglasses', 'have']\n",
      "Tokenized 50000 articles\n",
      "A gate dielectric is a dielect ['A', 'gate', 'dielectric', 'is']\n",
      "Tokenized 60000 articles\n",
      "Lyrick Studios was an American ['Lyrick', 'Studios', 'was', 'an']\n",
      "Tokenized 70000 articles\n",
      "In enzymology, an adenosylmeth ['In', 'enzymology,', 'an', 'adenosylmethionine']\n",
      "Tokenized 80000 articles\n",
      "Dancing on Coral is a Miles_Fr ['Dancing', 'on', 'Coral', 'is']\n",
      "Tokenized 90000 articles\n",
      "Difluorocarbene is the chemica ['Difluorocarbene', 'is', 'the', 'chemical_compound']\n",
      "Tokenized 100000 articles\n",
      "6-(2-Aminopropyl)-2,3-dihydrob ['6-(2-Aminopropyl)-2,3-dihydrobenzofuran', '(6-APDB,', '4-Desoxy-MDA,', 'EMA-3)']\n",
      "Tokenized 110000 articles\n",
      "Cariporide is a selective Na+/ ['Cariporide', 'is', 'a', 'selective']\n",
      "Tokenized 120000 articles\n",
      "Baker-Miller Pink is a tone of ['Baker-Miller', 'Pink', 'is', 'a']\n",
      "Tokenized 130000 articles\n",
      "Galloping Bungalows: The Rise  ['Galloping', 'Bungalows:', 'The', 'Rise']\n",
      "Tokenized 140000 articles\n",
      "The following is a list of cou ['The', 'following', 'is', 'a']\n",
      "Tokenized 150000 articles\n",
      "1296.5032212734222\n"
     ]
    }
   ],
   "source": [
    "t0 = time.time()\n",
    "count = 0\n",
    "\n",
    "with open('data/wiki_final_processed.json', 'r') as readfile:\n",
    "    with open('data/wiki_tokenized.json', 'w') as writefile:\n",
    "        for line in readfile:\n",
    "            d = json.loads(line)\n",
    "            tokens = [str(token) for token in tokenizer(d['text'])]\n",
    "            d['text'] = \" \".join(tokens)\n",
    "            writefile.write(json.dumps(d) + '\\n') \n",
    "            count += 1\n",
    "            if count%10000 == 0:\n",
    "                print(d['text'][:30], tokens[:4])\n",
    "                print(\"Tokenized\", count, \"articles\")\n",
    "t= time.time()\n",
    "print(t-t0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming with NLTK "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\Astha\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "stemmer = SnowballStemmer(\"english\", ignore_stopwords=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'run'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem('running')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data pre-processing for a 5-class chemical-physics classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From among the 300 most frequent categories, I have picked 5 categories to train a classifier on. For a single label classifier, let's make sure that the categories don't share a significant number of articles. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "choose = ['Fluid_dynamics', 'Quantum_mechanics', 'Condensed_matter_physics', 'Limestone_caves', 'Acoustics']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quantum_mechanics Fluid_dynamics 0\n",
      "Condensed_matter_physics Fluid_dynamics 2\n",
      "Condensed_matter_physics Quantum_mechanics 17\n",
      "Limestone_caves Fluid_dynamics 0\n",
      "Limestone_caves Quantum_mechanics 0\n",
      "Limestone_caves Condensed_matter_physics 0\n",
      "Acoustics Fluid_dynamics 8\n",
      "Acoustics Quantum_mechanics 0\n",
      "Acoustics Condensed_matter_physics 0\n",
      "Acoustics Limestone_caves 0\n"
     ]
    }
   ],
   "source": [
    "#from sets import Set\n",
    "intersect = []\n",
    "for i, cat_1 in enumerate(choose):\n",
    "    for j, cat_2 in enumerate(choose):\n",
    "        if i<=j:\n",
    "            continue\n",
    "        art_1 = set(cats_df[cats_df['category' ] == cat_1]['articles'].tolist()[0])\n",
    "        #print(art_1)\n",
    "        art_2 = set(cats_df[cats_df['category' ] == cat_2]['articles'].tolist()[0])\n",
    "        #intersect.append(len(list(art_1.intersection(art_2))))\n",
    "        print(cat_1, cat_2, len(list(art_1.intersection(art_2))))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some categories have some common articles, but less than 10% are common. Notably, :\n",
    "1. Condensed_matter_physics and Quantum_mechanics have the highest # of articles in common. \n",
    "2. Limestone_caves is the most separate category from the remaining 4. \n",
    "\n",
    "Let's write all the tokenized text for this selected set of articles to a file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "arts = defaultdict(list)\n",
    "for cat in choose:\n",
    "    articles = cats_df[cats_df['category']==cat]['articles'].tolist()[0]\n",
    "    for art in articles:\n",
    "        arts[art].append(cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cats_df[cats_df['category']==choose[0]]['articles'].tolist()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "written 1920 articles\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "written = 0\n",
    "with open('data\\wiki_tokenized.json', 'r') as readfile:\n",
    "    with open('data\\physchemx.json', 'w') as writefile:\n",
    "        for line in readfile:\n",
    "            d = json.loads(line)\n",
    "            if d['title'] in arts:\n",
    "                written+= 1\n",
    "                d['category'] = arts[d['title']]\n",
    "                writeline = json.dumps(d)\n",
    "                writefile.write(writeline + '\\n')\n",
    "print(\"written\", written, \"articles\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5 class single label classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>id</th>\n",
       "      <th>section</th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Acoustics]</td>\n",
       "      <td>1198</td>\n",
       "      <td>Abstract,History,History,History,History,Histo...</td>\n",
       "      <td>Acoustics is the interdisciplinary science tha...</td>\n",
       "      <td>Acoustics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Fluid_dynamics, Acoustics]</td>\n",
       "      <td>1234</td>\n",
       "      <td>Abstract,Derivation of the governing equations...</td>\n",
       "      <td>Acoustic theory is a scientific field that rel...</td>\n",
       "      <td>Acoustic_theory</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Condensed_matter_physics]</td>\n",
       "      <td>4474</td>\n",
       "      <td>Abstract,History,History,History,History,Criti...</td>\n",
       "      <td>A Bose–Einstein condensate (BEC) is a state_of...</td>\n",
       "      <td>Bose–Einstein_condensate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Quantum_mechanics]</td>\n",
       "      <td>4542</td>\n",
       "      <td>Abstract,Usage,Usage,Usage,Usage,Usage,Usage,V...</td>\n",
       "      <td>In quantum_mechanics, bra–ket notation is a st...</td>\n",
       "      <td>Bra–ket_notation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Condensed_matter_physics]</td>\n",
       "      <td>5346</td>\n",
       "      <td>Abstract,Classification,Classification,Classif...</td>\n",
       "      <td>A colloid, in chemistry, is a mixture in which...</td>\n",
       "      <td>Colloid</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      category    id  \\\n",
       "0                  [Acoustics]  1198   \n",
       "1  [Fluid_dynamics, Acoustics]  1234   \n",
       "2   [Condensed_matter_physics]  4474   \n",
       "3          [Quantum_mechanics]  4542   \n",
       "4   [Condensed_matter_physics]  5346   \n",
       "\n",
       "                                             section  \\\n",
       "0  Abstract,History,History,History,History,Histo...   \n",
       "1  Abstract,Derivation of the governing equations...   \n",
       "2  Abstract,History,History,History,History,Criti...   \n",
       "3  Abstract,Usage,Usage,Usage,Usage,Usage,Usage,V...   \n",
       "4  Abstract,Classification,Classification,Classif...   \n",
       "\n",
       "                                                text                     title  \n",
       "0  Acoustics is the interdisciplinary science tha...                 Acoustics  \n",
       "1  Acoustic theory is a scientific field that rel...           Acoustic_theory  \n",
       "2  A Bose–Einstein condensate (BEC) is a state_of...  Bose–Einstein_condensate  \n",
       "3  In quantum_mechanics, bra–ket notation is a st...          Bra–ket_notation  \n",
       "4  A colloid, in chemistry, is a mixture in which...                   Colloid  "
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_json('data\\physchemx.json', lines = True)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = data['title']\n",
    "X_raw = data['text']\n",
    "y_raw = data['category']\n",
    "y = [y[0] for y in y_raw] #keep only 1st category if there are multiple categories for a title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function text_process removes punctuation and stop-words. This will be passed to the count-vectorizer from sklearn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "punc_dict = dict((s,1) for s in string.punctuation)\n",
    "stop_dict = dict((c,1) for c in STOP_WORDS)\n",
    "stop_dict['\\n\\n'] = 1\n",
    "stop_dict['\\n'] = 1\n",
    "punc_dict.pop('_', None)\n",
    "#make all lowercase, remove punctuation and stop words\n",
    "def text_process(text, punc_dict= punc_dict, stop_dict=stop_dict):\n",
    "    nopunc = \"\"\n",
    "    text = text.lower()\n",
    "    for char in text:\n",
    "        if char not in punc_dict:\n",
    "            nopunc += char\n",
    "    processed = [word for word in nopunc.split(\" \") if word not in stop_dict]\n",
    "    return processed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what it does. It creates a list of words - tokens for each text that is passed to it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#text_process(X_raw[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split data into test and train sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train_raw, X_test_raw, y_train, y_test = train_test_split(X_raw,y, test_size = 0.3, random_state = 101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vectorize using sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer = CountVectorizer(analyzer = text_process)\n",
    "vector_fit = vectorizer.fit(X_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = vector_fit.transform(X_raw)\n",
    "X_train = vector_fit.transform(X_train_raw)\n",
    "X_test = vector_fit.transform(X_test_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the sparsity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.37775081116238585\n"
     ]
    }
   ],
   "source": [
    "density= X.nnz/X.shape[0]/X.shape[1]*100\n",
    "print(density)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Naive Bayes classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "model = MultinomialNB()\n",
    "fit = model.fit(X_train, y_train)\n",
    "predict = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 67   2   7   2   0]\n",
      " [  1  85   3   0  15]\n",
      " [  1   4 158   1   1]\n",
      " [  0   0   0  83   0]\n",
      " [  0  20   0   1 125]]\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "               Acoustics       0.97      0.86      0.91        78\n",
      "Condensed_matter_physics       0.77      0.82      0.79       104\n",
      "          Fluid_dynamics       0.94      0.96      0.95       165\n",
      "         Limestone_caves       0.95      1.00      0.98        83\n",
      "       Quantum_mechanics       0.89      0.86      0.87       146\n",
      "\n",
      "             avg / total       0.90      0.90      0.90       576\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "print(confusion_matrix(y_test, predict))\n",
    "print(classification_report(y_test, predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple Naive Bayes classifier is able to do pretty well on identifying the correct category for each article. The precision is very high for classes with no overlaps, while quantum mechanics and condensed matter physics have relatively lower precision. \n",
    "This is in-fact related to noisyness of the data itself. Condensed_matter_physics and Quantum_mechanics share 17 articles in common, but we forced only one of those categories for each of those articles - this introduced noise in the training data itself. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Broader classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the list of top 100 categories with the most articles. Recall that this list of articles was obtained by traversing the wikipedia graph starting at the categories 'materials' and traversing up to depth 5, with the motive of getting a broad range of articles about materials science and related topics such as chemistry, engineering and life-science. \n",
    "\n",
    "However, looking at the top 100 categories, \n",
    "1. a huge number of them seem to have nothing to do with materials science or related topics! For example 'Living_people', American_films. \n",
    "2. A lot of articles belong to very narrow categories such as 'phenol_ethers' or 'transcription_factors'. For future categorization, we would like to use a broader set of categories. \n",
    "\n",
    "Thus the problem statement is to do an unsupervised clustering of these categories. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>articles</th>\n",
       "      <th>n_articles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Living_people</td>\n",
       "      <td>['Anatoly_Karpov', 'Anita_Hill', 'Dale_Chihuly...</td>\n",
       "      <td>3536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>American_films</td>\n",
       "      <td>['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...</td>\n",
       "      <td>3438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>American_black-and-white_films</td>\n",
       "      <td>['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...</td>\n",
       "      <td>3036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Enzymes_of_unknown_structure</td>\n",
       "      <td>['L-lactate_dehydrogenase_(cytochrome)', '1,4-...</td>\n",
       "      <td>2087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>English-language_films</td>\n",
       "      <td>['King_Kong_(1933_film)', 'U-571_(film)', 'Cro...</td>\n",
       "      <td>1964</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         category  \\\n",
       "0                   Living_people   \n",
       "1                  American_films   \n",
       "2  American_black-and-white_films   \n",
       "3    Enzymes_of_unknown_structure   \n",
       "4          English-language_films   \n",
       "\n",
       "                                            articles  n_articles  \n",
       "0  ['Anatoly_Karpov', 'Anita_Hill', 'Dale_Chihuly...        3536  \n",
       "1  ['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...        3438  \n",
       "2  ['The_Birth_of_a_Nation', 'King_Kong_(1933_fil...        3036  \n",
       "3  ['L-lactate_dehydrogenase_(cytochrome)', '1,4-...        2087  \n",
       "4  ['King_Kong_(1933_film)', 'U-571_(film)', 'Cro...        1964  "
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "cats_df = pd.read_csv('data\\cats_articles')\n",
    "cats_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach\n",
    "\n",
    "Choice of features: \n",
    "1. bag of words feature vector for each document\n",
    "2. topic vectors as features for each document. The topic vectors are obtained using LDA topic modeling\n",
    "3. doc2vec vectors as feature for each document. \n",
    "\n",
    "Choice of clustering algorithms \n",
    "1. k-means clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling with Latent Dirichlet Allocation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Astha\\Anaconda2\\envs\\tensorflow\\lib\\site-packages\\gensim\\utils.py:860: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    }
   ],
   "source": [
    "from gensim.corpora import Dictionary, MmCorpus\n",
    "from gensim.models.ldamulticore import LdaMulticore\n",
    "from gensim.models.word2vec import LineSentence\n",
    "\n",
    "import pyLDAvis\n",
    "import pyLDAvis.gensim\n",
    "import warnings\n",
    "import os\n",
    "import numpy as np\n",
    "import _pickle as pickle\n",
    "import json\n",
    "\n",
    "from gensim.parsing.preprocessing import stem_text\n",
    "from gensim.parsing.preprocessing import strip_punctuation\n",
    "from gensim.parsing.preprocessing import strip_numeric\n",
    "\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "stop_dict = dict((c,1) for c in STOP_WORDS)\n",
    "stop_dict['\\n\\n'] = 1\n",
    "stop_dict['\\n'] = 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text normalization and creating a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000  records processed. Length of dict =  367411\n",
      "filtered to length  77869\n",
      "20000  records processed. Length of dict =  279278\n",
      "filtered to length  81451\n",
      "30000  records processed. Length of dict =  246511\n",
      "filtered to length  82826\n",
      "40000  records processed. Length of dict =  217731\n",
      "filtered to length  83515\n",
      "50000  records processed. Length of dict =  211192\n",
      "filtered to length  84254\n",
      "60000  records processed. Length of dict =  210922\n",
      "filtered to length  84829\n",
      "70000  records processed. Length of dict =  202569\n",
      "filtered to length  85421\n",
      "80000  records processed. Length of dict =  173796\n",
      "filtered to length  86079\n",
      "90000  records processed. Length of dict =  188285\n",
      "filtered to length  86499\n",
      "100000  records processed. Length of dict =  206546\n",
      "filtered to length  87115\n",
      "110000  records processed. Length of dict =  198681\n",
      "filtered to length  87565\n",
      "120000  records processed. Length of dict =  201271\n",
      "filtered to length  88013\n",
      "130000  records processed. Length of dict =  197962\n",
      "filtered to length  88583\n",
      "140000  records processed. Length of dict =  194764\n",
      "filtered to length  89043\n",
      "150000  records processed. Length of dict =  198108\n",
      "filtered to length  89583\n",
      "352.3198137283325\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "t0 = time.time()\n",
    "corpus = []\n",
    "dct = Dictionary(corpus)\n",
    "with open('data/wiki_tokenized.json', 'r') as readfile:\n",
    "    with open('data/wiki_nostem.json', 'w') as writefile:\n",
    "        for i, line in enumerate(readfile):\n",
    "            if 5==5:\n",
    "                d = json.loads(line)\n",
    "                text = strip_punctuation(strip_numeric(d['text']))\n",
    "                text = \" \".join([word for word in text.split(\" \") if word not in stop_dict])\n",
    "                d['text'] = text\n",
    "                writeline = json.dumps(d)\n",
    "                writefile.write(writeline + '\\n')\n",
    "                text = text.split(\" \")\n",
    "                \n",
    "                text = np.array(text).reshape(1,-1)\n",
    "                dct.add_documents(text)\n",
    "                if i%10000 == 0 and i!=0:\n",
    "                    print(i, \" records processed. Length of dict = \", len(dct) )\n",
    "                    dct.filter_extremes(keep_n=800000)\n",
    "                    dct.compactify()\n",
    "                    print(\"filtered to length \", len(dct))\n",
    "\n",
    "dct.filter_extremes(keep_n=800000)\n",
    "dct.compactify()\n",
    "t= time.time()\n",
    "print(t-t0)\n",
    "gensim_dict_filepath = os.path.join(\"data\",'gensim_dict_nostem.dict')\n",
    "#dct.save_as_text(gensim_dict_filepath)\n",
    "dct.save(gensim_dict_filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "gensim_dict_filepath = os.path.join(\"data\",'gensim_dict_nostem.dict')\n",
    "#dct = Dictionary.load_from_text(gensim_dict_filepath)\n",
    "dct = Dictionary.load(gensim_dict_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LDA implementation below is based on a tutorial by Patrick Harrison - https://youtu.be/6zm9NC9uRkk, accompanied by a Github notebook - https://github.com/skipgram/modern-nlp-in-python/blob/master/executable/Modern_NLP_in_Python.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bow_filepath = os.path.join('data','bow_corpus.mm')\n",
    "def bow_generator(filepath):\n",
    "    \"\"\"\n",
    "    generator function to read reviews from a file\n",
    "    and yield a bag-of-words representation\n",
    "    \"\"\"\n",
    "    with open(filepath, 'r') as readfile:\n",
    "        if 0==0:\n",
    "            for line in readfile:\n",
    "                d = json.loads(line)\n",
    "                text = d['text'].split()\n",
    "                yield dct.doc2bow(text)\n",
    "\n",
    "    #for review in LineSentence(filepath):\n",
    "        #yield trigram_dictionary.doc2bow(review)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 47.1 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "if 0 == 1:\n",
    "\n",
    "    # generate bag-of-words representations for\n",
    "    # all text and save them as a matrix\n",
    "    MmCorpus.serialize(bow_filepath,\n",
    "                       bow_generator('data/wiki_nostem.json'))\n",
    "    \n",
    "# load the finished bag-of-words corpus from disk\n",
    "bow_corpus = MmCorpus(bow_filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.59 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "lda_filepath = os.path.join('data', 'lda')\n",
    "if 1 == 0:\n",
    "\n",
    "    with warnings.catch_warnings():\n",
    "        warnings.simplefilter('ignore')\n",
    "        \n",
    "        # workers => sets the parallelism, and should be\n",
    "        # set to your number of physical cores minus one\n",
    "        lda = LdaMulticore(bow_corpus,\n",
    "                           num_topics=100,\n",
    "                           id2word=dct,\n",
    "                           workers=4)\n",
    "    \n",
    "    lda.save(lda_filepath)\n",
    "    \n",
    "# load the finished LDA model from disk\n",
    "lda = LdaMulticore.load(lda_filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def explore_topic(topic_number, topn=25):\n",
    "    \"\"\"\n",
    "    accept a user-supplied topic number and\n",
    "    print out a formatted list of the top terms\n",
    "    \"\"\"\n",
    "        \n",
    "    print( u'{:20} {}'.format(u'term', u'frequency') + u'\\n')\n",
    "\n",
    "    for term, frequency in lda.show_topic(topic_number, topn=25):\n",
    "        print (u'{:20} {:.3f}'.format(term, round(frequency, 5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "term                 frequency\n",
      "\n",
      "s                    0.024\n",
      "The                  0.016\n",
      "He                   0.007\n",
      "She                  0.005\n",
      "In                   0.005\n",
      "novel                0.004\n",
      "book                 0.004\n",
      "father               0.003\n",
      "time                 0.003\n",
      "It                   0.003\n",
      "family               0.003\n",
      "A                    0.003\n",
      "story                0.003\n",
      "life                 0.003\n",
      "I                    0.003\n",
      "death                0.002\n",
      "years                0.002\n",
      "later                0.002\n",
      "–                    0.002\n",
      "son                  0.002\n",
      "After                0.002\n",
      "mother               0.002\n",
      "published            0.002\n",
      "man                  0.002\n",
      "work                 0.002\n"
     ]
    }
   ],
   "source": [
    "explore_topic(topic_number=28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Astha\\Anaconda2\\envs\\tensorflow\\lib\\site-packages\\pyLDAvis\\_prepare.py:387: DeprecationWarning: \n",
      ".ix is deprecated. Please use\n",
      ".loc for label based indexing or\n",
      ".iloc for positional indexing\n",
      "\n",
      "See the documentation here:\n",
      "http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated\n",
      "  topic_term_dists = topic_term_dists.ix[topic_order]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 17min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "LDAvis_data_filepath = os.path.join('data', 'ldavis_prepared')\n",
    "# this is a bit time consuming - make the if statement True\n",
    "# if you want to execute data prep yourself.\n",
    "if 0 == 0:\n",
    "\n",
    "    LDAvis_prepared = pyLDAvis.gensim.prepare(lda, bow_corpus,\n",
    "                                              dct)\n",
    "\n",
    "    #with open(LDAvis_data_filepath, 'wb') as f:\n",
    "        #pickle.dump(LDAvis_prepared, f)\n",
    "        \n",
    "# load the pre-prepared pyLDAvis data from disk\n",
    "#with open(LDAvis_data_filepath) as f:\n",
    "    #LDAvis_prepared = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#pyLDAvis.display(LDAvis_prepared)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the visualization above, topic modeling seems like a very promising approach in separating out the articles. Specifically, in the visualization below, ![image.png](attachment:image.png) topics on the left are more social science, politics, literature related while topics on the right are primarily talk about more sciency articles. \n",
    "There are a large number of small topics in the center, these are harder to classify in one of the two categories - science or social science. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow]",
   "language": "python",
   "name": "conda-env-tensorflow-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}