Article Data Extraction and Removal of Irrelevant Pages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Article Data Extraction and Removal of Irrelevant Pages\n",
    "Author: Sean Flannery [sflanner@purdue.edu](sflanner@purdue.edu)\n",
    "\n",
    "Last Updated: June 14th, 2019\n",
    "\n",
    "This notebook was developed with the intent of satisfying local data parsing needs for work with\n",
    "Professor Daisuke Kihara [dkihara@purdue.edu](dkihara@purdue.edu).\n",
    "### Description\n",
    "We are interested in grabbing the Abstract, Author, & Introduction Data from a given set of article html files (assumed to have been already downloaded). \n",
    "\n",
    "We have provided a folder `articles` that contains the respective **years** and **\\*.html** files where we may find pertinent articles.\n",
    "\n",
    "**Libraries Needed:** \n",
    "[pandas](https://pandas.pydata.org/pandas-docs/stable/install.html), \n",
    "[numpy](https://www.numpy.org), \n",
    "[tqdm](https://github.com/tqdm/tqdm), \n",
    "[bs4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "import re\n",
    "import os\n",
    "import time\n",
    "import random\n",
    "random.seed(42)\n",
    "\n",
    "from bs4 import BeautifulSoup, SoupStrainer\n",
    "\n",
    "from multiprocessing import Pool\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "nar_df = pd.read_csv('article-path-data.csv',\n",
    "                     names = ['year','article-link', 'local-path', 'title', 'abstract', 'authors', 'introduction'],\n",
    "                     skiprows=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>article-link</th>\n",
       "      <th>local-path</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>introduction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1267</td>\n",
       "      <td>articles/2019/0-NAR.html</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky993</td>\n",
       "      <td>articles/2019/1-NAR.html</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1124</td>\n",
       "      <td>articles/2019/2-NAR.html</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1069</td>\n",
       "      <td>articles/2019/3-NAR.html</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky843</td>\n",
       "      <td>articles/2019/4-NAR.html</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year                         article-link                local-path  title  \\\n",
       "0  2019  https://doi.org/10.1093/nar/gky1267  articles/2019/0-NAR.html    NaN   \n",
       "1  2019   https://doi.org/10.1093/nar/gky993  articles/2019/1-NAR.html    NaN   \n",
       "2  2019  https://doi.org/10.1093/nar/gky1124  articles/2019/2-NAR.html    NaN   \n",
       "3  2019  https://doi.org/10.1093/nar/gky1069  articles/2019/3-NAR.html    NaN   \n",
       "4  2019   https://doi.org/10.1093/nar/gky843  articles/2019/4-NAR.html    NaN   \n",
       "\n",
       "   abstract  authors  introduction  \n",
       "0       NaN      NaN           NaN  \n",
       "1       NaN      NaN           NaN  \n",
       "2       NaN      NaN           NaN  \n",
       "3       NaN      NaN           NaN  \n",
       "4       NaN      NaN           NaN  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nar_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab the title of the article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getTitle(filename):\n",
    "    # Open html file for processing\n",
    "    f = open(filename)\n",
    "    soup = BeautifulSoup(f, 'html.parser')\n",
    "    f.close()\n",
    "    # Grab section of html named after the abstract\n",
    "    tmp = str(soup.findAll(\"h1\", {\"class\": \"article-title-main\"}))\n",
    "    soup = BeautifulSoup(tmp, 'html.parser')\n",
    "    # Clean up the text and store it in the result\n",
    "    return soup.get_text(strip=True)[1:-1] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab all text from the section with class abstract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getAbstract(filename):\n",
    "    # Open html file for processing\n",
    "    f = open(filename)\n",
    "    soup = BeautifulSoup(f, 'html.parser')\n",
    "    f.close()\n",
    "    # Grab section of html named after the abstract\n",
    "    tmp = str(soup.findAll(\"section\", {\"class\": \"abstract\"}))\n",
    "    soup = BeautifulSoup(tmp, 'html.parser')\n",
    "    # Clean up the text and store it in the result\n",
    "    return soup.get_text(strip=True)[1:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab the information inside the info card at the top of the webpage. There are 2 methods to use here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getAuthor(filename):\n",
    "    f = open(filename)\n",
    "    soup = BeautifulSoup(f, 'html.parser')\n",
    "    f.close()\n",
    "    tmp = str(soup.findAll(\"div\", {\"class\": \"info-card-name\"}))\n",
    "    if len(tmp) == 0:\n",
    "        tmp = soup.findAll(\"div\", {\"class\": \"meta-authors--limited\"})\n",
    "    if len(tmp) == 0:\n",
    "        tmp = soup.findAll(\"div\", {\"class\": \"wi-authors\"})\n",
    "    soup = BeautifulSoup(str(tmp), 'html.parser')\n",
    "    return soup.get_text(strip=True)[1:-1].split(',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grab the first `<h2>` html tag that indicates a section title on the page, confirm that it contains the words \"introduction\", and then return it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getIntroductionFromHeaders(filename):\n",
    "    f = open(filename)\n",
    "    soup = BeautifulSoup(f, 'html.parser')\n",
    "    f.close()\n",
    "    # Seek to the first header\n",
    "    found = soup.findAll(\"h2\", {\"class\": \"section-title\"})\n",
    "    # Initially assume that neither intro nor the section after is found\n",
    "    intro = None\n",
    "    after = None\n",
    "    # If we find only 1 header, it's likely just the acknowledgements\n",
    "    if len(found) <= 1:\n",
    "        return None\n",
    "    else:\n",
    "        found_intro = False\n",
    "        for ent in found:\n",
    "            if found_intro:\n",
    "                after = ent\n",
    "                break\n",
    "            if 'introduction' in str(ent).lower():\n",
    "                intro = ent\n",
    "        # Start at the first paragraph containing h2\n",
    "        intro = found[0]\n",
    "        # End before the next h2 section (AKA the next section of text -- likely not introduction)\n",
    "        after = found[1]\n",
    "    \n",
    "    resStr = ''\n",
    "    # If we were not able to identify individual sections\n",
    "    # just take anything in paragraph tags in the article body\n",
    "    if intro is None or after is None:\n",
    "        res['introduction'] = None\n",
    "        return res\n",
    "    else:      \n",
    "        next_p = intro.find_next(\"p\")\n",
    "        last_p = after.find_previous_sibling(\"p\")\n",
    "        if last_p is None:\n",
    "            return None\n",
    "        last_p = last_p.get_text(strip=True)\n",
    "        if last_p is None or next_p is None:\n",
    "            return None\n",
    "        while str(next_p.get_text(strip=True)) != str(last_p):\n",
    "            tmpStr = str(next_p.get_text(strip=True))\n",
    "            resStr += tmpStr + '\\n'\n",
    "            next_p = next_p.find_next(\"p\")\n",
    "        resStr += str(last_p)\n",
    "    return resStr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to handle the results with no intro now with a deeper level of granularity. This method focuses on just grabbing as many `<p>` tags as possible.\n",
    "\n",
    "This also requires us to **define some number of paragraphs** to grab. We initially just grab up to 3 paragraphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "PARAGRAPH_NUMBER = 3\n",
    "def getIntroductionFromParagraphs(filename):\n",
    "    abstract = getAbstract(filename)\n",
    "    f = open(filename)\n",
    "    soup = BeautifulSoup(f, 'html.parser')\n",
    "    f.close()\n",
    "    # Seek to the first header\n",
    "    resStr = ''  \n",
    "    content_blocks = soup.findAll(\"div\", {\"id\": \"ContentTab\"})\n",
    "    if content_blocks is None or len(content_blocks) == 0:\n",
    "        return None\n",
    "    content_block = content_blocks[0]\n",
    "    soup = BeautifulSoup(str(content_block), 'html.parser')\n",
    "    pars = soup.findAll(\"p\")\n",
    "    if len(pars) == 0:\n",
    "        return None\n",
    "    \n",
    "    parCount = 0\n",
    "    next_p = str(pars[parCount].get_text(strip=True))\n",
    "    parCount += 1\n",
    "    while not abstract in next_p and parCount < len(pars):\n",
    "        next_p = str(pars[parCount].get_text(strip=True))\n",
    "        parCount += 1\n",
    "    counter = 0\n",
    "    while next_p is not None and counter < PARAGRAPH_NUMBER and parCount < len(pars):\n",
    "        counter = counter + 1\n",
    "        next_p = str(pars[parCount].get_text(strip=True))\n",
    "        parCount += 1 \n",
    "        resStr += str(next_p + '\\n')\n",
    "    if resStr == '':\n",
    "        return None\n",
    "    else:\n",
    "        return resStr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getIntroduction(filename):\n",
    "    res = getIntroductionFromHeaders(filename)\n",
    "    if res is None or len(res) < 4:\n",
    "        res = getIntroductionFromParagraphs(filename)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to grab a url's various page information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getWebPageData(urlID):\n",
    "    res = dict()    \n",
    "    res['urlID'] = urlID\n",
    "    filename = str(nar_df.loc[urlID,'local-path'])\n",
    "    ### Title \n",
    "    res['title'] = getTitle(filename)\n",
    "    ### Abstract\n",
    "    res['abstract'] = getAbstract(filename)\n",
    "    ### Authors\n",
    "    res['authors'] = getAuthor(filename)\n",
    "    ### Introduction\n",
    "    res['introduction'] = getIntroductionFromHeaders(filename)\n",
    "    if res['introduction'] is None or len(res['introduction']) <=2:\n",
    "        res['introduction'] = getIntroductionFromParagraphs(filename)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate several processes to do these processes concurrently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=3149), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    }
   ],
   "source": [
    "req_list = []\n",
    "entry_range = range(len(nar_df))\n",
    "with Pool(8) as p:\n",
    "    req_list = list(tqdm(p.imap(getWebPageData, entry_range), total=len(entry_range), leave=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now going to transfer our returned results to the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick fix for authors column so it accepts valid datatype\n",
    "nar_df['authors'] = nar_df['authors'].astype(object)\n",
    "\n",
    "for res in req_list:\n",
    "    urlID = res['urlID']\n",
    "    nar_df.loc[urlID, 'title'] = res['title']\n",
    "    nar_df.loc[urlID, 'abstract'] = res['abstract']\n",
    "    nar_df.at[urlID,'authors'] = res['authors']\n",
    "    nar_df.loc[urlID, 'introduction'] = res['introduction']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's store the original version of the nar database before we drop entries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>article-link</th>\n",
       "      <th>local-path</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>introduction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1267</td>\n",
       "      <td>articles/2019/0-NAR.html</td>\n",
       "      <td>The 26th annual Nucleic Acids Research databas...</td>\n",
       "      <td>The 2019 Nucleic Acids Research (NAR) Database...</td>\n",
       "      <td>[Daniel J Rigden, Xosé M Fernández]</td>\n",
       "      <td>The Nucleic Acids Research (NAR) Database Issu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky993</td>\n",
       "      <td>articles/2019/1-NAR.html</td>\n",
       "      <td>Database Resources of the BIG Data Center in 2019</td>\n",
       "      <td>The BIG Data Center at Beijing Institute of Ge...</td>\n",
       "      <td>[BIG Data Center Members]</td>\n",
       "      <td>The BIG Data Center (http://bigd.big.ac.cn) at...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1124</td>\n",
       "      <td>articles/2019/2-NAR.html</td>\n",
       "      <td>The European Bioinformatics Institute in 2018:...</td>\n",
       "      <td>The European Bioinformatics Institute (https:/...</td>\n",
       "      <td>[Charles E Cook, Rodrigo Lopez, Oana Stroe, Gu...</td>\n",
       "      <td>A primary mission of EMBL-EBI is to collect, o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1069</td>\n",
       "      <td>articles/2019/3-NAR.html</td>\n",
       "      <td>Database resources of the National Center for ...</td>\n",
       "      <td>The National Center for Biotechnology Informat...</td>\n",
       "      <td>[Eric W Sayers, Richa Agarwala, Evan E Bolton,...</td>\n",
       "      <td>The National Center for Biotechnology Informat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky843</td>\n",
       "      <td>articles/2019/4-NAR.html</td>\n",
       "      <td>AmtDB: a database of ancient human mitochondri...</td>\n",
       "      <td>Ancient mitochondrial DNA is used for tracing ...</td>\n",
       "      <td>[Edvard Ehler, Jiří Novotný, Anna Juras, Macie...</td>\n",
       "      <td>Ancient DNA (aDNA) is a genetic material obtai...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year                         article-link                local-path  \\\n",
       "0  2019  https://doi.org/10.1093/nar/gky1267  articles/2019/0-NAR.html   \n",
       "1  2019   https://doi.org/10.1093/nar/gky993  articles/2019/1-NAR.html   \n",
       "2  2019  https://doi.org/10.1093/nar/gky1124  articles/2019/2-NAR.html   \n",
       "3  2019  https://doi.org/10.1093/nar/gky1069  articles/2019/3-NAR.html   \n",
       "4  2019   https://doi.org/10.1093/nar/gky843  articles/2019/4-NAR.html   \n",
       "\n",
       "                                               title  \\\n",
       "0  The 26th annual Nucleic Acids Research databas...   \n",
       "1  Database Resources of the BIG Data Center in 2019   \n",
       "2  The European Bioinformatics Institute in 2018:...   \n",
       "3  Database resources of the National Center for ...   \n",
       "4  AmtDB: a database of ancient human mitochondri...   \n",
       "\n",
       "                                            abstract  \\\n",
       "0  The 2019 Nucleic Acids Research (NAR) Database...   \n",
       "1  The BIG Data Center at Beijing Institute of Ge...   \n",
       "2  The European Bioinformatics Institute (https:/...   \n",
       "3  The National Center for Biotechnology Informat...   \n",
       "4  Ancient mitochondrial DNA is used for tracing ...   \n",
       "\n",
       "                                             authors  \\\n",
       "0                [Daniel J Rigden, Xosé M Fernández]   \n",
       "1                          [BIG Data Center Members]   \n",
       "2  [Charles E Cook, Rodrigo Lopez, Oana Stroe, Gu...   \n",
       "3  [Eric W Sayers, Richa Agarwala, Evan E Bolton,...   \n",
       "4  [Edvard Ehler, Jiří Novotný, Anna Juras, Macie...   \n",
       "\n",
       "                                        introduction  \n",
       "0  The Nucleic Acids Research (NAR) Database Issu...  \n",
       "1  The BIG Data Center (http://bigd.big.ac.cn) at...  \n",
       "2  A primary mission of EMBL-EBI is to collect, o...  \n",
       "3  The National Center for Biotechnology Informat...  \n",
       "4  Ancient DNA (aDNA) is a genetic material obtai...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_nar_df = nar_df.copy()\n",
    "nar_df.to_csv('original-article-data.csv', index=False)\n",
    "nar_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display Statistics!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No introduction 11/3149: 0.35%\n",
      "No abstracts 0/3149: 0.00%\n",
      "Intro contains abstract 18/3138: 0.57%\n"
     ]
    }
   ],
   "source": [
    "no_abstracts = nar_df[nar_df.abstract.isna()]\n",
    "no_intro = nar_df[nar_df.introduction.isna()]\n",
    "has_intro = nar_df[nar_df.introduction.notna()]\n",
    "        \n",
    "print(\"No introduction %s/%s: %.2f%%\" % (len(no_intro), len(nar_df), (100. * len(no_intro))/ len(nar_df)))\n",
    "print(\"No abstracts %s/%s: %.2f%%\" % (len(no_abstracts), len(nar_df), (100. * len(no_abstracts))/ len(nar_df)))\n",
    "count = 0\n",
    "for uid in has_intro.index:\n",
    "    introStr = str(nar_df.loc[uid, 'introduction'])\n",
    "    abstractStr = str(nar_df.loc[uid, 'abstract'])\n",
    "    if abstractStr in introStr:\n",
    "        count = count + 1\n",
    "print(\"Intro contains abstract %s/%s: %.2f%%\" % (count, len(has_intro), (100. * count / len(has_intro))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Cleaning\n",
    "\n",
    "We want to drop all unneeded pages based on title or filetype.\n",
    "\n",
    "Drop entries leading to a PDF file (likely a pdf saying \"under development\", but nevertheless unscrapable)\n",
    "\n",
    "We include a method to check if the downloaded file is a PDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def checkPDF(filename):\n",
    "    f = open(filename)\n",
    "    file_head = f.read(4)\n",
    "    f.close()\n",
    "    return \"%PDF\" == file_head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index, row in nar_df.iterrows():\n",
    "    # Remove all editorial, subscription pages\n",
    "    if 'editorial' in row['title'].lower() or 'subscription' in row['title'].lower():\n",
    "        nar_df.drop(index, inplace=True)\n",
    "    # Drop all files that are just PDFs\n",
    "    if checkPDF(row['local-path']):\n",
    "        nar_df.drop(index, inplace=True)\n",
    "    # Remove all database issues\n",
    "    if 'database issue' in row['title'].lower():\n",
    "        nar_df.drop(index, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No introduction 1/3113: 0.03%\n",
      "No abstracts 0/3113: 0.00%\n",
      "Intro contains abstract 0/3112: 0.00%\n"
     ]
    }
   ],
   "source": [
    "no_abstracts = nar_df[nar_df.abstract.isna()]\n",
    "no_intro = nar_df[nar_df.introduction.isna()]\n",
    "has_intro = nar_df[nar_df.introduction.notna()]\n",
    "        \n",
    "print(\"No introduction %s/%s: %.2f%%\" % (len(no_intro), len(nar_df), (100. * len(no_intro))/ len(nar_df)))\n",
    "print(\"No abstracts %s/%s: %.2f%%\" % (len(no_abstracts), len(nar_df), (100. * len(no_abstracts))/ len(nar_df)))\n",
    "count = 0\n",
    "for uid in nar_df.index:\n",
    "    introStr = str(nar_df.loc[uid, 'introduction'])\n",
    "    abstractStr = str(nar_df.loc[uid, 'abstract'])\n",
    "    # Check if abstractStr isn't empty before seeing if its in the intro\n",
    "    if abstractStr in introStr and len(abstractStr) > 0:\n",
    "        count = count + 1\n",
    "print(\"Intro contains abstract %s/%s: %.2f%%\" % (count, len(has_intro), (100. * count / len(has_intro))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop undesirable attributes (those that don't have the data we need)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "for uid in nar_df.index:\n",
    "# Drop the undesired entries\n",
    "    introStr = str(nar_df.loc[uid, 'introduction'])\n",
    "    abstractStr = str(nar_df.loc[uid, 'abstract'])\n",
    "    if len(abstractStr) == 0 or len(introStr) == 0:\n",
    "        nar_df.drop(uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we should **ensure we haven't dropped articles erroneously by spot-checking the articles we've dropped**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total dropped entries: 36\n"
     ]
    }
   ],
   "source": [
    "difference = original_nar_df[~original_nar_df.index.isin(nar_df.index)]\n",
    "print(\"Total dropped entries:\", len(difference))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should run through these entries and see if there are any we ought to restore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>article-link</th>\n",
       "      <th>local-path</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>introduction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1267</td>\n",
       "      <td>articles/2019/0-NAR.html</td>\n",
       "      <td>The 26th annual Nucleic Acids Research databas...</td>\n",
       "      <td>The 2019 Nucleic Acids Research (NAR) Database...</td>\n",
       "      <td>[Daniel J Rigden, Xosé M Fernández]</td>\n",
       "      <td>The Nucleic Acids Research (NAR) Database Issu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky1050</td>\n",
       "      <td>articles/2019/31-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>2019</td>\n",
       "      <td>https://doi.org/10.1093/nar/gky992</td>\n",
       "      <td>articles/2019/85-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>2018</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkx1235</td>\n",
       "      <td>articles/2018/150-NAR.html</td>\n",
       "      <td>The 2018Nucleic Acids Researchdatabase issue a...</td>\n",
       "      <td>The 2018Nucleic Acids ResearchDatabase Issue c...</td>\n",
       "      <td>[Daniel J Rigden, Xosé M Fernández]</td>\n",
       "      <td>This 2018Nucleic Acids ResearchDatabase Issue ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>2018</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkx897</td>\n",
       "      <td>articles/2018/152-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>2018</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkx1097</td>\n",
       "      <td>articles/2018/157-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>2018</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkx864</td>\n",
       "      <td>articles/2018/179-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>262</th>\n",
       "      <td>2018</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkx1020</td>\n",
       "      <td>articles/2018/262-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>2018</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkx892</td>\n",
       "      <td>articles/2018/298-NAR.html</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>2017</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkw1188</td>\n",
       "      <td>articles/2017/300-NAR.html</td>\n",
       "      <td>The 24th annualNucleic Acids Researchdatabase ...</td>\n",
       "      <td>This year's Database Issue ofNucleic Acids Res...</td>\n",
       "      <td>[Michael Y Galperin, Xosé M Fernández-Suárez, ...</td>\n",
       "      <td>The current 2017Nucleic Acids ResearchDatabase...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>450</th>\n",
       "      <td>2016</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkv1356</td>\n",
       "      <td>articles/2016/450-NAR.html</td>\n",
       "      <td>The 2016 database issue ofNucleic Acids Resear...</td>\n",
       "      <td>The 2016 Database Issue ofNucleic Acids Resear...</td>\n",
       "      <td>[Daniel J. Rigden, Xosé M. Fernández-Suárez, M...</td>\n",
       "      <td>The 2016Nucleic Acids ResearchDatabase Issue i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>600</th>\n",
       "      <td>2015</td>\n",
       "      <td>https://doi.org/10.1093/nar/gku1241</td>\n",
       "      <td>articles/2015/600-NAR.html</td>\n",
       "      <td>The 2015Nucleic Acids ResearchDatabase Issue a...</td>\n",
       "      <td>The 2015Nucleic Acids ResearchDatabase Issue c...</td>\n",
       "      <td>[Michael Y. Galperin, Daniel J. Rigden, Xosé M...</td>\n",
       "      <td>The currentNucleic Acids Research(NAR) Databas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>750</th>\n",
       "      <td>2014</td>\n",
       "      <td>https://doi.org/10.1093/nar/gks1388</td>\n",
       "      <td>articles/2014/750-NAR.html</td>\n",
       "      <td>Nucleic Acids Research: Volume 42 Database Iss...</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>ONLINE ISSN: 1362-4962\\nwww.nar.oxfordjournals...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>751</th>\n",
       "      <td>2014</td>\n",
       "      <td>https://doi.org/10.1093/nar/gks1412</td>\n",
       "      <td>articles/2014/751-NAR.html</td>\n",
       "      <td>Nucleic Acids Research - Editorial Board</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>Senior Editors: K.R.Fox,Southampton, UK; foxna...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>752</th>\n",
       "      <td>2014</td>\n",
       "      <td>https://doi.org/10.1093/nar/gks1436</td>\n",
       "      <td>articles/2014/752-NAR.html</td>\n",
       "      <td>Subscriptions</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>Annual Subscription Rate (Volume 42, 22 issues...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>753</th>\n",
       "      <td>2014</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkt1282</td>\n",
       "      <td>articles/2014/753-NAR.html</td>\n",
       "      <td>The 2014Nucleic Acids ResearchDatabase Issue a...</td>\n",
       "      <td>The 2014 Nucleic Acids Research Database Issue...</td>\n",
       "      <td>[Xosé M. Fernández-Suárez, Daniel J. Rigden, M...</td>\n",
       "      <td>The 21st annual Nucleic Acids Research Databas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>900</th>\n",
       "      <td>2013</td>\n",
       "      <td>https://doi.org/10.1093/nar/gks1297</td>\n",
       "      <td>articles/2013/900-NAR.html</td>\n",
       "      <td>The 2013Nucleic Acids ResearchDatabase Issue a...</td>\n",
       "      <td>The 20th annual Database Issue ofNucleic Acids...</td>\n",
       "      <td>[Xosé M. Fernández-Suárez, Michael Y. Galperin]</td>\n",
       "      <td>This 1300-page virtual volume represents the 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1050</th>\n",
       "      <td>2012</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkr1196</td>\n",
       "      <td>articles/2012/1050-NAR.html</td>\n",
       "      <td>The 2012 Nucleic Acids Research Database Issue...</td>\n",
       "      <td>The 19th annual Database Issue ofNucleic Acids...</td>\n",
       "      <td>[Michael Y. Galperin, Xosé M. Fernández-Suárez]</td>\n",
       "      <td>This current, 19th annual Database Issue ofNuc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1201</th>\n",
       "      <td>2011</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkq1331</td>\n",
       "      <td>articles/2011/1201-NAR.html</td>\n",
       "      <td>Nucleic Acids Research: VOLUME 39 DATABASE ISS...</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>online issn: 1362-4962\\nwww.nar.oxfordjournals...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1202</th>\n",
       "      <td>2011</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkq1332</td>\n",
       "      <td>articles/2011/1202-NAR.html</td>\n",
       "      <td>Subscriptions</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>Annual Subscription Rate (Volume 38, 22 issues...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1317</th>\n",
       "      <td>2011</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkq1243</td>\n",
       "      <td>articles/2011/1317-NAR.html</td>\n",
       "      <td>The 2011Nucleic Acids ResearchDatabase Issue a...</td>\n",
       "      <td>The current 18th Database Issue ofNucleic Acid...</td>\n",
       "      <td>[Michael Y. Galperin, Guy R. Cochrane]</td>\n",
       "      <td>This current, 18th annual Database Issue ofNuc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1350</th>\n",
       "      <td>2010</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkp1077</td>\n",
       "      <td>articles/2010/1350-NAR.html</td>\n",
       "      <td>The 2010Nucleic Acids ResearchDatabase Issue a...</td>\n",
       "      <td>The current issue ofNucleic Acids Researchincl...</td>\n",
       "      <td>[Guy R. Cochrane, Michael Y. Galperin]</td>\n",
       "      <td>This Database Issue ofNucleic Acids Research(N...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1485</th>\n",
       "      <td>2010</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkp1212</td>\n",
       "      <td>articles/2010/1485-NAR.html</td>\n",
       "      <td>Nucleic Acids Research: VOLUME 38 DATABASE ISS...</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>\\nNow Open Access\\nNo barriers to access – all...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1487</th>\n",
       "      <td>2010</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkp1213</td>\n",
       "      <td>articles/2010/1487-NAR.html</td>\n",
       "      <td>Subscriptions</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>Annual Subscription Rate (Volume 38, 22 issues...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1488</th>\n",
       "      <td>2009</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkn942</td>\n",
       "      <td>articles/2009/1488-NAR.html</td>\n",
       "      <td>Nucleic Acids Researchannual Database Issue an...</td>\n",
       "      <td>The current issue ofNucleic Acids Researchincl...</td>\n",
       "      <td>[Michael Y. Galperin, Guy R. Cochrane]</td>\n",
       "      <td>The 2009Nucleic Acids Research(NAR) annual Dat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1638</th>\n",
       "      <td>2008</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkm1139</td>\n",
       "      <td>articles/2008/1638-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[Alex Bateman]</td>\n",
       "      <td>Michael Galperin has continued to produce and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1788</th>\n",
       "      <td>2007</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkl1051</td>\n",
       "      <td>articles/2007/1788-NAR.html</td>\n",
       "      <td>EDITORIAL</td>\n",
       "      <td></td>\n",
       "      <td>[Alex Batemen]</td>\n",
       "      <td>Michael Galperin continues to produce and enla...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1938</th>\n",
       "      <td>2006</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkj150</td>\n",
       "      <td>articles/2006/1938-NAR.html</td>\n",
       "      <td>EDITORIAL</td>\n",
       "      <td></td>\n",
       "      <td>[Alex Bateman]</td>\n",
       "      <td>Michael Galperin continues to produce and enla...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2088</th>\n",
       "      <td>2005</td>\n",
       "      <td>https://doi.org/10.1093/nar/gki133</td>\n",
       "      <td>articles/2005/2088-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[Alex Bateman]</td>\n",
       "      <td>Michael Galperin has produced the Molecular Bi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2089</th>\n",
       "      <td>2005</td>\n",
       "      <td>https://doi.org/10.1093/nar/gki134</td>\n",
       "      <td>articles/2005/2089-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[Dov Greenbaum, Andrew Smith, Mark Gerstein]</td>\n",
       "      <td>The significant growth of this annual over the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2227</th>\n",
       "      <td>2004</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkh142</td>\n",
       "      <td>articles/2004/2227-NAR.html</td>\n",
       "      <td>EDITORIAL</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>\\nFor several years, Andy Baxevanis has been p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2370</th>\n",
       "      <td>2003</td>\n",
       "      <td>https://doi.org/10.1093/nar/gkg173</td>\n",
       "      <td>articles/2003/2370-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>At a recent meeting of the Editorial Board ofN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2501</th>\n",
       "      <td>2002</td>\n",
       "      <td>https://doi.org/10.1093/nar/30.1.0</td>\n",
       "      <td>articles/2002/2501-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2615</th>\n",
       "      <td>2001</td>\n",
       "      <td>https://doi.org/10.1093/nar/29.1.0</td>\n",
       "      <td>articles/2001/2615-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2713</th>\n",
       "      <td>2000</td>\n",
       "      <td>https://doi.org/10.1093/nar/28.1.0</td>\n",
       "      <td>articles/2000/2713-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2829</th>\n",
       "      <td>1999</td>\n",
       "      <td>https://doi.org/10.1093/nar/27.1.i</td>\n",
       "      <td>articles/1999/2829-NAR.html</td>\n",
       "      <td>Editorial</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>The deadlines for submission of articles inten...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      year                         article-link                   local-path  \\\n",
       "0     2019  https://doi.org/10.1093/nar/gky1267     articles/2019/0-NAR.html   \n",
       "31    2019  https://doi.org/10.1093/nar/gky1050    articles/2019/31-NAR.html   \n",
       "85    2019   https://doi.org/10.1093/nar/gky992    articles/2019/85-NAR.html   \n",
       "150   2018  https://doi.org/10.1093/nar/gkx1235   articles/2018/150-NAR.html   \n",
       "152   2018   https://doi.org/10.1093/nar/gkx897   articles/2018/152-NAR.html   \n",
       "157   2018  https://doi.org/10.1093/nar/gkx1097   articles/2018/157-NAR.html   \n",
       "179   2018   https://doi.org/10.1093/nar/gkx864   articles/2018/179-NAR.html   \n",
       "262   2018  https://doi.org/10.1093/nar/gkx1020   articles/2018/262-NAR.html   \n",
       "298   2018   https://doi.org/10.1093/nar/gkx892   articles/2018/298-NAR.html   \n",
       "300   2017  https://doi.org/10.1093/nar/gkw1188   articles/2017/300-NAR.html   \n",
       "450   2016  https://doi.org/10.1093/nar/gkv1356   articles/2016/450-NAR.html   \n",
       "600   2015  https://doi.org/10.1093/nar/gku1241   articles/2015/600-NAR.html   \n",
       "750   2014  https://doi.org/10.1093/nar/gks1388   articles/2014/750-NAR.html   \n",
       "751   2014  https://doi.org/10.1093/nar/gks1412   articles/2014/751-NAR.html   \n",
       "752   2014  https://doi.org/10.1093/nar/gks1436   articles/2014/752-NAR.html   \n",
       "753   2014  https://doi.org/10.1093/nar/gkt1282   articles/2014/753-NAR.html   \n",
       "900   2013  https://doi.org/10.1093/nar/gks1297   articles/2013/900-NAR.html   \n",
       "1050  2012  https://doi.org/10.1093/nar/gkr1196  articles/2012/1050-NAR.html   \n",
       "1201  2011  https://doi.org/10.1093/nar/gkq1331  articles/2011/1201-NAR.html   \n",
       "1202  2011  https://doi.org/10.1093/nar/gkq1332  articles/2011/1202-NAR.html   \n",
       "1317  2011  https://doi.org/10.1093/nar/gkq1243  articles/2011/1317-NAR.html   \n",
       "1350  2010  https://doi.org/10.1093/nar/gkp1077  articles/2010/1350-NAR.html   \n",
       "1485  2010  https://doi.org/10.1093/nar/gkp1212  articles/2010/1485-NAR.html   \n",
       "1487  2010  https://doi.org/10.1093/nar/gkp1213  articles/2010/1487-NAR.html   \n",
       "1488  2009   https://doi.org/10.1093/nar/gkn942  articles/2009/1488-NAR.html   \n",
       "1638  2008  https://doi.org/10.1093/nar/gkm1139  articles/2008/1638-NAR.html   \n",
       "1788  2007  https://doi.org/10.1093/nar/gkl1051  articles/2007/1788-NAR.html   \n",
       "1938  2006   https://doi.org/10.1093/nar/gkj150  articles/2006/1938-NAR.html   \n",
       "2088  2005   https://doi.org/10.1093/nar/gki133  articles/2005/2088-NAR.html   \n",
       "2089  2005   https://doi.org/10.1093/nar/gki134  articles/2005/2089-NAR.html   \n",
       "2227  2004   https://doi.org/10.1093/nar/gkh142  articles/2004/2227-NAR.html   \n",
       "2370  2003   https://doi.org/10.1093/nar/gkg173  articles/2003/2370-NAR.html   \n",
       "2501  2002   https://doi.org/10.1093/nar/30.1.0  articles/2002/2501-NAR.html   \n",
       "2615  2001   https://doi.org/10.1093/nar/29.1.0  articles/2001/2615-NAR.html   \n",
       "2713  2000   https://doi.org/10.1093/nar/28.1.0  articles/2000/2713-NAR.html   \n",
       "2829  1999   https://doi.org/10.1093/nar/27.1.i  articles/1999/2829-NAR.html   \n",
       "\n",
       "                                                  title  \\\n",
       "0     The 26th annual Nucleic Acids Research databas...   \n",
       "31                                                        \n",
       "85                                                        \n",
       "150   The 2018Nucleic Acids Researchdatabase issue a...   \n",
       "152                                                       \n",
       "157                                                       \n",
       "179                                                       \n",
       "262                                                       \n",
       "298                                                       \n",
       "300   The 24th annualNucleic Acids Researchdatabase ...   \n",
       "450   The 2016 database issue ofNucleic Acids Resear...   \n",
       "600   The 2015Nucleic Acids ResearchDatabase Issue a...   \n",
       "750   Nucleic Acids Research: Volume 42 Database Iss...   \n",
       "751            Nucleic Acids Research - Editorial Board   \n",
       "752                                       Subscriptions   \n",
       "753   The 2014Nucleic Acids ResearchDatabase Issue a...   \n",
       "900   The 2013Nucleic Acids ResearchDatabase Issue a...   \n",
       "1050  The 2012 Nucleic Acids Research Database Issue...   \n",
       "1201  Nucleic Acids Research: VOLUME 39 DATABASE ISS...   \n",
       "1202                                      Subscriptions   \n",
       "1317  The 2011Nucleic Acids ResearchDatabase Issue a...   \n",
       "1350  The 2010Nucleic Acids ResearchDatabase Issue a...   \n",
       "1485  Nucleic Acids Research: VOLUME 38 DATABASE ISS...   \n",
       "1487                                      Subscriptions   \n",
       "1488  Nucleic Acids Researchannual Database Issue an...   \n",
       "1638                                          Editorial   \n",
       "1788                                          EDITORIAL   \n",
       "1938                                          EDITORIAL   \n",
       "2088                                          Editorial   \n",
       "2089                                          Editorial   \n",
       "2227                                          EDITORIAL   \n",
       "2370                                          Editorial   \n",
       "2501                                          Editorial   \n",
       "2615                                          Editorial   \n",
       "2713                                          Editorial   \n",
       "2829                                          Editorial   \n",
       "\n",
       "                                               abstract  \\\n",
       "0     The 2019 Nucleic Acids Research (NAR) Database...   \n",
       "31                                                        \n",
       "85                                                        \n",
       "150   The 2018Nucleic Acids ResearchDatabase Issue c...   \n",
       "152                                                       \n",
       "157                                                       \n",
       "179                                                       \n",
       "262                                                       \n",
       "298                                                       \n",
       "300   This year's Database Issue ofNucleic Acids Res...   \n",
       "450   The 2016 Database Issue ofNucleic Acids Resear...   \n",
       "600   The 2015Nucleic Acids ResearchDatabase Issue c...   \n",
       "750                                                       \n",
       "751                                                       \n",
       "752                                                       \n",
       "753   The 2014 Nucleic Acids Research Database Issue...   \n",
       "900   The 20th annual Database Issue ofNucleic Acids...   \n",
       "1050  The 19th annual Database Issue ofNucleic Acids...   \n",
       "1201                                                      \n",
       "1202                                                      \n",
       "1317  The current 18th Database Issue ofNucleic Acid...   \n",
       "1350  The current issue ofNucleic Acids Researchincl...   \n",
       "1485                                                      \n",
       "1487                                                      \n",
       "1488  The current issue ofNucleic Acids Researchincl...   \n",
       "1638                                                      \n",
       "1788                                                      \n",
       "1938                                                      \n",
       "2088                                                      \n",
       "2089                                                      \n",
       "2227                                                      \n",
       "2370                                                      \n",
       "2501                                                      \n",
       "2615                                                      \n",
       "2713                                                      \n",
       "2829                                                      \n",
       "\n",
       "                                                authors  \\\n",
       "0                   [Daniel J Rigden, Xosé M Fernández]   \n",
       "31                                                   []   \n",
       "85                                                   []   \n",
       "150                 [Daniel J Rigden, Xosé M Fernández]   \n",
       "152                                                  []   \n",
       "157                                                  []   \n",
       "179                                                  []   \n",
       "262                                                  []   \n",
       "298                                                  []   \n",
       "300   [Michael Y Galperin, Xosé M Fernández-Suárez, ...   \n",
       "450   [Daniel J. Rigden, Xosé M. Fernández-Suárez, M...   \n",
       "600   [Michael Y. Galperin, Daniel J. Rigden, Xosé M...   \n",
       "750                                                  []   \n",
       "751                                                  []   \n",
       "752                                                  []   \n",
       "753   [Xosé M. Fernández-Suárez, Daniel J. Rigden, M...   \n",
       "900     [Xosé M. Fernández-Suárez, Michael Y. Galperin]   \n",
       "1050    [Michael Y. Galperin, Xosé M. Fernández-Suárez]   \n",
       "1201                                                 []   \n",
       "1202                                                 []   \n",
       "1317             [Michael Y. Galperin, Guy R. Cochrane]   \n",
       "1350             [Guy R. Cochrane, Michael Y. Galperin]   \n",
       "1485                                                 []   \n",
       "1487                                                 []   \n",
       "1488             [Michael Y. Galperin, Guy R. Cochrane]   \n",
       "1638                                     [Alex Bateman]   \n",
       "1788                                     [Alex Batemen]   \n",
       "1938                                     [Alex Bateman]   \n",
       "2088                                     [Alex Bateman]   \n",
       "2089       [Dov Greenbaum, Andrew Smith, Mark Gerstein]   \n",
       "2227                                                 []   \n",
       "2370                                                 []   \n",
       "2501                                                 []   \n",
       "2615                                                 []   \n",
       "2713                                                 []   \n",
       "2829                                                 []   \n",
       "\n",
       "                                           introduction  \n",
       "0     The Nucleic Acids Research (NAR) Database Issu...  \n",
       "31                                                 None  \n",
       "85                                                 None  \n",
       "150   This 2018Nucleic Acids ResearchDatabase Issue ...  \n",
       "152                                                None  \n",
       "157                                                None  \n",
       "179                                                None  \n",
       "262                                                None  \n",
       "298                                                None  \n",
       "300   The current 2017Nucleic Acids ResearchDatabase...  \n",
       "450   The 2016Nucleic Acids ResearchDatabase Issue i...  \n",
       "600   The currentNucleic Acids Research(NAR) Databas...  \n",
       "750   ONLINE ISSN: 1362-4962\\nwww.nar.oxfordjournals...  \n",
       "751   Senior Editors: K.R.Fox,Southampton, UK; foxna...  \n",
       "752   Annual Subscription Rate (Volume 42, 22 issues...  \n",
       "753   The 21st annual Nucleic Acids Research Databas...  \n",
       "900   This 1300-page virtual volume represents the 2...  \n",
       "1050  This current, 19th annual Database Issue ofNuc...  \n",
       "1201  online issn: 1362-4962\\nwww.nar.oxfordjournals...  \n",
       "1202  Annual Subscription Rate (Volume 38, 22 issues...  \n",
       "1317  This current, 18th annual Database Issue ofNuc...  \n",
       "1350  This Database Issue ofNucleic Acids Research(N...  \n",
       "1485  \\nNow Open Access\\nNo barriers to access – all...  \n",
       "1487  Annual Subscription Rate (Volume 38, 22 issues...  \n",
       "1488  The 2009Nucleic Acids Research(NAR) annual Dat...  \n",
       "1638  Michael Galperin has continued to produce and ...  \n",
       "1788  Michael Galperin continues to produce and enla...  \n",
       "1938  Michael Galperin continues to produce and enla...  \n",
       "2088  Michael Galperin has produced the Molecular Bi...  \n",
       "2089  The significant growth of this annual over the...  \n",
       "2227  \\nFor several years, Andy Baxevanis has been p...  \n",
       "2370  At a recent meeting of the Editorial Board ofN...  \n",
       "2501                                               None  \n",
       "2615                                               None  \n",
       "2713                                               None  \n",
       "2829  The deadlines for submission of articles inten...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "difference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these appear to be either editorials, database editions (which just describes the database for that year), or a subscription page. It seems reasonable that they ought to be dropped. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "nar_df.to_csv('complete-article-data.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}