make_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building the dataset of research papers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Entrez](http://biopython.org/DIST/docs/api/Bio.Entrez-module.html) module, a part of the [Biopython](http://biopython.org/) library, will be used to interface with [PubMed](http://www.ncbi.nlm.nih.gov/pubmed).<br>\n",
    "You can download Biopython from [here](http://biopython.org/wiki/Download).\n",
    "\n",
    "In this notebook we will be covering several of the steps taken in the [Biopython Tutorial](http://biopython.org/DIST/docs/tutorial/Tutorial.html), specifically in [Chapter 9  Accessing NCBI’s Entrez databases](http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc109)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Bio import Entrez\n",
    "\n",
    "# NCBI requires you to set your email address to make use of NCBI's E-utilities\n",
    "Entrez.email = \"Your.Name.Here@example.org\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datasets will be saved as serialized Python objects, compressed with bzip2.\n",
    "Saving/loading them will therefore require the [pickle](http://docs.python.org/3/library/pickle.html) and [bz2](http://docs.python.org/3/library/bz2.html) modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle, bz2, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EInfo: Obtaining information about the Entrez databases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accessing extended information about the PubMed database\n",
    "pubmed = Entrez.read( Entrez.einfo(db=\"pubmed\"), validate=False )[u'DbInfo']\n",
    "\n",
    "# list of possible search fields for use with ESearch:\n",
    "search_fields = { f['Name']:f['Description'] for f in pubmed[\"FieldList\"] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In search_fields, we find 'TIAB' ('Free text associated with Abstract/Title') as a possible search field to use in searches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ALL': 'All terms from all searchable fields',\n",
       " 'UID': 'Unique number assigned to publication',\n",
       " 'FILT': 'Limits the records',\n",
       " 'TITL': 'Words in title of publication',\n",
       " 'WORD': 'Free text associated with publication',\n",
       " 'MESH': 'Medical Subject Headings assigned to publication',\n",
       " 'MAJR': 'MeSH terms of major importance to publication',\n",
       " 'AUTH': 'Author(s) of publication',\n",
       " 'JOUR': 'Journal abbreviation of publication',\n",
       " 'AFFL': \"Author's institutional affiliation and address\",\n",
       " 'ECNO': 'EC number for enzyme or CAS registry number',\n",
       " 'SUBS': 'CAS chemical name or MEDLINE Substance Name',\n",
       " 'PDAT': 'Date of publication',\n",
       " 'EDAT': 'Date publication first accessible through Entrez',\n",
       " 'VOL': 'Volume number of publication',\n",
       " 'PAGE': 'Page number(s) of publication',\n",
       " 'PTYP': 'Type of publication (e.g., review)',\n",
       " 'LANG': 'Language of publication',\n",
       " 'ISS': 'Issue number of publication',\n",
       " 'SUBH': 'Additional specificity for MeSH term',\n",
       " 'SI': 'Cross-reference from publication to other databases',\n",
       " 'MHDA': 'Date publication was indexed with MeSH terms',\n",
       " 'TIAB': 'Free text associated with Abstract/Title',\n",
       " 'OTRM': 'Other terms associated with publication',\n",
       " 'INVR': 'Investigator',\n",
       " 'COLN': 'Corporate Author of publication',\n",
       " 'CNTY': 'Country of publication',\n",
       " 'PAPX': 'MeSH pharmacological action pre-explosions',\n",
       " 'GRNT': 'NIH Grant Numbers',\n",
       " 'MDAT': 'Date of last modification',\n",
       " 'CDAT': 'Date of completion',\n",
       " 'PID': 'Publisher ID',\n",
       " 'FAUT': 'First Author of publication',\n",
       " 'FULL': 'Full Author Name(s) of publication',\n",
       " 'FINV': 'Full name of investigator',\n",
       " 'TT': 'Words in transliterated title of publication',\n",
       " 'LAUT': 'Last Author of publication',\n",
       " 'PPDT': 'Date of print publication',\n",
       " 'EPDT': 'Date of Electronic publication',\n",
       " 'LID': 'ELocation ID',\n",
       " 'CRDT': 'Date publication first accessible through Entrez',\n",
       " 'BOOK': 'ID of the book that contains the document',\n",
       " 'ED': \"Section's Editor\",\n",
       " 'ISBN': 'ISBN',\n",
       " 'PUBN': \"Publisher's name\",\n",
       " 'AUCL': 'Author Cluster ID',\n",
       " 'EID': 'Extended PMID',\n",
       " 'DSO': 'Additional text from the summary',\n",
       " 'AUID': 'Author Identifier',\n",
       " 'PS': 'Personal Name as Subject',\n",
       " 'COIS': 'Conflict of Interest Statements'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ESearch: Searching the Entrez databases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To have a look at the kind of data we get when searching the database, we'll perform a search for papers authored by Haasdijk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DictElement({'Count': '36', 'RetMax': '20', 'RetStart': '0', 'IdList': ['29311830', '28513205', '28513201', '28323435', '28140628', '26933487', '24977986', '24901702', '24852945', '24708899', '24252306', '23580075', '23144668', '22174697', '22154920', '21870131', '21760539', '20662596', '20602234', '20386726'], 'TranslationSet': [], 'TranslationStack': [DictElement({'Term': 'Haasdijk E[Author]', 'Field': 'Author', 'Count': '36', 'Explode': 'N'}, attributes={}), 'GROUP'], 'QueryTranslation': 'Haasdijk E[Author]'}, attributes={})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_authors = ['Haasdijk E']\n",
    "example_search = Entrez.read( Entrez.esearch( db=\"pubmed\", term=' AND '.join([a+'[AUTH]' for a in example_authors]) ) )\n",
    "example_search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the result being produced is not in Python's native string format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Bio.Entrez.Parser.StringElement"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type( example_search['IdList'][0] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The part of the query's result we are most interested in is accessible through"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29311830, 28513205, 28513201, 28323435, 28140628, 26933487, 24977986, 24901702, 24852945, 24708899, 24252306, 23580075, 23144668, 22174697, 22154920, 21870131, 21760539, 20662596, 20602234, 20386726]\n"
     ]
    }
   ],
   "source": [
    "example_ids = [ int(id) for id in example_search['IdList'] ]\n",
    "print(example_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PubMed IDs dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now assemble a dataset comprised of research articles containing the given keyword, in either their titles or abstracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_term = 'allergy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "Ids_file = 'data/' + search_term + '_Ids.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching IDs of results [0,10000]\n",
      "Fetching IDs of results [10000,20000]\n",
      "Fetching IDs of results [20000,30000]\n",
      "Fetching IDs of results [30000,40000]\n",
      "Fetching IDs of results [40000,50000]\n",
      "Fetching IDs of results [50000,60000]\n",
      "Fetching IDs of results [60000,70000]\n",
      "Fetching IDs of results [70000,80000]\n",
      "73740 documents contain the search term \"allergy\".\n"
     ]
    }
   ],
   "source": [
    "if os.path.exists( Ids_file ):\n",
    "    Ids = pickle.load( bz2.BZ2File( Ids_file, 'rb' ) )\n",
    "else:\n",
    "    # determine the number of hits for the search term\n",
    "    search = Entrez.read( Entrez.esearch( db=\"pubmed\", term=search_term+'[TIAB]', retmax=0 ) )\n",
    "    total = int( search['Count'] )\n",
    "    \n",
    "    # `Ids` will be incrementally assembled, by performing multiple queries,\n",
    "    # each returning at most `retrieve_per_query` entries.\n",
    "    Ids_str = []\n",
    "    retrieve_per_query = 10000\n",
    "    \n",
    "    for start in range( 0, total, retrieve_per_query ):\n",
    "        print('Fetching IDs of results [%d,%d]' % ( start, start+retrieve_per_query ) )\n",
    "        s = Entrez.read( Entrez.esearch( db=\"pubmed\", term=search_term+'[TIAB]', retstart=start, retmax=retrieve_per_query ) )\n",
    "        Ids_str.extend( s[ u'IdList' ] )\n",
    "    \n",
    "    # convert Ids to integers (and ensure that the conversion is reversible)\n",
    "    Ids = [ int(id) for id in Ids_str ]\n",
    "    \n",
    "    for (id_str, id_int) in zip(Ids_str, Ids):\n",
    "        if str(id_int) != id_str:\n",
    "            raise Exception('Conversion of PubMed ID %s from string to integer it not reversible.' % id_str )\n",
    "    \n",
    "    # Remove IDs that would cause problems below:\n",
    "    Ids.remove(30823833)\n",
    "    Ids.remove(30223658)\n",
    "    Ids.remove(29623708)\n",
    "    Ids.remove(29394076)\n",
    "    Ids.remove(29303278)\n",
    "    Ids.remove(28948806)\n",
    "    Ids.remove(28832171)\n",
    "    Ids.remove(28691830)\n",
    "    Ids.remove(28264572)\n",
    "    Ids.remove(28252306)\n",
    "    Ids.remove(28078899)\n",
    "    Ids.remove(27585420)\n",
    "    \n",
    "    # Save list of Ids\n",
    "    pickle.dump( Ids, bz2.BZ2File( Ids_file, 'wb' ) )\n",
    "    \n",
    "total = len( Ids )\n",
    "print('%d documents contain the search term \"%s\".' % ( total, search_term ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a look at what we just retrieved, here are the last 5 elements of the `Ids` list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[31574307, 31573992, 31573731, 31573676, 31572454]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ids[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ESummary: Retrieving summaries from primary IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To have a look at the kind of metadata we get from a call to `Entrez.esummary()`, we now fetch the summary of one of Haasdijk's papers (using one of the PubMed IDs we obtained in the previous section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item\n",
      "\t []\n",
      "Id\n",
      "\t 31574307\n",
      "PubDate\n",
      "\t 2019 Sep 28\n",
      "EPubDate\n",
      "\t 2019 Sep 28\n",
      "Source\n",
      "\t Ann Allergy Asthma Immunol\n",
      "AuthorList\n",
      "\t ['Harper V', 'Wagner A', 'Sowa P', 'Rutkowski K']\n",
      "LastAuthor\n",
      "\t Rutkowski K\n",
      "Title\n",
      "\t Allergy in frequent travelers: the alpha-gal story and the first UK case series.\n",
      "Volume\n",
      "\t \n",
      "Issue\n",
      "\t \n",
      "Pages\n",
      "\t \n",
      "LangList\n",
      "\t ['English']\n",
      "NlmUniqueID\n",
      "\t 9503580\n",
      "ISSN\n",
      "\t 1081-1206\n",
      "ESSN\n",
      "\t 1534-4436\n",
      "PubTypeList\n",
      "\t ['Journal Article']\n",
      "RecordStatus\n",
      "\t PubMed - as supplied by publisher\n",
      "PubStatus\n",
      "\t aheadofprint\n",
      "ArticleIds\n",
      "\t DictElement({'pubmed': ['31574307'], 'medline': [], 'pii': 'S1081-1206(19)31239-6', 'doi': '10.1016/j.anai.2019.09.018', 'rid': '31574307', 'eid': '31574307'}, attributes={})\n",
      "DOI\n",
      "\t 10.1016/j.anai.2019.09.018\n",
      "History\n",
      "\t DictElement({'pubmed': ['2019/10/02 06:00'], 'medline': ['2019/10/02 06:00'], 'received': '2019/07/17 00:00', 'revised': '2019/09/18 00:00', 'accepted': '2019/09/22 00:00', 'entrez': '2019/10/02 06:00'}, attributes={})\n",
      "References\n",
      "\t []\n",
      "HasAbstract\n",
      "\t 0\n",
      "PmcRefCount\n",
      "\t 0\n",
      "FullJournalName\n",
      "\t Annals of allergy, asthma & immunology : official publication of the American College of Allergy, Asthma, & Immunology\n",
      "ELocationID\n",
      "\t pii: S1081-1206(19)31239-6. doi: 10.1016/j.anai.2019.09.018\n",
      "SO\n",
      "\t 2019 Sep 28;\n"
     ]
    }
   ],
   "source": [
    "example_paper = Entrez.read( Entrez.esummary(db=\"pubmed\", id='31574307') )[0]\n",
    "\n",
    "def print_dict( p ):\n",
    "    for k,v in p.items():\n",
    "        print(k)\n",
    "        print('\\t', v)\n",
    "\n",
    "print_dict(example_paper)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, we'll keep just some basic information for each paper: title, list of authors, publication year, and [DOI](https://en.wikipedia.org/wiki/Digital_object_identifier).\n",
    "\n",
    "In case you are not familiar with the DOI system, know that the paper above can be accessed through the link  `https://www.doi.org/` followed by the paper's DOI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Allergy in frequent travelers: the alpha-gal story and the first UK case series.',\n",
       " ['Harper V', 'Wagner A', 'Sowa P', 'Rutkowski K'],\n",
       " 2019,\n",
       " '10.1016/j.anai.2019.09.018')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "( example_paper['Title'], example_paper['AuthorList'], int(example_paper['PubDate'][:4]), example_paper['DOI'] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summaries dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to assemble a dataset containing the summaries of all the paper `Ids` we previously fetched.\n",
    "\n",
    "To reduce the memory footprint, and to ensure the saved datasets won't depend on Biopython being installed to be properly loaded, values returned by `Entrez.read()` will be converted to their corresponding native Python types. We start by defining a function for helping with the conversion of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "Summaries_file = 'data/' + search_term + '_Summaries.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching Summaries of results: \n",
      "\n",
      "0...\n",
      "Error with ID 30823833: invalid literal for int() with base 10: 'Wint'\n",
      "Manually remove this ID above and re-run code.\n",
      "...\n",
      "Error with ID 30223658: invalid literal for int() with base 10: 'Summ'\n",
      "Manually remove this ID above and re-run code.\n",
      "...\n",
      "Error with ID 29623708: invalid literal for int() with base 10: 'Spri'\n",
      "Manually remove this ID above and re-run code.\n",
      ".\n",
      "Error with ID 29394076: invalid literal for int() with base 10: 'Fall'\n",
      "Manually remove this ID above and re-run code.\n",
      ".\n",
      "Error with ID 29303278: invalid literal for int() with base 10: 'Wint'\n",
      "Manually remove this ID above and re-run code.\n",
      ".\n",
      "Error with ID 28948806: invalid literal for int() with base 10: 'Fall'\n",
      "Manually remove this ID above and re-run code.\n",
      ".\n",
      "Error with ID 28832171: invalid literal for int() with base 10: 'Summ'\n",
      "Manually remove this ID above and re-run code.\n",
      ".\n",
      "Error with ID 28691830: invalid literal for int() with base 10: 'Summ'\n",
      "Manually remove this ID above and re-run code.\n",
      "..\n",
      "Error with ID 28264572: invalid literal for int() with base 10: 'Wint'\n",
      "Manually remove this ID above and re-run code.\n",
      "\n",
      "Error with ID 28252306: invalid literal for int() with base 10: 'Wint'\n",
      "Manually remove this ID above and re-run code.\n",
      ".\n",
      "Error with ID 28078899: invalid literal for int() with base 10: 'Fall'\n",
      "Manually remove this ID above and re-run code.\n",
      "..\n",
      "Error with ID 27585420: invalid literal for int() with base 10: 'Fall'\n",
      "Manually remove this ID above and re-run code.\n",
      "\n",
      "10000...................\n",
      "20000...................\n",
      "30000...................\n",
      "40000...................\n",
      "50000...................\n",
      "60000...................\n",
      "70000......."
     ]
    }
   ],
   "source": [
    "if os.path.exists( Summaries_file ):\n",
    "    Summaries = pickle.load( bz2.BZ2File( Summaries_file, 'rb' ) )\n",
    "else:\n",
    "    # `Summaries` will be incrementally assembled, by performing multiple queries,\n",
    "    # each returning at most `retrieve_per_query` entries.\n",
    "    Summaries = []\n",
    "    retrieve_per_query = 500\n",
    "    \n",
    "    print('Fetching Summaries of results: ')\n",
    "    for start in range( 0, len(Ids), retrieve_per_query ):\n",
    "        if (start % 10000 == 0):\n",
    "            print('')\n",
    "            print(start, end='')\n",
    "        else:\n",
    "            print('.', end='')\n",
    "        \n",
    "        # build comma separated string with the ids at indexes [start, start+retrieve_per_query)\n",
    "        query_ids = ','.join( [ str(id) for id in Ids[ start : start+retrieve_per_query ] ] )\n",
    "        \n",
    "        s = Entrez.read( Entrez.esummary( db=\"pubmed\", id=query_ids ) )\n",
    "        \n",
    "        # out of the retrieved data, we will keep only a tuple (title, authors, year, DOI), associated with the paper's id.\n",
    "        # (all values converted to native Python formats)\n",
    "        for p in s:\n",
    "            try:\n",
    "                f = [\n",
    "                    ( int( p['Id'] ), (\n",
    "                        str( p['Title'] ),\n",
    "                        [ str(a) for a in p['AuthorList'] ],\n",
    "                        int( p['PubDate'][:4] ),                # keeps just the publication year\n",
    "                        str( p.get('DOI', '') )            # papers for which no DOI is available get an empty string in their place\n",
    "                        ) )\n",
    "                    ]\n",
    "                Summaries.extend( f )\n",
    "            except ValueError as e:\n",
    "                print(\"\\nError with ID \" + p['Id'] + \": \" + str(e))\n",
    "                print(\"Manually remove this ID above and re-run code.\")\n",
    "    \n",
    "    # Save Summaries, as a dictionary indexed by Ids\n",
    "    Summaries = dict( Summaries )\n",
    "    \n",
    "    pickle.dump( Summaries, bz2.BZ2File( Summaries_file, 'wb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us take a look at the first 3 retrieved summaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{31574307: ('Allergy in frequent travelers: the alpha-gal story and the first UK case series.',\n",
       "  ['Harper V', 'Wagner A', 'Sowa P', 'Rutkowski K'],\n",
       "  2019,\n",
       "  '10.1016/j.anai.2019.09.018'),\n",
       " 31573992: ('Precision medicine in atopic diseases.',\n",
       "  ['Cianferoni A', 'Annesi-Maesano I'],\n",
       "  2019,\n",
       "  '10.1097/ACI.0000000000000591'),\n",
       " 31573731: ('Cinnamon extract inhibits allergen-specific immune responses in human and murine allergy models.',\n",
       "  ['Ose R',\n",
       "   'Tu J',\n",
       "   'Schink A',\n",
       "   'Maxeiner J',\n",
       "   'Schuster P',\n",
       "   'Lucas K',\n",
       "   'Saloga J',\n",
       "   'Bellinghausen I'],\n",
       "  2019,\n",
       "  '10.1111/cea.13507')}"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{ id : Summaries[id] for id in Ids[:3] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EFetch: Downloading full records from Entrez"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Entrez.efetch()` is the function that will allow us to obtain paper abstracts. Let us start by taking a look at the kind of data it returns when we query PubMed's database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "q = Entrez.read( Entrez.efetch(db=\"pubmed\", id='18317433', retmode=\"xml\") )['PubmedArticle']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`q` is a list, with each member corresponding to a queried id. Because here we only queried for one id, its results are then in `q[0]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(list, 1)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(q), len(q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1\n",
    "At `q[0]` we find a dictionary containing two keys, the contents of which we print below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Bio.Entrez.Parser.DictionaryElement,\n",
       " dict_keys(['MedlineCitation', 'PubmedData']))"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(q[0]), q[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ReferenceList\n",
      "\t [DictElement({'Reference': [DictElement({'Citation': 'Diabetes. 2000 Apr;49(4):539-47', 'ArticleIdList': [StringElement('10871190', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Nat Rev Immunol. 2002 Oct;2(10):748-59', 'ArticleIdList': [StringElement('12360213', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Anticancer Drugs. 2007 Mar;18(3):237-44', 'ArticleIdList': [StringElement('17264754', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 1999 Nov 5;274(45):32048-54', 'ArticleIdList': [StringElement('10542237', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 2000 Oct 27;275(43):33201-4', 'ArticleIdList': [StringElement('10944516', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 2005 Dec 23;280(51):42464-75', 'ArticleIdList': [StringElement('16227625', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Proc Natl Acad Sci U S A. 2002 Mar 5;99(5):2613-8', 'ArticleIdList': [StringElement('11867749', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Nature. 1990 Oct 18;347(6294):645-50', 'ArticleIdList': [StringElement('2129546', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Cell. 1999 Apr 16;97(2):161-3', 'ArticleIdList': [StringElement('10219237', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Cell. 1998 Apr 17;93(2):229-40', 'ArticleIdList': [StringElement('9568715', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Proc Natl Acad Sci U S A. 1993 Feb 15;90(4):1440-4', 'ArticleIdList': [StringElement('8381967', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'N Engl J Med. 2004 Sep 9;351(11):1106-18', 'ArticleIdList': [StringElement('15356308', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Oncogene. 2004 Jan 8;23 (1):100-8', 'ArticleIdList': [StringElement('14712215', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Am J Respir Cell Mol Biol. 2004 Jan;30(1):1-5', 'ArticleIdList': [StringElement('14512375', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Nature. 1999 Jul 22;400(6742):378-82', 'ArticleIdList': [StringElement('10432118', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Circ Res. 2003 May 30;92(10):1162-9', 'ArticleIdList': [StringElement('12714563', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 2001 Oct 12;276(41):37731-4', 'ArticleIdList': [StringElement('11459852', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Expert Opin Ther Targets. 2005 Aug;9(4):861-73', 'ArticleIdList': [StringElement('16083348', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 2002 Sep 13;277(37):34176-81', 'ArticleIdList': [StringElement('12107164', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Cell Biochem Biophys. 2000;32 Spring:187-204', 'ArticleIdList': [StringElement('11330046', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Clin Invest. 2000 Aug;106(4):467-72', 'ArticleIdList': [StringElement('10953021', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 1998 Oct 2;273(40):25573-80', 'ArticleIdList': [StringElement('9748221', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Circ Res. 2003 Mar 21;92(5):518-24', 'ArticleIdList': [StringElement('12600885', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Nature. 2002 Feb 14;415(6873):813-7', 'ArticleIdList': [StringElement('11845213', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Proc Natl Acad Sci U S A. 2005 Feb 15;102(7):2340-5', 'ArticleIdList': [StringElement('15701701', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Am J Respir Cell Mol Biol. 2003 Dec;29(6):677-82', 'ArticleIdList': [StringElement('12805087', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 1995 Jun 2;270(22):12953-6', 'ArticleIdList': [StringElement('7768881', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'J Biol Chem. 1998 Dec 4;273(49):32679-84', 'ArticleIdList': [StringElement('9830009', attributes={'IdType': 'pubmed'})]}, attributes={}), DictElement({'Citation': 'Cell. 1998 Apr 17;93(2):241-52', 'ArticleIdList': [StringElement('9568716', attributes={'IdType': 'pubmed'})]}, attributes={})], 'ReferenceList': []}, attributes={})]\n",
      "History\n",
      "\t [DictElement({'Year': '2008', 'Month': '3', 'Day': '5', 'Hour': '9', 'Minute': '0'}, attributes={'PubStatus': 'pubmed'}), DictElement({'Year': '2008', 'Month': '4', 'Day': '25', 'Hour': '9', 'Minute': '0'}, attributes={'PubStatus': 'medline'}), DictElement({'Year': '2008', 'Month': '3', 'Day': '5', 'Hour': '9', 'Minute': '0'}, attributes={'PubStatus': 'entrez'})]\n",
      "PublicationStatus\n",
      "\t ppublish\n",
      "ArticleIdList\n",
      "\t [StringElement('18317433', attributes={'IdType': 'pubmed'}), StringElement('10.2310/JIM.0b013e318165e89d', attributes={'IdType': 'doi'}), StringElement('00042871-200802000-00004', attributes={'IdType': 'pii'}), StringElement('PMC4487779', attributes={'IdType': 'pmc'}), StringElement('NIHMS211777', attributes={'IdType': 'mid'})]\n"
     ]
    }
   ],
   "source": [
    "print_dict( q[0][ 'PubmedData' ] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key `'MedlineCitation'` maps into another dictionary. In that dictionary, most of the information is contained under the key `'Article'`. To minimize the clutter, below we show the contents of `'MedlineCitation'` excluding its `'Article'` member, and below that we then show the contents of `'Article'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GeneralNote\n",
      "\t []\n",
      "KeywordList\n",
      "\t []\n",
      "SpaceFlightMission\n",
      "\t []\n",
      "CitationSubset\n",
      "\t ['IM']\n",
      "OtherID\n",
      "\t []\n",
      "OtherAbstract\n",
      "\t []\n",
      "PMID\n",
      "\t 18317433\n",
      "DateCompleted\n",
      "\t DictElement({'Year': '2008', 'Month': '04', 'Day': '24'}, attributes={})\n",
      "DateRevised\n",
      "\t DictElement({'Year': '2018', 'Month': '11', 'Day': '13'}, attributes={})\n",
      "MedlineJournalInfo\n",
      "\t DictElement({'Country': 'England', 'MedlineTA': 'J Investig Med', 'NlmUniqueID': '9501229', 'ISSNLinking': '1081-5589'}, attributes={})\n",
      "ChemicalList\n",
      "\t [DictElement({'RegistryNumber': '0', 'NameOfSubstance': StringElement('PPAR gamma', attributes={'UI': 'D047495'})}, attributes={})]\n",
      "MeshHeadingList\n",
      "\t [DictElement({'QualifierName': [], 'DescriptorName': StringElement('Humans', attributes={'UI': 'D006801', 'MajorTopicYN': 'N'})}, attributes={}), DictElement({'QualifierName': [StringElement('physiopathology', attributes={'UI': 'Q000503', 'MajorTopicYN': 'Y'})], 'DescriptorName': StringElement('Lung Diseases', attributes={'UI': 'D008171', 'MajorTopicYN': 'N'})}, attributes={}), DictElement({'QualifierName': [StringElement('physiology', attributes={'UI': 'Q000502', 'MajorTopicYN': 'Y'})], 'DescriptorName': StringElement('PPAR gamma', attributes={'UI': 'D047495', 'MajorTopicYN': 'N'})}, attributes={})]\n"
     ]
    }
   ],
   "source": [
    "print_dict( { k:v for k,v in q[0][ 'MedlineCitation' ].items() if k!='Article' } )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ELocationID\n",
      "\t [StringElement('10.2310/JIM.0b013e318165e89d', attributes={'EIdType': 'doi', 'ValidYN': 'Y'})]\n",
      "ArticleDate\n",
      "\t []\n",
      "Language\n",
      "\t ['eng']\n",
      "Journal\n",
      "\t DictElement({'ISSN': StringElement('1081-5589', attributes={'IssnType': 'Print'}), 'JournalIssue': DictElement({'Volume': '56', 'Issue': '2', 'PubDate': DictElement({'Year': '2008', 'Month': 'Feb'}, attributes={})}, attributes={'CitedMedium': 'Print'}), 'Title': 'Journal of investigative medicine : the official publication of the American Federation for Clinical Research', 'ISOAbbreviation': 'J. Investig. Med.'}, attributes={})\n",
      "ArticleTitle\n",
      "\t PPARgamma: a novel molecular target in lung disease.\n",
      "Pagination\n",
      "\t DictElement({'MedlinePgn': '515-7'}, attributes={})\n",
      "Abstract\n",
      "\t DictElement({'AbstractText': ['Interest in peroxisome proliferator-activated receptors (PPARs) has steadily increased over the past 15 years. The recognition that subclasses of this receptor played critical roles in regulation of metabolism led to the development of synthetic ligands and their widespread application in the treatment of type 2 diabetes. At the same time, emerging evidence demonstrated that the influence of PPARs extends well beyond metabolism and diabetes. A salient example of this can be seen in studies that explore the role of PPARs in lung cell biology. In fact, current literature suggests that PPAR receptors may well represent exciting new targets for treatment in a variety of lung disorders. In an attempt to keep the scientific and medical communities abreast of these developments, a symposium sponsored by the American Federation for Medical Research entitled \"PPARgamma: A Novel Molecular Target in Lung Disease\" was convened on April 29, 2007, at the Experimental Biology Meeting in Washington, DC. During that symposium, 4 speakers reviewed the latest developments in basic and translational research as they relate to specific lung diseases. Jesse Roman, MD, professor and director of the Emory University Division of Pulmonary, Allergy, and Critical Care Medicine, reviewed the role of PPARgamma in the pathogenesis of lung cancer and its implications for therapy. Raju Reddy, MD, assistant professor of Medicine at the University of Michigan, presented data regarding the immunomodulatory role of PPARgamma in alveolar macrophages. Patricia J. Sime, MD, associate professor of Medicine, Environmental Medicine, and Oncology at the University of Rochester School of Medicine, discussed the antifibrogenic potential of PPARgamma ligands in pulmonary fibrosis. Finally, C. Michael Hart, MD, professor of Medicine at Emory University and chief of the Atlanta Veterans Affairs Medical Center Pulmonary Section, reviewed the role of PPARgamma in pulmonary vascular disease. This brief introduction to the symposium will provide background information about PPARs to facilitate the general reader\\'s appreciation of the more in-depth and disease-specific discussions that follow.']}, attributes={})\n",
      "AuthorList\n",
      "\t ListElement([DictElement({'AffiliationInfo': [DictElement({'Identifier': [], 'Affiliation': 'Division of Pulmonary, Allergy and Critical Care Medicine, Emory University Medical Centers, Atlanta, GA, USA. Michael.hart3@va.gov'}, attributes={})], 'Identifier': [], 'LastName': 'Hart', 'ForeName': 'C Michael', 'Initials': 'CM'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [], 'Identifier': [], 'LastName': 'Roman', 'ForeName': 'Jesse', 'Initials': 'J'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [], 'Identifier': [], 'LastName': 'Reddy', 'ForeName': 'Raju', 'Initials': 'R'}, attributes={'ValidYN': 'Y'}), DictElement({'AffiliationInfo': [], 'Identifier': [], 'LastName': 'Sime', 'ForeName': 'Patricia J', 'Initials': 'PJ'}, attributes={'ValidYN': 'Y'})], attributes={'CompleteYN': 'Y'})\n",
      "GrantList\n",
      "\t ListElement([DictElement({'GrantID': 'R13 RR023236', 'Acronym': 'RR', 'Agency': 'NCRR NIH HHS', 'Country': 'United States'}, attributes={}), DictElement({'GrantID': 'R13 RR023236-01', 'Acronym': 'RR', 'Agency': 'NCRR NIH HHS', 'Country': 'United States'}, attributes={})], attributes={'CompleteYN': 'Y'})\n",
      "PublicationTypeList\n",
      "\t [StringElement('Introductory Journal Article', attributes={'UI': 'D054711'})]\n"
     ]
    }
   ],
   "source": [
    "print_dict( q[0][ 'MedlineCitation' ][ 'Article' ] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A paper's abstract can therefore be accessed with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{18317433: 'Interest in peroxisome proliferator-activated receptors (PPARs) has steadily increased over the past 15 years. The recognition that subclasses of this receptor played critical roles in regulation of metabolism led to the development of synthetic ligands and their widespread application in the treatment of type 2 diabetes. At the same time, emerging evidence demonstrated that the influence of PPARs extends well beyond metabolism and diabetes. A salient example of this can be seen in studies that explore the role of PPARs in lung cell biology. In fact, current literature suggests that PPAR receptors may well represent exciting new targets for treatment in a variety of lung disorders. In an attempt to keep the scientific and medical communities abreast of these developments, a symposium sponsored by the American Federation for Medical Research entitled \"PPARgamma: A Novel Molecular Target in Lung Disease\" was convened on April 29, 2007, at the Experimental Biology Meeting in Washington, DC. During that symposium, 4 speakers reviewed the latest developments in basic and translational research as they relate to specific lung diseases. Jesse Roman, MD, professor and director of the Emory University Division of Pulmonary, Allergy, and Critical Care Medicine, reviewed the role of PPARgamma in the pathogenesis of lung cancer and its implications for therapy. Raju Reddy, MD, assistant professor of Medicine at the University of Michigan, presented data regarding the immunomodulatory role of PPARgamma in alveolar macrophages. Patricia J. Sime, MD, associate professor of Medicine, Environmental Medicine, and Oncology at the University of Rochester School of Medicine, discussed the antifibrogenic potential of PPARgamma ligands in pulmonary fibrosis. Finally, C. Michael Hart, MD, professor of Medicine at Emory University and chief of the Atlanta Veterans Affairs Medical Center Pulmonary Section, reviewed the role of PPARgamma in pulmonary vascular disease. This brief introduction to the symposium will provide background information about PPARs to facilitate the general reader\\'s appreciation of the more in-depth and disease-specific discussions that follow.'}"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{ int(q[0]['MedlineCitation']['PMID']) : str(q[0]['MedlineCitation']['Article']['Abstract']['AbstractText'][0]) }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the ids in our dataset refer to books from the [NCBI Bookshelf](http://www.ncbi.nlm.nih.gov/books/), a collection of freely available, downloadable, on-line versions of selected biomedical books. For such ids, `Entrez.efetch()` returns a slightly different structure, where the keys `[u'BookDocument', u'PubmedBookData']` take the place of the `[u'MedlineCitation', u'PubmedData']` keys we saw above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Abstracts dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now assemble a dataset mapping paper ids to their abstracts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "Abstracts_file = 'data/' + search_term + '_Abstracts.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching Abstracts of results: \n",
      "\n",
      "0...................\n",
      "10000...................\n",
      "20000...................\n",
      "30000...................\n",
      "40000...................\n",
      "50000...................\n",
      "60000...................\n",
      "70000......."
     ]
    }
   ],
   "source": [
    "import http.client\n",
    "from collections import deque\n",
    "from xml.dom import minidom\n",
    "import re\n",
    "\n",
    "def ch(node, childtype):\n",
    "    return node.getElementsByTagName(childtype)[0]\n",
    "\n",
    "if os.path.exists( Abstracts_file ):\n",
    "    Abstracts = pickle.load( bz2.BZ2File( Abstracts_file, 'rb' ) )\n",
    "else:\n",
    "    # `Abstracts` will be incrementally assembled, by performing multiple queries,\n",
    "    # each returning at most `retrieve_per_query` entries.\n",
    "    Abstracts = deque()\n",
    "    retrieve_per_query = 500\n",
    "    \n",
    "    print('Fetching Abstracts of results: ')\n",
    "    for start in range( 0, len(Ids), retrieve_per_query ):\n",
    "        if (start % 10000 == 0):\n",
    "            print('')\n",
    "            print(start, end='')\n",
    "        else:\n",
    "            print('.', end='')\n",
    "        \n",
    "        # build comma separated string with the ids at indexes [start, start+retrieve_per_query)\n",
    "        query_ids = ','.join( [ str(id) for id in Ids[ start : start+retrieve_per_query ] ] )\n",
    "        \n",
    "        # issue requests to the server, until we get the full amount of data we expect\n",
    "        while True:\n",
    "            try:\n",
    "                #s = Entrez.read( Entrez.efetch(db=\"pubmed\", id=query_ids, retmode=\"xml\" ) )['PubmedArticle']\n",
    "                s = minidom.parse( Entrez.efetch(db=\"pubmed\", id=query_ids, retmode=\"xml\" ) ).getElementsByTagName(\"PubmedArticle\")\n",
    "            except http.client.IncompleteRead:\n",
    "                print('r', end='')\n",
    "                continue\n",
    "            break\n",
    "        \n",
    "        i = 0\n",
    "        for p in s:\n",
    "            abstr = ''\n",
    "            if (p.getElementsByTagName('MedlineCitation')):\n",
    "                citNode = ch(p,'MedlineCitation')\n",
    "                pmid = ch(citNode,'PMID').firstChild.data\n",
    "                articleNode = ch(citNode,'Article')\n",
    "                if (articleNode.getElementsByTagName('Abstract')):\n",
    "                    try:\n",
    "                        abstr = ch(ch(articleNode,'Abstract'),'AbstractText').firstChild.data\n",
    "                    except AttributeError:\n",
    "                        abstr = ch(ch(articleNode,'Abstract'),'AbstractText').toprettyxml(\"  \")\n",
    "                        abstr = re.sub(r\"\\s+\", \" \", re.sub(\"<[^>]*>\", \"\", abstr))\n",
    "            elif (p.getElementsByTagName('BookDocument')):\n",
    "                bookNode = ch(p,'BookDocument')\n",
    "                pmid = ch(bookNode,'PMID').firstChild.data\n",
    "                if (bookNode.getElementsByTagName('Abstract')):\n",
    "                    try:\n",
    "                        abstr = ch(ch(bookNode,'Abstract'),'AbstractText').firstChild.data\n",
    "                    except AttributeError:\n",
    "                        abstr = ch(ch(bookNode,'Abstract'),'AbstractText').toprettyxml(\"  \")\n",
    "                        abstr = re.sub(r\"\\s+\", \" \", re.sub(\"<[^>]*>\", \"\", abstr))\n",
    "            else:\n",
    "                raise Exception('Unrecognized record type, for id %d (keys: %s)' % (Ids[start+i], str(p.keys())) )\n",
    "            \n",
    "            Abstracts.append( (int(pmid), str(abstr)) )\n",
    "            i += 1\n",
    "    \n",
    "    # Save Abstracts, as a dictionary indexed by Ids\n",
    "    Abstracts = dict( Abstracts )\n",
    "    \n",
    "    pickle.dump( Abstracts, bz2.BZ2File( Abstracts_file, 'wb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a look at one paper's abstract:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Interest in peroxisome proliferator-activated receptors (PPARs) has steadily increased over the past 15 years. The recognition that subclasses of this receptor played critical roles in regulation of metabolism led to the development of synthetic ligands and their widespread application in the treatment of type 2 diabetes. At the same time, emerging evidence demonstrated that the influence of PPARs extends well beyond metabolism and diabetes. A salient example of this can be seen in studies that explore the role of PPARs in lung cell biology. In fact, current literature suggests that PPAR receptors may well represent exciting new targets for treatment in a variety of lung disorders. In an attempt to keep the scientific and medical communities abreast of these developments, a symposium sponsored by the American Federation for Medical Research entitled \"PPARgamma: A Novel Molecular Target in Lung Disease\" was convened on April 29, 2007, at the Experimental Biology Meeting in Washington, DC. During that symposium, 4 speakers reviewed the latest developments in basic and translational research as they relate to specific lung diseases. Jesse Roman, MD, professor and director of the Emory University Division of Pulmonary, Allergy, and Critical Care Medicine, reviewed the role of PPARgamma in the pathogenesis of lung cancer and its implications for therapy. Raju Reddy, MD, assistant professor of Medicine at the University of Michigan, presented data regarding the immunomodulatory role of PPARgamma in alveolar macrophages. Patricia J. Sime, MD, associate professor of Medicine, Environmental Medicine, and Oncology at the University of Rochester School of Medicine, discussed the antifibrogenic potential of PPARgamma ligands in pulmonary fibrosis. Finally, C. Michael Hart, MD, professor of Medicine at Emory University and chief of the Atlanta Veterans Affairs Medical Center Pulmonary Section, reviewed the role of PPARgamma in pulmonary vascular disease. This brief introduction to the symposium will provide background information about PPARs to facilitate the general reader\\'s appreciation of the more in-depth and disease-specific discussions that follow.'"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Abstracts[18317433]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ELink: Searching for related items in NCBI Entrez"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand how to obtain paper citations with Entrez, we will first assemble a small set of PubMed IDs, and then query for their citations.\n",
    "To that end, we search here for papers published in the Nature journal with our given keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['24037376', '22538601', '22318520', '19587753', '17898760', '17581592', '17450126', '17377532', '16688147', '15496927', '15118685', '12490951', '12226655', '11449275', '11343123', '11130078', '10917520', '10688206', '10586891', '10586890']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CA_search_term = search_term+'[TIAB] AND Nature[JOUR]'\n",
    "CA_ids = Entrez.read( Entrez.esearch( db=\"pubmed\", term=CA_search_term ) )['IdList']\n",
    "CA_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'24037376': ('Type 2 innate lymphoid cells control eosinophil homeostasis.',\n",
       "  ['Nussbaum JC', 'Van Dyken SJ', 'von Moltke J', 'Cheng LE', 'Mohapatra A', 'Molofsky AB', 'Thornton EE', 'Krummel MF', 'Chawla A', 'Liang HE', 'Locksley RM'],\n",
       "  '2013',\n",
       "  'Nature',\n",
       "  '10.1038/nature12526'),\n",
       " '22538601': ('Forum: Immunology: Allergy challenged.',\n",
       "  ['Artis D', 'Maizels RM', 'Finkelman FD'],\n",
       "  '2012',\n",
       "  'Nature',\n",
       "  '10.1038/484458a'),\n",
       " '22318520': ('Extrathymically generated regulatory T cells control mucosal TH2 inflammation.',\n",
       "  ['Josefowicz SZ', 'Niec RE', 'Kim HY', 'Treuting P', 'Chinen T', 'Zheng Y', 'Umetsu DT', 'Rudensky AY'],\n",
       "  '2012',\n",
       "  'Nature',\n",
       "  '10.1038/nature10772'),\n",
       " '19587753': ('Vision: New light on allergy receptor.',\n",
       "  ['Grant M'],\n",
       "  '2009',\n",
       "  'Nature',\n",
       "  '10.1038/460182a'),\n",
       " '17898760': ('Taking dendritic cells into medicine.',\n",
       "  ['Steinman RM', 'Banchereau J'],\n",
       "  '2007',\n",
       "  'Nature',\n",
       "  '10.1038/nature06175'),\n",
       " '17581592': ('CD1d-lipid-antigen recognition by the semi-invariant NKT T-cell receptor.',\n",
       "  ['Borg NA', 'Wun KS', 'Kjer-Nielsen L', 'Wilce MC', 'Pellicci DG', 'Koh R', 'Besra GS', 'Bharadwaj M', 'Godfrey DI', 'McCluskey J', 'Rossjohn J'],\n",
       "  '2007',\n",
       "  'Nature',\n",
       "  '10.1038/nature05907'),\n",
       " '17450126': ('Chitin induces accumulation in tissue of innate immune cells associated with allergy.',\n",
       "  ['Reese TA', 'Liang HE', 'Tager AM', 'Luster AD', 'Van Rooijen N', 'Voehringer D', 'Locksley RM'],\n",
       "  '2007',\n",
       "  'Nature',\n",
       "  '10.1038/nature05746'),\n",
       " '17377532': ('Foxp3 controls regulatory T-cell function by interacting with AML1/Runx1.',\n",
       "  ['Ono M', 'Yaguchi H', 'Ohkura N', 'Kitabayashi I', 'Nagamura Y', 'Nomura T', 'Miyachi Y', 'Tsukada T', 'Sakaguchi S'],\n",
       "  '2007',\n",
       "  'Nature',\n",
       "  '10.1038/nature05673'),\n",
       " '16688147': ('Allergy test might have avoided drug-trial disaster.',\n",
       "  ['Weis JH'],\n",
       "  '2006',\n",
       "  'Nature',\n",
       "  '10.1038/441150c'),\n",
       " '15496927': ('Essential role for the p110delta phosphoinositide 3-kinase in the allergic response.',\n",
       "  ['Ali K', 'Bilancio A', 'Thomas M', 'Pearce W', 'Gilfillan AM', 'Tkaczyk C', 'Kuehn N', 'Gray A', 'Giddings J', 'Peskett E', 'Fox R', 'Bruce I', 'Walker C', 'Sawyer C', 'Okkenhaug K', 'Finan P', 'Vanhaesebroeck B'],\n",
       "  '2004',\n",
       "  'Nature',\n",
       "  '10.1038/nature02991'),\n",
       " '15118685': ('Trainee let down as allergy institute withdraws support.',\n",
       "  ['Wadman M'],\n",
       "  '2004',\n",
       "  'Nature',\n",
       "  '10.1038/428879b'),\n",
       " '12490951': ('Helper T cells regulate type-2 innate immunity in vivo.',\n",
       "  ['Shinkai K', 'Mohrs M', 'Locksley RM'],\n",
       "  '2002',\n",
       "  'Nature',\n",
       "  '10.1038/nature01202'),\n",
       " '12226655': ('Metallurgy: high nickel release from 1- and 2-euro coins.',\n",
       "  ['Nestle FO', 'Speidel H', 'Speidel MO'],\n",
       "  '2002',\n",
       "  'Nature',\n",
       "  '10.1038/419132a'),\n",
       " '11449275': ('Essential role for Gab2 in the allergic response.',\n",
       "  ['Gu H', 'Saito K', 'Klaman LD', 'Shen J', 'Fleming T', 'Wang Y', 'Pratt JC', 'Lin G', 'Lim B', 'Kinet JP', 'Neel BG'],\n",
       "  '2001',\n",
       "  'Nature',\n",
       "  '10.1038/35084076'),\n",
       " '11343123': ('ICOS is essential for effective T-helper-cell responses.',\n",
       "  ['Tafuri A', 'Shahinian A', 'Bladt F', 'Yoshinaga SK', 'Jordana M', 'Wakeham A', 'Boucher LM', 'Bouchard D', 'Chan VS', 'Duncan G', 'Odermatt B', 'Ho A', 'Itie A', 'Horan T', 'Whoriskey JS', 'Pawson T', 'Penninger JM', 'Ohashi PS', 'Mak TW'],\n",
       "  '2001',\n",
       "  'Nature',\n",
       "  '10.1038/35051113'),\n",
       " '11130078': ('A Toll-like receptor recognizes bacterial DNA.',\n",
       "  ['Hemmi H', 'Takeuchi O', 'Kawai T', 'Kaisho T', 'Sato S', 'Sanjo H', 'Matsumoto M', 'Hoshino K', 'Wagner H', 'Takeda K', 'Akira S'],\n",
       "  '2000',\n",
       "  'Nature',\n",
       "  '10.1038/35047123'),\n",
       " '10917520': ('Structure of the Fc fragment of human IgE bound to its high-affinity receptor Fc epsilonRI alpha.',\n",
       "  ['Garman SC', 'Wurzburg BA', 'Tarchevskaya SS', 'Kinet JP', 'Jardetzky TS'],\n",
       "  '2000',\n",
       "  'Nature',\n",
       "  '10.1038/35018500'),\n",
       " '10688206': ('How self-tolerance and the immunosuppressive drug FK506 prevent B-cell mitogenesis.',\n",
       "  ['Glynne R', 'Akkaraju S', 'Healy JI', 'Rayner J', 'Goodnow CC', 'Mack DH'],\n",
       "  '2000',\n",
       "  'Nature',\n",
       "  '10.1038/35001102'),\n",
       " '10586891': ('Induction and regulation of the IgE response.',\n",
       "  ['Corry DB', 'Kheradmand F'],\n",
       "  '1999',\n",
       "  'Nature',\n",
       "  '10.1038/35037014'),\n",
       " '10586890': ('The role of allergy in the development of asthma.',\n",
       "  ['Holt PG', 'Macaubas C', 'Stumbles PA', 'Sly PD'],\n",
       "  '1999',\n",
       "  'Nature',\n",
       "  '10.1038/35037009')}"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CA_summ = {\n",
    "    p['Id'] : ( p['Title'], p['AuthorList'], p['PubDate'][:4], p['FullJournalName'], p.get('DOI', '') )\n",
    "    for p in Entrez.read( Entrez.esummary(db=\"pubmed\", id=','.join( CA_ids )) )\n",
    "    }\n",
    "CA_summ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we restricted our search to papers in an open-access journal, you can then follow their DOIs to freely access their PDFs at the journal's website."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now issue calls to `Entrez.elink()` using these PubMed IDs, to retrieve the IDs of papers that cite them.\n",
    "The database from which the IDs will be retrieved is [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc/), a free digital database of full-text scientific literature in the biomedical and life sciences.\n",
    "\n",
    "A complete list of the kinds of links you can retrieve with `Entrez.elink()` can be found [here](http://eutils.ncbi.nlm.nih.gov/entrez/query/static/entrezlinks.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "CA_citing = {\n",
    "    id : Entrez.read( Entrez.elink(\n",
    "            cmd = \"neighbor\",               # ELink command mode: \"neighbor\", returns\n",
    "                                            #     a set of UIDs in `db` linked to the input UIDs in `dbfrom`.\n",
    "            dbfrom = \"pubmed\",              # Database containing the input UIDs: PubMed\n",
    "            db = \"pmc\",                     # Database from which to retrieve UIDs: PubMed Central\n",
    "            LinkName = \"pubmed_pmc_refs\",   # Name of the Entrez link to retrieve: \"pubmed_pmc_refs\", gets\n",
    "                                            #     \"Full-text articles in the PubMed Central Database that cite the current articles\"\n",
    "            from_uid = id                   # input UIDs\n",
    "            ) )\n",
    "    for id in CA_ids\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[DictElement({'ERROR': [], 'LinkSetDb': [DictElement({'Link': [DictElement({'Id': '6033188'}, attributes={}), DictElement({'Id': '5525434'}, attributes={}), DictElement({'Id': '5010491'}, attributes={}), DictElement({'Id': '4790257'}, attributes={}), DictElement({'Id': '4729269'}, attributes={}), DictElement({'Id': '4715494'}, attributes={}), DictElement({'Id': '4593748'}, attributes={}), DictElement({'Id': '4547136'}, attributes={}), DictElement({'Id': '4265371'}, attributes={}), DictElement({'Id': '4164235'}, attributes={}), DictElement({'Id': '4026961'}, attributes={}), DictElement({'Id': '3924148'}, attributes={}), DictElement({'Id': '3789590'}, attributes={}), DictElement({'Id': '3459673'}, attributes={}), DictElement({'Id': '3434398'}, attributes={})], 'DbTo': 'pmc', 'LinkName': 'pubmed_pmc_refs'}, attributes={})], 'LinkSetDbHistory': [], 'DbFrom': 'pubmed', 'IdList': ['22538601']}, attributes={})]"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CA_citing['22538601']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have in `CA_citing[paper_id][0]['LinkSetDb'][0]['Link']` the list of papers citing `paper_id`. To get it as just a list of ids, we can do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['6033188',\n",
       " '5525434',\n",
       " '5010491',\n",
       " '4790257',\n",
       " '4729269',\n",
       " '4715494',\n",
       " '4593748',\n",
       " '4547136',\n",
       " '4265371',\n",
       " '4164235',\n",
       " '4026961',\n",
       " '3924148',\n",
       " '3789590',\n",
       " '3459673',\n",
       " '3434398']"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cits = [ l['Id'] for l in CA_citing['22538601'][0]['LinkSetDb'][0]['Link'] ]\n",
    "cits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, one more step is needed, as what we have now are PubMed Central IDs, and not PubMed IDs. Their conversion can be achieved through an additional call to `Entrez.elink()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[DictElement({'ERROR': [], 'LinkSetDb': [DictElement({'Link': [DictElement({'Id': '29944755'}, attributes={}), DictElement({'Id': '28790503'}, attributes={}), DictElement({'Id': '27225312'}, attributes={}), DictElement({'Id': '26776074'}, attributes={}), DictElement({'Id': '26410782'}, attributes={}), DictElement({'Id': '26299686'}, attributes={}), DictElement({'Id': '26228472'}, attributes={}), DictElement({'Id': '26210895'}, attributes={}), DictElement({'Id': '25525298'}, attributes={}), DictElement({'Id': '24657550'}, attributes={}), DictElement({'Id': '24592267'}, attributes={}), DictElement({'Id': '24210352'}, attributes={}), DictElement({'Id': '23827958'}, attributes={}), DictElement({'Id': '22970345'}, attributes={}), DictElement({'Id': '22886110'}, attributes={})], 'DbTo': 'pubmed', 'LinkName': 'pmc_pubmed'}, attributes={})], 'LinkSetDbHistory': [], 'DbFrom': 'pmc', 'IdList': ['6033188', '5525434', '5010491', '4790257', '4729269', '4715494', '4593748', '4547136', '4265371', '4164235', '4026961', '3924148', '3789590', '3459673', '3434398']}, attributes={})]"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cits_pm = Entrez.read( Entrez.elink( dbfrom=\"pmc\", db=\"pubmed\", LinkName=\"pmc_pubmed\", from_uid=\",\".join(cits)) )\n",
    "cits_pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'6033188': '29944755',\n",
       " '5525434': '28790503',\n",
       " '5010491': '27225312',\n",
       " '4790257': '26776074',\n",
       " '4729269': '26410782',\n",
       " '4715494': '26299686',\n",
       " '4593748': '26228472',\n",
       " '4547136': '26210895',\n",
       " '4265371': '25525298',\n",
       " '4164235': '24657550',\n",
       " '4026961': '24592267',\n",
       " '3924148': '24210352',\n",
       " '3789590': '23827958',\n",
       " '3459673': '22970345',\n",
       " '3434398': '22886110'}"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ids_map = { pmc_id : link['Id'] for (pmc_id,link) in zip(cits_pm[0]['IdList'], cits_pm[0]['LinkSetDb'][0]['Link']) }\n",
    "ids_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to check these papers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'29944755': ('Immunoglobulin gene analysis as a tool for investigating human immune responses.',\n",
       "  ['Dunn-Walters D', 'Townsend C', 'Sinclair E', 'Stewart A'],\n",
       "  '2018',\n",
       "  'Immunological reviews',\n",
       "  '10.1111/imr.12659'),\n",
       " '28790503': ('Mast Cells and IgE can Enhance Survival During Innate and Acquired Host Responses to Venoms.',\n",
       "  ['Galli SJ', 'Starkl P', 'Marichal T', 'Tsai M'],\n",
       "  '2017',\n",
       "  'Transactions of the American Clinical and Climatological Association',\n",
       "  ''),\n",
       " '27225312': ('IgE and mast cells in host defense against parasites and venoms.',\n",
       "  ['Mukai K', 'Tsai M', 'Starkl P', 'Marichal T', 'Galli SJ'],\n",
       "  '2016',\n",
       "  'Seminars in immunopathology',\n",
       "  '10.1007/s00281-016-0565-1'),\n",
       " '26776074': ('The Mast Cell-IgE Paradox: From Homeostasis to Anaphylaxis.',\n",
       "  ['Galli SJ'],\n",
       "  '2016',\n",
       "  'The American journal of pathology',\n",
       "  '10.1016/j.ajpath.2015.07.025'),\n",
       " '26410782': ('IgE antibodies, FcεRIα, and IgE-mediated local anaphylaxis can limit snake venom toxicity.',\n",
       "  ['Starkl P', 'Marichal T', 'Gaudenzio N', 'Reber LL', 'Sibilano R', 'Tsai M', 'Galli SJ'],\n",
       "  '2016',\n",
       "  'The Journal of allergy and clinical immunology',\n",
       "  '10.1016/j.jaci.2015.08.005'),\n",
       " '26299686': ('Non-immune immunoglobulins shield Schistosoma japonicum from host immunorecognition.',\n",
       "  ['Wu C', 'Hou N', 'Piao X', 'Liu S', 'Cai P', 'Xiao Y', 'Chen Q'],\n",
       "  '2015',\n",
       "  'Scientific reports',\n",
       "  '10.1038/srep13434'),\n",
       " '26228472': ('Why do we study animal toxins?',\n",
       "  ['Zhang Y'],\n",
       "  '2015',\n",
       "  'Dong wu xue yan jiu = Zoological research',\n",
       "  ''),\n",
       " '26210895': (\"Testing the 'toxin hypothesis of allergy': mast cells, IgE, and innate and acquired immune responses to venoms.\",\n",
       "  ['Tsai M', 'Starkl P', 'Marichal T', 'Galli SJ'],\n",
       "  '2015',\n",
       "  'Current opinion in immunology',\n",
       "  '10.1016/j.coi.2015.07.001'),\n",
       " '25525298': ('Osteopontin expression in the brain triggers localized inflammation and cell death when immune cells are activated by pertussis toxin.',\n",
       "  ['Marcondes MC', 'Ojakian R', 'Bortell N', 'Flynn C', 'Conti B', 'Fox HS'],\n",
       "  '2014',\n",
       "  'Mediators of inflammation',\n",
       "  '10.1155/2014/358218'),\n",
       " '24657550': ('Human IgE responses to different splice variants of Schistosoma mansoni tropomyosin: associations with immunity.',\n",
       "  ['Silas S', 'Fitzsimmons CM', 'Jones FM', 'Pinot de Moira A', 'Wawrzyniak J', 'Tukahebwa EM', 'Dunne DW'],\n",
       "  '2014',\n",
       "  'International journal for parasitology',\n",
       "  '10.1016/j.ijpara.2014.02.004'),\n",
       " '24592267': ('Helminth Allergens, Parasite-Specific IgE, and Its Protective Role in Human Immunity.',\n",
       "  ['Fitzsimmons CM', 'Falcone FH', 'Dunne DW'],\n",
       "  '2014',\n",
       "  'Frontiers in immunology',\n",
       "  '10.3389/fimmu.2014.00061'),\n",
       " '24210352': ('A beneficial role for immunoglobulin E in host defense against honeybee venom.',\n",
       "  ['Marichal T', 'Starkl P', 'Reber LL', 'Kalesnikoff J', 'Oettgen HC', 'Tsai M', 'Metz M', 'Galli SJ'],\n",
       "  '2013',\n",
       "  'Immunity',\n",
       "  '10.1016/j.immuni.2013.10.005'),\n",
       " '23827958': ('Type 2 immunity and wound healing: evolutionary refinement of adaptive immunity by helminths.',\n",
       "  ['Gause WC', 'Wynn TA', 'Allen JE'],\n",
       "  '2013',\n",
       "  'Nature reviews. Immunology',\n",
       "  '10.1038/nri3476'),\n",
       " '22970345': ('Human schistosome infection and allergic sensitisation.',\n",
       "  ['Rujeni N', 'Taylor DW', 'Mutapi F'],\n",
       "  '2012',\n",
       "  'Journal of parasitology research',\n",
       "  '10.1155/2012/154743'),\n",
       " '22886110': ('Innate immunostimulatory properties of allergens and their relevance to food allergy.',\n",
       "  ['Ruiter B', 'Shreffler WG'],\n",
       "  '2012',\n",
       "  'Seminars in immunopathology',\n",
       "  '10.1007/s00281-012-0334-8')}"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{   p['Id'] : ( p['Title'], p['AuthorList'], p['PubDate'][:4], p['FullJournalName'], p.get('DOI', '') )\n",
    "    for p in Entrez.read( Entrez.esummary(db=\"pubmed\", id=','.join( ids_map.values() )) )\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now seen all the steps required to assemble a dataset of citations to each of the papers in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "Citations_file = 'data/' + search_term + '_Citations.pkl.bz2'\n",
    "Citations = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At least one server query will be issued per paper in `Ids`. Because NCBI allows for at most 3 queries per second (see [here](http://biopython.org/DIST/docs/api/Bio.Entrez-pysrc.html#_open)), this dataset will take a long time to assemble. Should you need to interrupt it for some reason, or the connection fail at some point, it is safe to just rerun the cell below until all data is collected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "....\tsaving at checkpoint 70848\n",
      ".......\tsaving at checkpoint 71586\n",
      "........\tsaving at checkpoint 72324\n",
      ".......\tsaving at checkpoint 73062\n",
      ".......\n",
      " done.\n"
     ]
    }
   ],
   "source": [
    "import http.client\n",
    "\n",
    "if Citations == [] and os.path.exists( Citations_file ):\n",
    "    Citations = pickle.load( bz2.BZ2File( Citations_file, 'rb' ) )\n",
    "\n",
    "if len(Citations) < len(Ids):\n",
    "    \n",
    "    i = len(Citations)\n",
    "    checkpoint = int(len(Ids) / 100) + 1      # save to hard drive at every 1% of Ids fetched\n",
    "    \n",
    "    for pm_id in Ids[i:]:               # either starts from index 0, or resumes from where we previously left off\n",
    "        \n",
    "        while True:\n",
    "            try:\n",
    "                # query for papers archived in PubMed Central that cite the paper with PubMed ID `pm_id`\n",
    "                c = Entrez.read( Entrez.elink( dbfrom = \"pubmed\", db=\"pmc\", LinkName = \"pubmed_pmc_refs\", id=str(pm_id) ) )\n",
    "                \n",
    "                c = c[0]['LinkSetDb']\n",
    "                if len(c) == 0:\n",
    "                    # no citations found for the current paper\n",
    "                    c = []\n",
    "                else:\n",
    "                    c = [ l['Id'] for l in c[0]['Link'] ]\n",
    "                    \n",
    "                    # convert citations from PubMed Central IDs to PubMed IDs\n",
    "                    p = []\n",
    "                    retrieve_per_query = 500\n",
    "                    for start in range( 0, len(c), retrieve_per_query ):\n",
    "                        query_ids = ','.join( c[start : start+retrieve_per_query] )\n",
    "                        r = Entrez.read( Entrez.elink( dbfrom=\"pmc\", db=\"pubmed\", LinkName=\"pmc_pubmed\", from_uid=query_ids ) )\n",
    "                        # select the IDs. If no matching PubMed ID was found, [] is returned instead\n",
    "                        p.extend( [] if r[0]['LinkSetDb']==[] else [ int(link['Id']) for link in r[0]['LinkSetDb'][0]['Link'] ] )\n",
    "                    c = p\n",
    "            \n",
    "            except http.client.BadStatusLine:\n",
    "                # Presumably, the server closed the connection before sending a valid response. Retry until we have the data.\n",
    "                print('r')\n",
    "                continue\n",
    "            except HTTPError:\n",
    "                print('r')\n",
    "                continue\n",
    "            break\n",
    "        \n",
    "        Citations.append( (pm_id, c) )\n",
    "        if (i % 10000 == 0):\n",
    "            print('')\n",
    "            print(i, end='')\n",
    "        if (i % 100 == 0):\n",
    "            print('.', end='')\n",
    "        i += 1\n",
    "        \n",
    "        if i % checkpoint == 0:\n",
    "            print('\\tsaving at checkpoint', i)\n",
    "            pickle.dump( Citations, bz2.BZ2File( Citations_file, 'wb' ) )\n",
    "    \n",
    "    print('\\n done.')\n",
    "    \n",
    "    # Save Citations, as a dictionary indexed by Ids\n",
    "    Citations = dict( Citations )\n",
    "    \n",
    "    pickle.dump( Citations, bz2.BZ2File( Citations_file, 'wb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see that we have indeed obtained the data we expected, you can match the ids below, with the ids listed at the end of last section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[29944755,\n",
       " 28790503,\n",
       " 27225312,\n",
       " 26776074,\n",
       " 26410782,\n",
       " 26299686,\n",
       " 26228472,\n",
       " 26210895,\n",
       " 25525298,\n",
       " 24657550,\n",
       " 24592267,\n",
       " 24210352,\n",
       " 23827958,\n",
       " 22970345,\n",
       " 22886110]"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Citations[22538601]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where do we go from here?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the code above generates multiple local files, containing the datasets we'll be working with. Loading them into memory is a matter of just issuing a call like<br>\n",
    "``data = pickle.load( bz2.BZ2File( data_file, 'rb' ) )``.\n",
    "\n",
    "The Entrez module will therefore no longer be needed, unless you wish to extend your data processing with additional information retrieved from PubMed.\n",
    "\n",
    "Should you be interested in looking at alternative ways to handle the data, have a look at the [sqlite3](http://docs.python.org/3/library/sqlite3.html) module included in Python's standard library, or [Pandas](http://pandas.pydata.org/), the Python Data Analysis Library."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}