notebooks/11a-QuickUMLS_Extraction_Defaults.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "from os import path\n",
    "from pathlib import Path\n",
    "\n",
    "from quickumls.constants import MEDSPACY_DEFAULT_SPAN_GROUP_NAME\n",
    "import quickumls.spacy_component\n",
    "\n",
    "sys.path.insert(0, \"..\")\n",
    "from medspacy.util import get_quickumls_demo_dir\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook provides a brief introduction into using QuickUMLS in medspacy as well as details on how it can be used in any spacy pipeline\n",
    "### Data: A full QuickUMLS resource of the UMLS is not provided here as this would violate license agreements.  However, below are some resources on how to generate such resources.  In this medspacy repo is a small sample of UMLS (RRF files) which contains less than 100 concepts which can be found here:\n",
    "https://www.nlm.nih.gov/research/umls/new_users/online_learning/Meta_006.html\n",
    "### Usage: The cells below show how to use the QuickUMLS component on its own or in combination with other medspacy components out of the box such as `medspacy.context` for detecting semantic modifiers and attributes of entities, including negation, uncertainty and others.\n",
    "### NOTE: There are two major result \"modes\" for using QuickUMLS.  The default is to use entities in spacy documents.  The limitation of this is that matches cannot overlap.  The other \"mode\" is to use (SpanGroups)[https://spacy.io/api/spangroup] which were introduced in spacy 3.  These spans can overlap and several users have requested such functionality since UMLS has many concepts which may overlap which you may want to process after the QuickUMLS component.  More of this is demonstrated in the notebook for Advanced QuickUMLS behavior.\n",
    "### Generating QuickUMLS resources: Given RRF UMLS files, you can generate your own QuickUMLS resources with parameters such as language, character case and more.  To see more, consult the documentation here from the original QuickUMLS repo:\n",
    "https://github.com/Georgetown-IR-Lab/QuickUMLS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import spacy\n",
    "import nltk\n",
    "sys.path.insert(0, \"..\")\n",
    "import medspacy\n",
    "\n",
    "from medspacy.util import DEFAULT_PIPE_NAMES\n",
    "from medspacy.visualization import visualize_ent\n",
    "from medspacy.section_detection import Sectionizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on platform: darwin\n"
     ]
    }
   ],
   "source": [
    "print('Running on platform: {}'.format(sys.platform))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Enable the QuickUMLS component by name since it is not enabled by default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'medspacy_tokenizer', 'medspacy_quickumls', 'medspacy_target_matcher', 'medspacy_pyrush', 'medspacy_context'}\n"
     ]
    }
   ],
   "source": [
    "medspacy_pipes = DEFAULT_PIPE_NAMES.copy()\n",
    "\n",
    "if 'medspacy_quickumls' not in medspacy_pipes: \n",
    "    medspacy_pipes.add('medspacy_quickumls')\n",
    "    \n",
    "print(medspacy_pipes)\n",
    "    \n",
    "nlp = medspacy.load(enable = medspacy_pipes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check which pipe components have been enabled.  This way we ensure that the QuickUMLS matcher is in the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['medspacy_pyrush', 'medspacy_target_matcher', 'medspacy_context']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First's see a visualization of one of the concepts in the small sample of UMLS provided.  This concept is \"dipalmitoyllecithin\" which is Concept Unique Identifier (CUI) C0000039 in UMLS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "concept_text = 'Decreased dipalmitoyllecithin content found in lung specimens'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(concept_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Decreased dipalmitoyllecithin content found in lung specimens</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_ent(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## However, there is additional metadata about any concept extracted by QuickUMLS.  For example, any CUI like this one can be a member of multiple Semantic Type.  In this case, the concept belongs to more than one Semantic Type.  Additionally, since QuickUMLS performs approximate matching, note that the similarity of the extracted concept from our text to the canonical resources can be observed.  In this case, since there is no lexical difference, we see 100% (1.0) similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ent in doc.ents:\n",
    "    print('Entity text : {}'.format(ent.text))\n",
    "    \n",
    "    # each span may have multiple matches (different CUIs and similarity) so let's loop over that\n",
    "    for umls_match in ent._.umls_matches:\n",
    "        print('Label (UMLS CUI) : {}'.format(ent.label_))\n",
    "        print('CUI: {}'.format(umls_match.cui))\n",
    "        print('Similarity : {}'.format(umls_match.similarity))\n",
    "        print('Semtypes : {}'.format(umls_match.semtypes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So this is an example of how to use QuickUMLS on its own.  What if we want to see negation as well?  Remember that QuickUMLS enables the `medspacy.context` component by default.  It's here in this list and we did not need to explicitly enable it.  Further, notice that the QuickUMLS component is ordered just before context.  This ensures that the Entity objects are added to the spacy `Doc` before `context` runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['medspacy_pyrush', 'medspacy_target_matcher', 'medspacy_context']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now let's come up with a different example with negation on a different lexical variant for the same UMLS concept in the relatively small \"sample\" resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "negation_text = 'No findings of Dipalmitoyl Phosphatidylcholine in pulmonary specimen.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "negation_doc = nlp(negation_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ent in negation_doc.ents:\n",
    "    print('Entity text : {}'.format(ent.text))\n",
    "    \n",
    "    # each span may have multiple matches (different CUIs and similarity) so let's loop over that\n",
    "    for umls_match in ent._.umls_matches:\n",
    "        print('Label (UMLS CUI) : {}'.format(ent.label_))\n",
    "        print('CUI: {}'.format(umls_match.cui))\n",
    "        print('Similarity : {}'.format(umls_match.similarity))\n",
    "        print('Semtypes : {}'.format(umls_match.semtypes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note that the `context` component adds the attributes about the entity such as negation and others into members of the \"underscore\" (`_`) which can also be examined like this: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ent in negation_doc.ents:\n",
    "    if any([ent._.is_negated, ent._.is_uncertain, ent._.is_historical, ent._.is_family, ent._.is_hypothetical, ]):\n",
    "        print(\"'{0}' modified by {1} in: '{2}'\".format(ent, ent._.modifiers, ent.sent))\n",
    "        print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}