notebooks/nlp/spaCy/howto-section_detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section Detection Using Medspacy\n",
    "\n",
    "This example demonstreates how find sections based on regex pattern (not medspacy related) and dynamically build sections from results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Load medspacy dependencies.\n",
    "Install python dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "bat"
    }
   },
   "outputs": [],
   "source": [
    "!python -m pip install medspacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 460,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class Sectionizer in module medspacy.section_detection.sectionizer:\n",
      "\n",
      "class Sectionizer(builtins.object)\n",
      " |  Sectionizer(nlp: spacy.language.Language, name: str = 'medspacy_sectionizer', rules: Optional[str] = 'default', max_section_length: Optional[int] = None, phrase_matcher_attr: str = 'LOWER', require_start_line: bool = False, require_end_line: bool = False, newline_pattern: str = '[\\\\n\\\\r]+[\\\\s]*$', input_span_type: Optional[Literal['ents', 'group']] = 'ents', span_group_name: str = 'medspacy_spans', span_attrs: Union[Literal['default'], Dict[str, Dict[str, Any]], NoneType] = 'default')\n",
      " |  \n",
      " |  The Sectionizer will search for spans in the text which match section header rules, such as 'Past Medical History:'.\n",
      " |  Sections will be represented in custom attributes as:\n",
      " |      category: A normalized title of the section. Example: 'past_medical_history'\n",
      " |      section_title: The Span of the doc which was matched as a section header.\n",
      " |          Example: 'Past Medical History:'\n",
      " |      section_span: The entire section of the note, starting with section_header and up until the end\n",
      " |          of the section, which will be either the start of the next section header of some pre-specified\n",
      " |          scope. Example: 'Past Medical History: Type II DM'\n",
      " |  \n",
      " |  Section attributes will be registered for each Doc, Span, and Token in the following attributes:\n",
      " |      Doc._.sections: A list of namedtuples of type Section with 4 elements:\n",
      " |          - section_title\n",
      " |          - section_header\n",
      " |          - section_parent\n",
      " |          - section_span.\n",
      " |      A Doc will also have attributes corresponding to lists of each\n",
      " |          (ie., Doc._.section_titles, Doc._.section_headers, Doc._.section_parents, Doc._.section_list)\n",
      " |      (Span|Token)._.section_title\n",
      " |      (Span|Token)._.section_header\n",
      " |      (Span|Token)._.section_parent\n",
      " |      (Span|Token)._.section_span\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __call__(self, doc: spacy.tokens.doc.Doc) -> spacy.tokens.doc.Doc\n",
      " |      Call the Sectionizer on a spaCy doc. Sectionizer will identify sections using provided rules, then evaluate any\n",
      " |      section hierarchy as needed, create section spans, and modify attributes on existing spans based on the sections\n",
      " |      the entities spans in.\n",
      " |      \n",
      " |      Args:\n",
      " |          doc: The Doc to process.\n",
      " |      \n",
      " |      Returns:\n",
      " |          The processed spaCy Doc.\n",
      " |  \n",
      " |  __init__(self, nlp: spacy.language.Language, name: str = 'medspacy_sectionizer', rules: Optional[str] = 'default', max_section_length: Optional[int] = None, phrase_matcher_attr: str = 'LOWER', require_start_line: bool = False, require_end_line: bool = False, newline_pattern: str = '[\\\\n\\\\r]+[\\\\s]*$', input_span_type: Optional[Literal['ents', 'group']] = 'ents', span_group_name: str = 'medspacy_spans', span_attrs: Union[Literal['default'], Dict[str, Dict[str, Any]], NoneType] = 'default')\n",
      " |             Create a new Sectionizer component.\n",
      " |      \n",
      " |             Args:\n",
      " |                 nlp: A SpaCy Language object.\n",
      " |                 name: The name of the component.\n",
      " |                 rules: The rules to load. Default is \"default\", loads rules packaged with medspaCy that are derived from\n",
      " |                     SecTag, MIMIC-III, and practical refinement at the US Department of Veterans Affairs. If None, no rules\n",
      " |                     are loaded. Otherwise, must be a path to a json file containing rules. Add SectionRules directly through\n",
      " |                     `Sectionizer.add`.\n",
      " |                 max_section_length: Optional argument specifying the maximum number of tokens following a section header\n",
      " |                     which can be included in a section body. This can be useful if you think your section rules are\n",
      " |                     incomplete and want to prevent sections from running too long in the note. Default is None, meaning that\n",
      " |                     the scope of a section will be until either the next section header or the end of the document.\n",
      " |                 phrase_matcher_attr: The token attribute to use for PhraseMatcher for rules where `pattern` is None. Default\n",
      " |                     is 'LOWER'.\n",
      " |                 require_start_line: Optionally require a section header to start on a new line. Default False.\n",
      " |                 require_end_line: Optionally require a section header to end with a new line. Default False.\n",
      " |                 newline_pattern: Regular expression to match the new line either preceding or following a header\n",
      " |                     if either require_start_line or require_end_line are True. Default is r\"[\n",
      " |      ]+[\\s]*$\"\n",
      " |                 span_attrs: The optional span attributes to modify. Default option \"default\" uses attributes in\n",
      " |                     `DEFAULT_ATTRIBUTES`. If a dictionary of custom attributes, format is a dictionary mapping section\n",
      " |                     categories to a dictionary containing the attribute name and the value to set the attribute to when a\n",
      " |                     span is contained in a section of that category. Custom attributes must be assigned with\n",
      " |                     `Span.set_extension` before creating the Sectionizer. If None, sectionizer will not modify span\n",
      " |                     attributes.\n",
      " |                 input_span_type: \"ents\" or \"group\". Where to look for spans when modifying attributes of spans\n",
      " |                     contained in a section if `span_attrs` is not None. \"ents\" will modify attributes of spans in doc.ents.\n",
      " |                     \"group\" will modify attributes of spans in the span group specified by `span_group_name`.\n",
      " |                 span_group_name: The name of the span group used when `input_span_type` is \"group\". Default is\n",
      " |                     \"medspacy_spans\".\n",
      " |  \n",
      " |  add(self, rules)\n",
      " |      Adds SectionRules to the Sectionizer.\n",
      " |      \n",
      " |      Args:\n",
      " |          rules: A single SectionRule or a collection of SectionRules to add to the Sectionizer.\n",
      " |  \n",
      " |  filter_end_lines(self, doc: spacy.tokens.doc.Doc, matches: List[Tuple[int, int, int]]) -> List[Tuple[int, int, int]]\n",
      " |      Filter a list of matches to only contain spans where the start token is followed by a new line.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A list of match tuples (match_id, start, end) that meet the filter criteria.\n",
      " |  \n",
      " |  filter_start_lines(self, doc: spacy.tokens.doc.Doc, matches: List[Tuple[int, int, int]]) -> List[Tuple[int, int, int]]\n",
      " |      Filter a list of matches to only contain spans where the start token is the beginning of a new line.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A list of match tuples (match_id, start, end) that meet the filter criteria.\n",
      " |  \n",
      " |  set_assertion_attributes(self, spans: Iterable[spacy.tokens.span.Span])\n",
      " |      Add Span-level attributes to entities based on which section they occur in.\n",
      " |      \n",
      " |      Args:\n",
      " |          spans: the spans to modify.\n",
      " |  \n",
      " |  set_parent_sections(self, sections: List[Tuple[int, int, int]]) -> List[Tuple[int, int, int, int]]\n",
      " |      Determine the legal parent-child section relationships from the list\n",
      " |      of in-order sections of a document and the possible parents of each\n",
      " |      section as specified during direction creation.\n",
      " |      \n",
      " |      Args:\n",
      " |          sections: a list of spacy match tuples found in the doc\n",
      " |      \n",
      " |      Returns:\n",
      " |          A list of tuples (match_id, start, end, parent_idx) where the first three indices are the same as the input\n",
      " |          and the added parent_idx represents the index in the list that corresponds to the parent section. Might be a\n",
      " |          smaller list than the input due to pruning with `parent_required`.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods defined here:\n",
      " |  \n",
      " |  register_default_attributes() from builtins.type\n",
      " |      Register the default values for the Span attributes defined in `DEFAULT_ATTRIBUTES`.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Readonly properties defined here:\n",
      " |  \n",
      " |  rules\n",
      " |      Gets list of rules associated with the Sectionizer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          The list of SectionRules associated with the Sectionizer.\n",
      " |  \n",
      " |  section_categories\n",
      " |      Gets a list of categories used in the Sectionizer.\n",
      " |      \n",
      " |      Returns:\n",
      " |              The list of all section categories available to the Sectionizer.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors defined here:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      " |  \n",
      " |  input_span_type\n",
      " |      The input source of entities for the component. Must be either \"ents\" corresponding to doc.ents or \"group\" for\n",
      " |      a spaCy span group.\n",
      " |      \n",
      " |      Returns:\n",
      " |          The input type, \"ents\" or \"group\".\n",
      " |  \n",
      " |  span_group_name\n",
      " |      The name of the span group used by this component. If `input_type` is \"group\", calling this component will\n",
      " |      use spans in the span group with this name.\n",
      " |      \n",
      " |      Returns:\n",
      " |          The span group name.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from medspacy.section_detection import Sectionizer, SectionRule\n",
    "\n",
    "nlp = medspacy.load()\n",
    "sectionizer = nlp.add_pipe(\"medspacy_sectionizer\", config={\"rules\":None})\n",
    "\n",
    "help(Sectionizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Define Medical Document & Identify Sections\n",
    "\n",
    "In this experiment, regex patterns are used to identify new sections. The default medspacy sectionizer does not pickup all of the sections. Some pre-processing is done to clean up the section names since they will be used in a for loop to create new section rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 461,
   "metadata": {
    "vscode": {
     "languageId": "bat"
    }
   },
   "outputs": [],
   "source": [
    "medical_doc = \"\"\"REASON FOR REFERRAL: , Ms. A is a 60-year-old African-American female with 12 years of education who was referred for neuropsychological evaluation by Dr. X after she demonstrated mild cognitive deficits on a neuropsychological screening evaluation during a followup appointment with him for stroke in July.  A comprehensive evaluation was requested to assess current cognitive functioning and assist with diagnostic decisions and treatment planning.,RELEVANT BACKGROUND INFORMATION:,  Historical information was obtained from a review of available medical records and clinical interview with Ms. A.  A summary of pertinent information is presented below.  Please refer to the patient's medical chart for a more complete history.,HISTORY OF PRESENTING PROBLEM:,  Ms. A presented to the ABC Hospital emergency department on 07/26/2009 reporting that after she had woken up that morning she noticed numbness and weakness in her left hand, slurred speech, and left facial droop.  Neurological evaluation with Dr. X confirmed left hemiparesis.  Brain CT showed no evidence of intracranial hemorrhage or mass effect and that she received TPA and had moderate improvement in left-sided weakness.  These symptoms were thought to be due to a right middle cerebral artery stroke.  She was transferred to the ICU for monitoring.  Ultrasound of the carotids showed 20% to 30% stenosis of the right ICA and 0% to 19% stenosis of the left ICA.  On 07/29/2009, she was admitted for acute inpatient rehabilitation for the treatment of residual functional deficits of her acute ischemic right MCA/CVA.  At discharge on 08/06/2009, she was mainly on supervision for all ADLs and walking with a rolling walker, but tolerating increased ambulation with a cane.  She was discharged home with recommendations for outpatient physical therapy.  She returned to the Sinai ER on 08/2009/2009 due to reported left arm pain, numbness, and weakness, which lasted 10 to 15 minutes and she reported that it felt \"just like the stroke.\"  Brain CT on 08/2009/2009 was read as showing \"mild chronic microvascular ischemic change of deep white matter,\" but no acute or significant interval change compared to her previous scan.  Neurological examination with Dr. Y was within normal limits, but she was admitted for a more extensive workup.  Due to left arm pain an ultrasound was completed on her left upper extremity, but it did not show deep vein thrombosis.,Followup CT on 08/10/2009 showed no significant interval change.  MRI could not be completed due to the patient's weight.  She was discharged on 08/11/2009 in stable condition after it was determined that this event was not neurological in origin; however, note that Ms. A referred to this as a second stroke.,Ms. A presented for a followup outpatient neurological evaluation with Dr. X on 09/22/2009, at which time a brief neuropsychological screening was also conducted.  She demonstrated significant impairments in confrontation naming, abstract verbal reasoning, and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full-time work duty.  During the current interview, Ms. A reported that she noticed mild memory problems including some difficultly remembering conversations, events, and at times forgetting to take her medications.  She also reported mild difficulty finding words in conversation, solving novel problems and tasks (e.g. difficulty learning to use her camcorder), but overall denied significant cognitive deficits in attention, concentration, language or other areas of cognitive functioning.  When asked about her return to work, she said that she was still on light duty due to limited physical activity because of residual left leg weakness.  She reported that no one had indicated to her that she appeared less capable of performing her job duties, but said that she was also receiving fewer files to process and enter data into the computer at the Social Security Agency that she works at.  Note also that she had some difficulty explaining exactly what her job involved.  She also reported having problems falling asleep at work and that she is working full-time although on light duty.,OTHER MEDICAL HISTORY:  ,As mentioned, Ms. A continues to have some residual left leg weakness and continues to use a rolling walker for ambulation, but she reported that her motor functioning had improved significantly.  She was diagnosed with sleep apnea approximately two years ago and was recently counseled by Dr. X on the need to use her CPAP because she indicated she never used it at night.  She reported that since her appointment with Dr. X, she has been using it \"every other night.\"  When asked about daytime fatigue, Ms. A initially denied that she was having any difficulties, but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen.  She reported at times \"snoring\" and forgetting where she is at and said that a supervisor offered to give her coffee at one point.  She receives approximately two to five hours of sleep per night.  Other current untreated risk factors include obesity and hypercholesterolemia.  Her medical history is also significant for hypertension, asthma, abdominal adenocarcinoma status post hysterectomy with bilateral salpingo-oophorectomy, colonic benign polyps status post resection, benign lesions of the breast status post lumpectomy, and deep vein thrombosis in the left lower extremity status post six months of anticoagulation (which she had discontinued just prior to her stroke).,CURRENT MEDICATIONS: , Aspirin 81 mg daily, Colace 100 mg b.i.d., Lipitor 80 mg daily, and albuterol MDI p.r.n.,SUBSTANCE USE:,  Ms. A denied drinking alcohol or using illicit drugs.  She used to smoke a pack of cigarettes per day, but quit five to six years ago.,FAMILY MEDICAL HISTORY: , Ms. A had difficulty providing information on familial medical history.  She reported that her mother died three to four years ago from lung cancer.  Her father has gout and blood clots.  Siblings have reportedly been treated for asthma and GI tumors.  She was unsure of familial history of other conditions such as hypertension, high cholesterol, stroke, etc.,SOCIAL HISTORY: , Ms. A completed high school degree.  She reported that she primarily obtained B's and C's in school.  She received some tutoring for algebra in middle school, but denied ever having been held back a grade failing any classes or having any problems with attention or hyperactivity.,She currently works for the Social Security Administration in data processing.  As mentioned, she has returned to full-time work, but continues to perform only light duties due to her physical condition.  She is now living on her own.  She has never driven.  She reported that she continues to perform ADLs independently such as cooking and cleaning.  She lost her husband in 2005 and has three adult daughters.  She previously reported some concerns that her children wanted her to move into assisted living, but she did not discuss that during this current evaluation.  She also reported number of other family members who had recently passed away.  She has returned to activities she enjoys such as quire, knitting, and cooking and plans to go on a cruise to the Bahamas at the end of October.,PSYCHIATRIC HISTORY: , Ms. A did not report a history of psychological or psychiatric treatment.  She reported that her current mood was good, but did describe some anxiety and nervousness about various issues such as her return to work, her upcoming trip, and other events.  She reported that this only \"comes and goes.\",TASKS ADMINISTERED:,Clinical Interview,Adult History Questionnaire,Wechsler Test of Adult Reading (WTAR),Mini Mental Status Exam (MMSE),Cognistat Neurobehavioral Cognitive Status Examination,Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Form XX),Mattis Dementia Rating Scale, 2nd Edition (DRS-2),Neuropsychological Assessment Battery (NAB),Wechsler Adult Intelligence Scale, Third Edition (WAIS-III),Wechsler Adult Intelligence Scale, Fourth Edition (WAIS-IV),Wechsler Abbreviated Scale of Intelligence (WASI),Test of Variables of Attention (TOVA),Auditory Consonant Trigrams (ACT),Paced Auditory Serial Addition Test (PASAT),Ruff 2 & 7 Selective Attention Test,Symbol Digit Modalities Test (SDMT),Multilingual Aphasia Examination, Second Edition (MAE-II),  Token Test,  Sentence Repetition,  Visual Naming,  Controlled Oral Word Association,  Spelling Test,  Aural Comprehension,  Reading Comprehension,Boston Naming Test, Second Edition (BNT-2),Animal Naming Test\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 462,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['REASON FOR REFERRAL',\n",
       " 'RELEVANT BACKGROUND INFORMATION',\n",
       " 'HISTORY OF PRESENTING PROBLEM',\n",
       " 'OTHER MEDICAL HISTORY',\n",
       " 'CURRENT MEDICATIONS',\n",
       " 'SUBSTANCE USE',\n",
       " 'FAMILY MEDICAL HISTORY',\n",
       " 'SOCIAL HISTORY',\n",
       " 'PSYCHIATRIC HISTORY',\n",
       " 'TASKS ADMINISTERED']"
      ]
     },
     "execution_count": 462,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "# Find new sections using regex patterns & clean up strings\n",
    "sections = [x.replace(\":\",\"\").replace(\",\",\"\") for x in re.findall('[A-Z, ]*:', medical_doc)]\n",
    "sections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 - Create Section Rules From Sections List"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 463,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[SectionRule(literal=\"REASON FOR REFERRAL:\", category=\"reason_for_referral\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"RELEVANT BACKGROUND INFORMATION:\", category=\"relevant_background_information\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"HISTORY OF PRESENTING PROBLEM:\", category=\"history_of_presenting_problem\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"OTHER MEDICAL HISTORY:\", category=\"other_medical_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"CURRENT MEDICATIONS:\", category=\"current_medications\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"SUBSTANCE USE:\", category=\"substance_use\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"FAMILY MEDICAL HISTORY:\", category=\"family_medical_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"SOCIAL HISTORY:\", category=\"social_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"PSYCHIATRIC HISTORY:\", category=\"psychiatric_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " SectionRule(literal=\"TASKS ADMINISTERED:\", category=\"tasks_administered\", pattern=None, on_match=None, parents=None, parent_required=False)]"
      ]
     },
     "execution_count": 463,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_patterns = []\n",
    "\n",
    "for s in sections:\n",
    "    section_cat = s.lower().replace(\" \",\"_\").replace(\":\",\"\")\n",
    "    section_lit = \"{}:\".format(s)\n",
    "\n",
    "    new_patterns.append(SectionRule(category=section_cat, literal=section_lit))\n",
    "\n",
    "sectionizer.add(new_patterns)\n",
    "\n",
    "sectionizer.rules\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 464,
   "metadata": {
    "vscode": {
     "languageId": "bat"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    REASON FOR REFERRAL:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< REASON_FOR_REFERRAL >></span>\n",
       "</mark>\n",
       " , Ms. A is a 60-year-old African-American female with 12 years of education who was referred for neuropsychological evaluation by Dr. X after she demonstrated mild cognitive deficits on a neuropsychological screening evaluation during a followup appointment with him for stroke in July.  A comprehensive evaluation was requested to assess current cognitive functioning and assist with diagnostic decisions and treatment planning.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    RELEVANT BACKGROUND INFORMATION:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< RELEVANT_BACKGROUND_INFORMATION >></span>\n",
       "</mark>\n",
       ",  Historical information was obtained from a review of available medical records and clinical interview with Ms. A.  A summary of pertinent information is presented below.  Please refer to the patient's medical chart for a more complete history.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    HISTORY OF PRESENTING PROBLEM:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< HISTORY_OF_PRESENTING_PROBLEM >></span>\n",
       "</mark>\n",
       ",  Ms. A presented to the ABC Hospital emergency department on 07/26/2009 reporting that after she had woken up that morning she noticed numbness and weakness in her left hand, slurred speech, and left facial droop.  Neurological evaluation with Dr. X confirmed left hemiparesis.  Brain CT showed no evidence of intracranial hemorrhage or mass effect and that she received TPA and had moderate improvement in left-sided weakness.  These symptoms were thought to be due to a right middle cerebral artery stroke.  She was transferred to the ICU for monitoring.  Ultrasound of the carotids showed 20% to 30% stenosis of the right ICA and 0% to 19% stenosis of the left ICA.  On 07/29/2009, she was admitted for acute inpatient rehabilitation for the treatment of residual functional deficits of her acute ischemic right MCA/CVA.  At discharge on 08/06/2009, she was mainly on supervision for all ADLs and walking with a rolling walker, but tolerating increased ambulation with a cane.  She was discharged home with recommendations for outpatient physical therapy.  She returned to the Sinai ER on 08/2009/2009 due to reported left arm pain, numbness, and weakness, which lasted 10 to 15 minutes and she reported that it felt &quot;just like the stroke.&quot;  Brain CT on 08/2009/2009 was read as showing &quot;mild chronic microvascular ischemic change of deep white matter,&quot; but no acute or significant interval change compared to her previous scan.  Neurological examination with Dr. Y was within normal limits, but she was admitted for a more extensive workup.  Due to left arm pain an ultrasound was completed on her left upper extremity, but it did not show deep vein thrombosis.,Followup CT on 08/10/2009 showed no significant interval change.  MRI could not be completed due to the patient's weight.  She was discharged on 08/11/2009 in stable condition after it was determined that this event was not neurological in origin; however, note that Ms. A referred to this as a second stroke.,Ms. A presented for a followup outpatient neurological evaluation with Dr. X on 09/22/2009, at which time a brief neuropsychological screening was also conducted.  She demonstrated significant impairments in confrontation naming, abstract verbal reasoning, and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full-time work duty.  During the current interview, Ms. A reported that she noticed mild memory problems including some difficultly remembering conversations, events, and at times forgetting to take her medications.  She also reported mild difficulty finding words in conversation, solving novel problems and tasks (e.g. difficulty learning to use her camcorder), but overall denied significant cognitive deficits in attention, concentration, language or other areas of cognitive functioning.  When asked about her return to work, she said that she was still on light duty due to limited physical activity because of residual left leg weakness.  She reported that no one had indicated to her that she appeared less capable of performing her job duties, but said that she was also receiving fewer files to process and enter data into the computer at the Social Security Agency that she works at.  Note also that she had some difficulty explaining exactly what her job involved.  She also reported having problems falling asleep at work and that she is working full-time although on light duty.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    OTHER MEDICAL HISTORY:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< OTHER_MEDICAL_HISTORY >></span>\n",
       "</mark>\n",
       "  ,As mentioned, Ms. A continues to have some residual left leg weakness and continues to use a rolling walker for ambulation, but she reported that her motor functioning had improved significantly.  She was diagnosed with sleep apnea approximately two years ago and was recently counseled by Dr. X on the need to use her CPAP because she indicated she never used it at night.  She reported that since her appointment with Dr. X, she has been using it &quot;every other night.&quot;  When asked about daytime fatigue, Ms. A initially denied that she was having any difficulties, but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen.  She reported at times &quot;snoring&quot; and forgetting where she is at and said that a supervisor offered to give her coffee at one point.  She receives approximately two to five hours of sleep per night.  Other current untreated risk factors include obesity and hypercholesterolemia.  Her medical history is also significant for hypertension, asthma, abdominal adenocarcinoma status post hysterectomy with bilateral salpingo-oophorectomy, colonic benign polyps status post resection, benign lesions of the breast status post lumpectomy, and deep vein thrombosis in the left lower extremity status post six months of anticoagulation (which she had discontinued just prior to her stroke).,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    CURRENT MEDICATIONS:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< CURRENT_MEDICATIONS >></span>\n",
       "</mark>\n",
       " , Aspirin 81 mg daily, Colace 100 mg b.i.d., Lipitor 80 mg daily, and albuterol MDI p.r.n.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SUBSTANCE USE:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< SUBSTANCE_USE >></span>\n",
       "</mark>\n",
       ",  Ms. A denied drinking alcohol or using illicit drugs.  She used to smoke a pack of cigarettes per day, but quit five to six years ago.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    FAMILY MEDICAL HISTORY:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< FAMILY_MEDICAL_HISTORY >></span>\n",
       "</mark>\n",
       " , Ms. A had difficulty providing information on familial medical history.  She reported that her mother died three to four years ago from lung cancer.  Her father has gout and blood clots.  Siblings have reportedly been treated for asthma and GI tumors.  She was unsure of familial history of other conditions such as hypertension, high cholesterol, stroke, etc.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SOCIAL HISTORY:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< SOCIAL_HISTORY >></span>\n",
       "</mark>\n",
       " , Ms. A completed high school degree.  She reported that she primarily obtained B's and C's in school.  She received some tutoring for algebra in middle school, but denied ever having been held back a grade failing any classes or having any problems with attention or hyperactivity.,She currently works for the Social Security Administration in data processing.  As mentioned, she has returned to full-time work, but continues to perform only light duties due to her physical condition.  She is now living on her own.  She has never driven.  She reported that she continues to perform ADLs independently such as cooking and cleaning.  She lost her husband in 2005 and has three adult daughters.  She previously reported some concerns that her children wanted her to move into assisted living, but she did not discuss that during this current evaluation.  She also reported number of other family members who had recently passed away.  She has returned to activities she enjoys such as quire, knitting, and cooking and plans to go on a cruise to the Bahamas at the end of October.,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    PSYCHIATRIC HISTORY:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< PSYCHIATRIC_HISTORY >></span>\n",
       "</mark>\n",
       " , Ms. A did not report a history of psychological or psychiatric treatment.  She reported that her current mood was good, but did describe some anxiety and nervousness about various issues such as her return to work, her upcoming trip, and other events.  She reported that this only &quot;comes and goes.&quot;,\n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    TASKS ADMINISTERED:\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< TASKS_ADMINISTERED >></span>\n",
       "</mark>\n",
       ",Clinical Interview,Adult History Questionnaire,Wechsler Test of Adult Reading (WTAR),Mini Mental Status Exam (MMSE),Cognistat Neurobehavioral Cognitive Status Examination,Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Form XX),Mattis Dementia Rating Scale, 2nd Edition (DRS-2),Neuropsychological Assessment Battery (NAB),Wechsler Adult Intelligence Scale, Third Edition (WAIS-III),Wechsler Adult Intelligence Scale, Fourth Edition (WAIS-IV),Wechsler Abbreviated Scale of Intelligence (WASI),Test of Variables of Attention (TOVA),Auditory Consonant Trigrams (ACT),Paced Auditory Serial Addition Test (PASAT),Ruff 2 &amp; 7 Selective Attention Test,Symbol Digit Modalities Test (SDMT),Multilingual Aphasia Examination, Second Edition (MAE-II),  Token Test,  Sentence Repetition,  Visual Naming,  Controlled Oral Word Association,  Spelling Test,  Aural Comprehension,  Reading Comprehension,Boston Naming Test, Second Edition (BNT-2),Animal Naming Test</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(medical_doc)\n",
    "visualize_ent(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 465,
   "metadata": {
    "vscode": {
     "languageId": "bat"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[REASON FOR REFERRAL:,\n",
       " RELEVANT BACKGROUND INFORMATION:,\n",
       " HISTORY OF PRESENTING PROBLEM:,\n",
       " OTHER MEDICAL HISTORY:,\n",
       " CURRENT MEDICATIONS:,\n",
       " SUBSTANCE USE:,\n",
       " FAMILY MEDICAL HISTORY:,\n",
       " SOCIAL HISTORY:,\n",
       " PSYCHIATRIC HISTORY:,\n",
       " TASKS ADMINISTERED:]"
      ]
     },
     "execution_count": 465,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc._.section_titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 466,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CATEGORY.............. reason_for_referral\n",
      "TITLE................. REASON FOR REFERRAL:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Ms. A is a 60-year-old African-American female with 12 years of education who was referred for neuropsychological evaluation by Dr. X after she demonstrated mild cognitive deficits on a neuropsychological screening evaluation during a followup appointment with him for stroke in July.  A comprehensive evaluation was requested to assess current cognitive functioning and assist with diagnostic decisions and treatment planning.,\n",
      "----------------------\n",
      "CATEGORY.............. relevant_background_information\n",
      "TITLE................. RELEVANT BACKGROUND INFORMATION:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Historical information was obtained from a review of available medical records and clinical interview with Ms. A.  A summary of pertinent information is presented below.  Please refer to the patient's medical chart for a more complete history.,\n",
      "----------------------\n",
      "CATEGORY.............. history_of_presenting_problem\n",
      "TITLE................. HISTORY OF PRESENTING PROBLEM:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Ms. A presented to the ABC Hospital emergency department on 07/26/2009 reporting that after she had woken up that morning she noticed numbness and weakness in her left hand, slurred speech, and left facial droop.  Neurological evaluation with Dr. X confirmed left hemiparesis.  Brain CT showed no evidence of intracranial hemorrhage or mass effect and that she received TPA and had moderate improvement in left-sided weakness.  These symptoms were thought to be due to a right middle cerebral artery stroke.  She was transferred to the ICU for monitoring.  Ultrasound of the carotids showed 20% to 30% stenosis of the right ICA and 0% to 19% stenosis of the left ICA.  On 07/29/2009, she was admitted for acute inpatient rehabilitation for the treatment of residual functional deficits of her acute ischemic right MCA/CVA.  At discharge on 08/06/2009, she was mainly on supervision for all ADLs and walking with a rolling walker, but tolerating increased ambulation with a cane.  She was discharged home with recommendations for outpatient physical therapy.  She returned to the Sinai ER on 08/2009/2009 due to reported left arm pain, numbness, and weakness, which lasted 10 to 15 minutes and she reported that it felt \"just like the stroke.\"  Brain CT on 08/2009/2009 was read as showing \"mild chronic microvascular ischemic change of deep white matter,\" but no acute or significant interval change compared to her previous scan.  Neurological examination with Dr. Y was within normal limits, but she was admitted for a more extensive workup.  Due to left arm pain an ultrasound was completed on her left upper extremity, but it did not show deep vein thrombosis.,Followup CT on 08/10/2009 showed no significant interval change.  MRI could not be completed due to the patient's weight.  She was discharged on 08/11/2009 in stable condition after it was determined that this event was not neurological in origin; however, note that Ms. A referred to this as a second stroke.,Ms. A presented for a followup outpatient neurological evaluation with Dr. X on 09/22/2009, at which time a brief neuropsychological screening was also conducted.  She demonstrated significant impairments in confrontation naming, abstract verbal reasoning, and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full-time work duty.  During the current interview, Ms. A reported that she noticed mild memory problems including some difficultly remembering conversations, events, and at times forgetting to take her medications.  She also reported mild difficulty finding words in conversation, solving novel problems and tasks (e.g. difficulty learning to use her camcorder), but overall denied significant cognitive deficits in attention, concentration, language or other areas of cognitive functioning.  When asked about her return to work, she said that she was still on light duty due to limited physical activity because of residual left leg weakness.  She reported that no one had indicated to her that she appeared less capable of performing her job duties, but said that she was also receiving fewer files to process and enter data into the computer at the Social Security Agency that she works at.  Note also that she had some difficulty explaining exactly what her job involved.  She also reported having problems falling asleep at work and that she is working full-time although on light duty.,\n",
      "----------------------\n",
      "CATEGORY.............. other_medical_history\n",
      "TITLE................. OTHER MEDICAL HISTORY:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "As mentioned, Ms. A continues to have some residual left leg weakness and continues to use a rolling walker for ambulation, but she reported that her motor functioning had improved significantly.  She was diagnosed with sleep apnea approximately two years ago and was recently counseled by Dr. X on the need to use her CPAP because she indicated she never used it at night.  She reported that since her appointment with Dr. X, she has been using it \"every other night.\"  When asked about daytime fatigue, Ms. A initially denied that she was having any difficulties, but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen.  She reported at times \"snoring\" and forgetting where she is at and said that a supervisor offered to give her coffee at one point.  She receives approximately two to five hours of sleep per night.  Other current untreated risk factors include obesity and hypercholesterolemia.  Her medical history is also significant for hypertension, asthma, abdominal adenocarcinoma status post hysterectomy with bilateral salpingo-oophorectomy, colonic benign polyps status post resection, benign lesions of the breast status post lumpectomy, and deep vein thrombosis in the left lower extremity status post six months of anticoagulation (which she had discontinued just prior to her stroke).,\n",
      "----------------------\n",
      "CATEGORY.............. current_medications\n",
      "TITLE................. CURRENT MEDICATIONS:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Aspirin 81 mg daily, Colace 100 mg b.i.d., Lipitor 80 mg daily, and albuterol MDI p.r.n.,\n",
      "----------------------\n",
      "CATEGORY.............. substance_use\n",
      "TITLE................. SUBSTANCE USE:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Ms. A denied drinking alcohol or using illicit drugs.  She used to smoke a pack of cigarettes per day, but quit five to six years ago.,\n",
      "----------------------\n",
      "CATEGORY.............. family_medical_history\n",
      "TITLE................. FAMILY MEDICAL HISTORY:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Ms. A had difficulty providing information on familial medical history.  She reported that her mother died three to four years ago from lung cancer.  Her father has gout and blood clots.  Siblings have reportedly been treated for asthma and GI tumors.  She was unsure of familial history of other conditions such as hypertension, high cholesterol, stroke, etc.,\n",
      "----------------------\n",
      "CATEGORY.............. social_history\n",
      "TITLE................. SOCIAL HISTORY:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Ms. A completed high school degree.  She reported that she primarily obtained B's and C's in school.  She received some tutoring for algebra in middle school, but denied ever having been held back a grade failing any classes or having any problems with attention or hyperactivity.,She currently works for the Social Security Administration in data processing.  As mentioned, she has returned to full-time work, but continues to perform only light duties due to her physical condition.  She is now living on her own.  She has never driven.  She reported that she continues to perform ADLs independently such as cooking and cleaning.  She lost her husband in 2005 and has three adult daughters.  She previously reported some concerns that her children wanted her to move into assisted living, but she did not discuss that during this current evaluation.  She also reported number of other family members who had recently passed away.  She has returned to activities she enjoys such as quire, knitting, and cooking and plans to go on a cruise to the Bahamas at the end of October.,\n",
      "----------------------\n",
      "CATEGORY.............. psychiatric_history\n",
      "TITLE................. PSYCHIATRIC HISTORY:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Ms. A did not report a history of psychological or psychiatric treatment.  She reported that her current mood was good, but did describe some anxiety and nervousness about various issues such as her return to work, her upcoming trip, and other events.  She reported that this only \"comes and goes.\",\n",
      "----------------------\n",
      "CATEGORY.............. tasks_administered\n",
      "TITLE................. TASKS ADMINISTERED:\n",
      "PARENT................ None\n",
      "SECTION TEXT..........\n",
      "Clinical Interview,Adult History Questionnaire,Wechsler Test of Adult Reading (WTAR),Mini Mental Status Exam (MMSE),Cognistat Neurobehavioral Cognitive Status Examination,Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Form XX),Mattis Dementia Rating Scale, 2nd Edition (DRS-2),Neuropsychological Assessment Battery (NAB),Wechsler Adult Intelligence Scale, Third Edition (WAIS-III),Wechsler Adult Intelligence Scale, Fourth Edition (WAIS-IV),Wechsler Abbreviated Scale of Intelligence (WASI),Test of Variables of Attention (TOVA),Auditory Consonant Trigrams (ACT),Paced Auditory Serial Addition Test (PASAT),Ruff 2 & 7 Selective Attention Test,Symbol Digit Modalities Test (SDMT),Multilingual Aphasia Examination, Second Edition (MAE-II),  Token Test,  Sentence Repetition,  Visual Naming,  Controlled Oral Word Association,  Spelling Test,  Aural Comprehension,  Reading Comprehension,Boston Naming Test, Second Edition (BNT-2),Animal Naming Test\n",
      "----------------------\n"
     ]
    }
   ],
   "source": [
    "#source https://github.com/medspacy/medspacy/blob/master/notebooks/section_detection/3-Subsections.ipynb\n",
    "#additional post-processing added for this particular scenario (line 7)\n",
    "\n",
    "for section in doc._.sections:\n",
    "    title_span = doc[section.title_span[0] : section.title_span[1]]\n",
    "    body_span = doc[section.body_span[0] : section.body_span[1]]\n",
    "    body_span = re.sub(r\"^([, ]*\\b)\", \"\", body_span.text) # post process body_span. this could probably be moved to pre-processing.\n",
    "    \n",
    "    print(\"CATEGORY.............. {0}\".format(section.category))\n",
    "    print(\"TITLE................. {0}\".format(title_span))\n",
    "    if section.parent:\n",
    "        print(\"PARENT................ {0}\".format(section.parent.category))\n",
    "    else:\n",
    "        print(\"PARENT................ {0}\".format(section.parent))\n",
    "    print(\"SECTION TEXT..........\\n{0}\".format(body_span))\n",
    "    print(\"----------------------\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}