notebooks/nlp/spaCy/howto-custom_tokenizer_huggingface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Custom Tokenizer (not spaCy tokenizer) with spaCy\n",
    "\n",
    "This example demonstrates how to use a tokenizer from HF. Pretrained models can be used as well. The last example also includes a method to finding spaces based on the tokenizer output, followed by using this in a more complex pipeline with a sectionizer.\n",
    "\n",
    "Useful links:\n",
    "- https://huggingface.co/docs/tokenizers/quicktour\n",
    "- https://spacy.io/usage/linguistic-features#how-tokenizer-works\n",
    "\n",
    "```\n",
    "# Install tokenizer library\n",
    "pip install tokenizers\n",
    "```\n",
    "```\n",
    "# Get Data Set\n",
    "import urllib.request\n",
    "urllib.request.urlretrieve(\"https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip\", \"./data/wikitext-103-raw-v1/tokenizer-wiki.json\")\n",
    "```\n",
    "\n",
    "```\n",
    "# Unzip Data Set\n",
    "from zipfile import ZipFile\n",
    "with ZipFile('./data/wikitext-103-raw-v1/wikitext-103-raw-v1.zip', 'r') as f:\n",
    "    f.extractall('/wikitext-103-raw-v1/')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define Tokenizer\n",
    "from tokenizers import Tokenizer\n",
    "from tokenizers.models import BPE\n",
    "from tokenizers.pre_tokenizers import Whitespace\n",
    "tokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\n",
    "tokenizer.pre_tokenizer = Whitespace()\n",
    "\n",
    "# Define Trainer\n",
    "from tokenizers.trainers import BpeTrainer\n",
    "trainer = BpeTrainer(special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train Tokenizer\n",
    "files = [f\"./data/wikitext-103-raw-v1/wikitext-103-raw/wiki.{split}.raw\" for split in [\"test\", \"train\", \"valid\"]]\n",
    "tokenizer.train(files, trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save & Load Tokenizer Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.save(\"./data/wikitext-103-raw-v1/wikitext-103-raw/tokenizer-wiki.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tokenizers.Tokenizer at 0x1f581910ed0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.from_file(\"./data/wikitext-103-raw-v1/wikitext-103-raw/tokenizer-wiki.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', ',', 'y', \"'\", 'all', '!', 'How', 'are', 'you', '[UNK]', '?']\n"
     ]
    }
   ],
   "source": [
    "output = tokenizer.encode(\"Hello, y'all! How are you 😁 ?\")\n",
    "print(output.tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Custom Tokenizer With Spacy\n",
    "Tokenize the medical document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "medical_doc = \"\"\"REASON FOR REFERRAL: , Ms. A is a 60-year-old African-American female with 12 years of education who was referred for neuropsychological evaluation by Dr. X after she demonstrated mild cognitive deficits on a neuropsychological screening evaluation during a followup appointment with him for stroke in July.  A comprehensive evaluation was requested to assess current cognitive functioning and assist with diagnostic decisions and treatment planning.,RELEVANT BACKGROUND INFORMATION:,  Historical information was obtained from a review of available medical records and clinical interview with Ms. A.  A summary of pertinent information is presented below.  Please refer to the patient's medical chart for a more complete history.,HISTORY OF PRESENTING PROBLEM:,  Ms. A presented to the ABC Hospital emergency department on 07/26/2009 reporting that after she had woken up that morning she noticed numbness and weakness in her left hand, slurred speech, and left facial droop.  Neurological evaluation with Dr. X confirmed left hemiparesis.  Brain CT showed no evidence of intracranial hemorrhage or mass effect and that she received TPA and had moderate improvement in left-sided weakness.  These symptoms were thought to be due to a right middle cerebral artery stroke.  She was transferred to the ICU for monitoring.  Ultrasound of the carotids showed 20% to 30% stenosis of the right ICA and 0% to 19% stenosis of the left ICA.  On 07/29/2009, she was admitted for acute inpatient rehabilitation for the treatment of residual functional deficits of her acute ischemic right MCA/CVA.  At discharge on 08/06/2009, she was mainly on supervision for all ADLs and walking with a rolling walker, but tolerating increased ambulation with a cane.  She was discharged home with recommendations for outpatient physical therapy.  She returned to the Sinai ER on 08/2009/2009 due to reported left arm pain, numbness, and weakness, which lasted 10 to 15 minutes and she reported that it felt \"just like the stroke.\"  Brain CT on 08/2009/2009 was read as showing \"mild chronic microvascular ischemic change of deep white matter,\" but no acute or significant interval change compared to her previous scan.  Neurological examination with Dr. Y was within normal limits, but she was admitted for a more extensive workup.  Due to left arm pain an ultrasound was completed on her left upper extremity, but it did not show deep vein thrombosis.,Followup CT on 08/10/2009 showed no significant interval change.  MRI could not be completed due to the patient's weight.  She was discharged on 08/11/2009 in stable condition after it was determined that this event was not neurological in origin; however, note that Ms. A referred to this as a second stroke.,Ms. A presented for a followup outpatient neurological evaluation with Dr. X on 09/22/2009, at which time a brief neuropsychological screening was also conducted.  She demonstrated significant impairments in confrontation naming, abstract verbal reasoning, and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full-time work duty.  During the current interview, Ms. A reported that she noticed mild memory problems including some difficultly remembering conversations, events, and at times forgetting to take her medications.  She also reported mild difficulty finding words in conversation, solving novel problems and tasks (e.g. difficulty learning to use her camcorder), but overall denied significant cognitive deficits in attention, concentration, language or other areas of cognitive functioning.  When asked about her return to work, she said that she was still on light duty due to limited physical activity because of residual left leg weakness.  She reported that no one had indicated to her that she appeared less capable of performing her job duties, but said that she was also receiving fewer files to process and enter data into the computer at the Social Security Agency that she works at.  Note also that she had some difficulty explaining exactly what her job involved.  She also reported having problems falling asleep at work and that she is working full-time although on light duty.,OTHER MEDICAL HISTORY:  ,As mentioned, Ms. A continues to have some residual left leg weakness and continues to use a rolling walker for ambulation, but she reported that her motor functioning had improved significantly.  She was diagnosed with sleep apnea approximately two years ago and was recently counseled by Dr. X on the need to use her CPAP because she indicated she never used it at night.  She reported that since her appointment with Dr. X, she has been using it \"every other night.\"  When asked about daytime fatigue, Ms. A initially denied that she was having any difficulties, but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen.  She reported at times \"snoring\" and forgetting where she is at and said that a supervisor offered to give her coffee at one point.  She receives approximately two to five hours of sleep per night.  Other current untreated risk factors include obesity and hypercholesterolemia.  Her medical history is also significant for hypertension, asthma, abdominal adenocarcinoma status post hysterectomy with bilateral salpingo-oophorectomy, colonic benign polyps status post resection, benign lesions of the breast status post lumpectomy, and deep vein thrombosis in the left lower extremity status post six months of anticoagulation (which she had discontinued just prior to her stroke).,CURRENT MEDICATIONS: , Aspirin 81 mg daily, Colace 100 mg b.i.d., Lipitor 80 mg daily, and albuterol MDI p.r.n.,SUBSTANCE USE:,  Ms. A denied drinking alcohol or using illicit drugs.  She used to smoke a pack of cigarettes per day, but quit five to six years ago.,FAMILY MEDICAL HISTORY: , Ms. A had difficulty providing information on familial medical history.  She reported that her mother died three to four years ago from lung cancer.  Her father has gout and blood clots.  Siblings have reportedly been treated for asthma and GI tumors.  She was unsure of familial history of other conditions such as hypertension, high cholesterol, stroke, etc.,SOCIAL HISTORY: , Ms. A completed high school degree.  She reported that she primarily obtained B's and C's in school.  She received some tutoring for algebra in middle school, but denied ever having been held back a grade failing any classes or having any problems with attention or hyperactivity.,She currently works for the Social Security Administration in data processing.  As mentioned, she has returned to full-time work, but continues to perform only light duties due to her physical condition.  She is now living on her own.  She has never driven.  She reported that she continues to perform ADLs independently such as cooking and cleaning.  She lost her husband in 2005 and has three adult daughters.  She previously reported some concerns that her children wanted her to move into assisted living, but she did not discuss that during this current evaluation.  She also reported number of other family members who had recently passed away.  She has returned to activities she enjoys such as quire, knitting, and cooking and plans to go on a cruise to the Bahamas at the end of October.,PSYCHIATRIC HISTORY: , Ms. A did not report a history of psychological or psychiatric treatment.  She reported that her current mood was good, but did describe some anxiety and nervousness about various issues such as her return to work, her upcoming trip, and other events.  She reported that this only \"comes and goes.\",TASKS ADMINISTERED:,Clinical Interview,Adult History Questionnaire,Wechsler Test of Adult Reading (WTAR),Mini Mental Status Exam (MMSE),Cognistat Neurobehavioral Cognitive Status Examination,Repeatable Battery for the Assessment of Neuropsychological Status (RBANS; Form XX),Mattis Dementia Rating Scale, 2nd Edition (DRS-2),Neuropsychological Assessment Battery (NAB),Wechsler Adult Intelligence Scale, Third Edition (WAIS-III),Wechsler Adult Intelligence Scale, Fourth Edition (WAIS-IV),Wechsler Abbreviated Scale of Intelligence (WASI),Test of Variables of Attention (TOVA),Auditory Consonant Trigrams (ACT),Paced Auditory Serial Addition Test (PASAT),Ruff 2 & 7 Selective Attention Test,Symbol Digit Modalities Test (SDMT),Multilingual Aphasia Examination, Second Edition (MAE-II),  Token Test,  Sentence Repetition,  Visual Naming,  Controlled Oral Word Association,  Spelling Test,  Aural Comprehension,  Reading Comprehension,Boston Naming Test, Second Edition (BNT-2),Animal Naming Test\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['reason', 'for', 'refer', 'ral', ':', ',', 'ms', '.', 'a', 'is', 'a', '60', '-', 'year', '-', 'old', 'af', 'rican', '-', 'amer', 'ican', 'female', 'with', '12', 'years', 'of', 'education', 'who', 'was', 'referred', 'for', 'ne', 'u', 'rop', 'sy', 'ch', 'ological', 'evaluation', 'by', 'dr', '.', 'x', 'after', 'she', 'demonstrated', 'mild', 'cognitive', 'def', 'ic', 'its', 'on', 'a', 'ne', 'u', 'rop', 'sy', 'ch', 'ological', 'screening', 'evaluation', 'during', 'a', 'follow', 'up', 'appointment', 'with', 'him', 'for', 'stroke', 'in', 'ju', 'ly', '.', 'a', 'comprehensive', 'evaluation', 'was', 'requested', 'to', 'assess', 'current', 'cognitive', 'functioning', 'and', 'assist', 'with', 'diag', 'nostic', 'decisions', 'and', 'treatment', 'planning', '.', ',', 'relevant', 'background', 'information', ':', ',', 'historical', 'information', 'was', 'obtained', 'from', 'a', 'review', 'of', 'available', 'medical', 'records', 'and', 'clinical', 'interview', 'with', 'ms', '.', 'a', '.', 'a', 'summary', 'of', 'pert', 'inent', 'information', 'is', 'presented', 'below', '.', 'please', 'refer', 'to', 'the', 'patient', \"'\", 's', 'medical', 'chart', 'for', 'a', 'more', 'complete', 'history', '.', ',', 'history', 'of', 'presenting', 'problem', ':', ',', 'ms', '.', 'a', 'presented', 'to', 'the', 'ab', 'c', 'hospital', 'emergency', 'department', 'on', '07', '/', '26', '/', '2009', 'reporting', 'that', 'after', 'she', 'had', 'w', 'ok', 'en', 'up', 'that', 'morning', 'she', 'noticed', 'num', 'b', 'ness', 'and', 'weakness', 'in', 'her', 'left', 'hand', ',', 'sl', 'ur', 'red', 'speech', ',', 'and', 'left', 'facial', 'dro', 'op', '.', 'neuro', 'logical', 'evaluation', 'with', 'dr', '.', 'x', 'confirmed', 'left', 'hem', 'ip', 'ares', 'is', '.', 'brain', 'ct', 'showed', 'no', 'evidence', 'of', 'intr', 'ac', 'ran', 'ial', 'he', 'mor', 'rh', 'age', 'or', 'mass', 'effect', 'and', 'that', 'she', 'received', 't', 'pa', 'and', 'had', 'moderate', 'improvement', 'in', 'left', '-', 'sided', 'weakness', '.', 'these', 'symptoms', 'were', 'thought', 'to', 'be', 'due', 'to', 'a', 'right', 'middle', 'cerebral', 'ar', 'tery', 'stroke', '.', 'she', 'was', 'transferred', 'to', 'the', 'ic', 'u', 'for', 'monitoring', '.', 'ultr', 'as', 'ound', 'of', 'the', 'car', 'ot', 'ids', 'showed', '20', '%', 'to', '30', '%', 'sten', 'osis', 'of', 'the', 'right', 'ica', 'and', '0', '%', 'to', '19', '%', 'sten', 'osis', 'of', 'the', 'left', 'ica', '.', 'on', '07', '/', '29', '/', '2009', ',', 'she', 'was', 'admitted', 'for', 'acute', 'in', 'patient', 'rehabilitation', 'for', 'the', 'treatment', 'of', 'resid', 'ual', 'functional', 'def', 'ic', 'its', 'of', 'her', 'acute', 'is', 'che', 'mic', 'right', 'm', 'ca', '/', 'c', 'va', '.', 'at', 'discharge', 'on', '08', '/', '06', '/', '2009', ',', 'she', 'was', 'mainly', 'on', 'supervision', 'for', 'all', 'ad', 'ls', 'and', 'walking', 'with', 'a', 'rolling', 'walk', 'er', ',', 'but', 'toler', 'ating', 'increased', 'amb', 'ulation', 'with', 'a', 'cane', '.', 'she', 'was', 'discharged', 'home', 'with', 'recommendations', 'for', 'out', 'patient', 'physical', 'therapy', '.', 'she', 'returned', 'to', 'the', 's', 'ina', 'i', 'er', 'on', '08', '/', '2009', '/', '2009', 'due', 'to', 'reported', 'left', 'arm', 'pain', ',', 'num', 'b', 'ness', ',', 'and', 'weakness', ',', 'which', 'lasted', '10', 'to', '15', 'minutes', 'and', 'she', 'reported', 'that', 'it', 'felt', '\"', 'just', 'like', 'the', 'stroke', '.', '\"', 'brain', 'ct', 'on', '08', '/', '2009', '/', '2009', 'was', 'read', 'as', 'showing', '\"', 'mild', 'chronic', 'micro', 'v', 'ascular', 'is', 'che', 'mic', 'change', 'of', 'deep', 'white', 'matter', ',', '\"', 'but', 'no', 'acute', 'or', 'significant', 'interval', 'change', 'compared', 'to', 'her', 'previous', 'scan', '.', 'neuro', 'logical', 'examination', 'with', 'dr', '.', 'y', 'was', 'within', 'normal', 'limits', ',', 'but', 'she', 'was', 'admitted', 'for', 'a', 'more', 'extensive', 'work', 'up', '.', 'due', 'to', 'left', 'arm', 'pain', 'an', 'ultr', 'as', 'ound', 'was', 'completed', 'on', 'her', 'left', 'upper', 'extre', 'm', 'ity', ',', 'but', 'it', 'did', 'not', 'show', 'deep', 'vein', 'throm', 'b', 'osis', '.', ',', 'follow', 'up', 'ct', 'on', '08', '/', '10', '/', '2009', 'showed', 'no', 'significant', 'interval', 'change', '.', 'm', 'ri', 'could', 'not', 'be', 'completed', 'due', 'to', 'the', 'patient', \"'\", 's', 'weight', '.', 'she', 'was', 'discharged', 'on', '08', '/', '11', '/', '2009', 'in', 'stable', 'condition', 'after', 'it', 'was', 'determined', 'that', 'this', 'event', 'was', 'not', 'neuro', 'logical', 'in', 'origin', ';', 'however', ',', 'note', 'that', 'ms', '.', 'a', 'referred', 'to', 'this', 'as', 'a', 'second', 'stroke', '.', ',', 'ms', '.', 'a', 'presented', 'for', 'a', 'follow', 'up', 'out', 'patient', 'neuro', 'logical', 'evaluation', 'with', 'dr', '.', 'x', 'on', '09', '/', '22', '/', '2009', ',', 'at', 'which', 'time', 'a', 'brief', 'ne', 'u', 'rop', 'sy', 'ch', 'ological', 'screening', 'was', 'also', 'conducted', '.', 'she', 'demonstrated', 'significant', 'impair', 'ments', 'in', 'confrontation', 'naming', ',', 'abstract', 'verbal', 'reasoning', ',', 'and', 'visual', 'and', 'verbal', 'memory', 'and', 'thus', 'a', 'more', 'comprehensive', 'evaluation', 'was', 'suggested', 'due', 'to', 'her', 'intent', 'to', 'return', 'to', 'her', 'full', '-', 'time', 'work', 'duty', '.', 'during', 'the', 'current', 'interview', ',', 'ms', '.', 'a', 'reported', 'that', 'she', 'noticed', 'mild', 'memory', 'problems', 'including', 'some', 'difficult', 'ly', 'remember', 'ing', 'conversations', ',', 'events', ',', 'and', 'at', 'times', 'for', 'getting', 'to', 'take', 'her', 'medications', '.', 'she', 'also', 'reported', 'mild', 'difficulty', 'finding', 'words', 'in', 'conversation', ',', 'solving', 'novel', 'problems', 'and', 'tasks', '(', 'e', '.', 'g', '.', 'difficulty', 'learning', 'to', 'use', 'her', 'cam', 'corder', ')', ',', 'but', 'overall', 'denied', 'significant', 'cognitive', 'def', 'ic', 'its', 'in', 'attention', ',', 'concentration', ',', 'language', 'or', 'other', 'areas', 'of', 'cognitive', 'functioning', '.', 'when', 'asked', 'about', 'her', 'return', 'to', 'work', ',', 'she', 'said', 'that', 'she', 'was', 'still', 'on', 'light', 'duty', 'due', 'to', 'limited', 'physical', 'activity', 'because', 'of', 'resid', 'ual', 'left', 'leg', 'weakness', '.', 'she', 'reported', 'that', 'no', 'one', 'had', 'indicated', 'to', 'her', 'that', 'she', 'appeared', 'less', 'capable', 'of', 'performing', 'her', 'job', 'duties', ',', 'but', 'said', 'that', 'she', 'was', 'also', 'receiving', 'fewer', 'files', 'to', 'process', 'and', 'enter', 'data', 'into', 'the', 'computer', 'at', 'the', 'social', 'security', 'agency', 'that', 'she', 'works', 'at', '.', 'note', 'also', 'that', 'she', 'had', 'some', 'difficulty', 'explaining', 'exactly', 'what', 'her', 'job', 'involved', '.', 'she', 'also', 'reported', 'having', 'problems', 'falling', 'asleep', 'at', 'work', 'and', 'that', 'she', 'is', 'working', 'full', '-', 'time', 'although', 'on', 'light', 'duty', '.', ',', 'other', 'medical', 'history', ':', ',', 'as', 'mentioned', ',', 'ms', '.', 'a', 'continues', 'to', 'have', 'some', 'resid', 'ual', 'left', 'leg', 'weakness', 'and', 'continues', 'to', 'use', 'a', 'rolling', 'walk', 'er', 'for', 'amb', 'ulation', ',', 'but', 'she', 'reported', 'that', 'her', 'motor', 'functioning', 'had', 'improved', 'significantly', '.', 'she', 'was', 'diagnosed', 'with', 'sleep', 'ap', 'ne', 'a', 'approximately', 'two', 'years', 'ago', 'and', 'was', 'recently', 'coun', 'se', 'led', 'by', 'dr', '.', 'x', 'on', 'the', 'need', 'to', 'use', 'her', 'c', 'pap', 'because', 'she', 'indicated', 'she', 'never', 'used', 'it', 'at', 'night', '.', 'she', 'reported', 'that', 'since', 'her', 'appointment', 'with', 'dr', '.', 'x', ',', 'she', 'has', 'been', 'using', 'it', '\"', 'every', 'other', 'night', '.', '\"', 'when', 'asked', 'about', 'daytime', 'fatigue', ',', 'ms', '.', 'a', 'initially', 'denied', 'that', 'she', 'was', 'having', 'any', 'difficulties', ',', 'but', 'repeatedly', 'indicated', 'that', 'she', 'was', 'falling', 'asleep', 'at', 'work', 'and', 'thought', 'that', 'it', 'was', 'due', 'to', 'looking', 'at', 'a', 'computer', 'screen', '.', 'she', 'reported', 'at', 'times', '\"', 's', 'nor', 'ing', '\"', 'and', 'for', 'getting', 'where', 'she', 'is', 'at', 'and', 'said', 'that', 'a', 'supervisor', 'offered', 'to', 'give', 'her', 'coffee', 'at', 'one', 'point', '.', 'she', 'receives', 'approximately', 'two', 'to', 'five', 'hours', 'of', 'sleep', 'per', 'night', '.', 'other', 'current', 'unt', 're', 'ated', 'risk', 'factors', 'include', 'obes', 'ity', 'and', 'hyper', 'cho', 'le', 'ster', 'ole', 'm', 'ia', '.', 'her', 'medical', 'history', 'is', 'also', 'significant', 'for', 'hyper', 'tension', ',', 'as', 'th', 'ma', ',', 'abdominal', 'aden', 'ocar', 'cin', 'oma', 'status', 'post', 'hy', 'stere', 'cto', 'my', 'with', 'bil', 'ateral', 'sal', 'ping', 'o', '-', 'o', 'oph', 'ore', 'cto', 'my', ',', 'colon', 'ic', 'ben', 'ign', 'poly', 'ps', 'status', 'post', 're', 'section', ',', 'ben', 'ign', 'les', 'ions', 'of', 'the', 'breast', 'status', 'post', 'lum', 'pe', 'cto', 'my', ',', 'and', 'deep', 'vein', 'throm', 'b', 'osis', 'in', 'the', 'left', 'lower', 'extre', 'm', 'ity', 'status', 'post', 'six', 'months', 'of', 'antic', 'o', 'ag', 'ulation', '(', 'which', 'she', 'had', 'discontinued', 'just', 'prior', 'to', 'her', 'stroke', ')', '.', ',', 'current', 'medications', ':', ',', 'aspir', 'in', '81', 'mg', 'daily', ',', 'col', 'ace', '100', 'mg', 'b', '.', 'i', '.', 'd', '.', ',', 'lip', 'itor', '80', 'mg', 'daily', ',', 'and', 'al', 'bu', 'ter', 'ol', 'm', 'di', 'p', '.', 'r', '.', 'n', '.', ',', 'substance', 'use', ':', ',', 'ms', '.', 'a', 'denied', 'drinking', 'alcohol', 'or', 'using', 'ill', 'icit', 'drugs', '.', 'she', 'used', 'to', 'smoke', 'a', 'pack', 'of', 'cigaret', 'tes', 'per', 'day', ',', 'but', 'quit', 'five', 'to', 'six', 'years', 'ago', '.', ',', 'family', 'medical', 'history', ':', ',', 'ms', '.', 'a', 'had', 'difficulty', 'providing', 'information', 'on', 'fam', 'il', 'ial', 'medical', 'history', '.', 'she', 'reported', 'that', 'her', 'mother', 'died', 'three', 'to', 'four', 'years', 'ago', 'from', 'lung', 'cancer', '.', 'her', 'father', 'has', 'g', 'out', 'and', 'blood', 'cl', 'ots', '.', 'siblings', 'have', 'reportedly', 'been', 'treated', 'for', 'as', 'th', 'ma', 'and', 'gi', 'tum', 'ors', '.', 'she', 'was', 'unsure', 'of', 'fam', 'il', 'ial', 'history', 'of', 'other', 'conditions', 'such', 'as', 'hyper', 'tension', ',', 'high', 'cho', 'le', 'ster', 'ol', ',', 'stroke', ',', 'etc', '.', ',', 'social', 'history', ':', ',', 'ms', '.', 'a', 'completed', 'high', 'school', 'degree', '.', 'she', 'reported', 'that', 'she', 'primarily', 'obtained', 'b', \"'\", 's', 'and', 'c', \"'\", 's', 'in', 'school', '.', 'she', 'received', 'some', 'tut', 'oring', 'for', 'al', 'gebra', 'in', 'middle', 'school', ',', 'but', 'denied', 'ever', 'having', 'been', 'held', 'back', 'a', 'grade', 'failing', 'any', 'classes', 'or', 'having', 'any', 'problems', 'with', 'attention', 'or', 'hyper', 'activity', '.', ',', 'she', 'currently', 'works', 'for', 'the', 'social', 'security', 'administration', 'in', 'data', 'processing', '.', 'as', 'mentioned', ',', 'she', 'has', 'returned', 'to', 'full', '-', 'time', 'work', ',', 'but', 'continues', 'to', 'perform', 'only', 'light', 'duties', 'due', 'to', 'her', 'physical', 'condition', '.', 'she', 'is', 'now', 'living', 'on', 'her', 'own', '.', 'she', 'has', 'never', 'driven', '.', 'she', 'reported', 'that', 'she', 'continues', 'to', 'perform', 'ad', 'ls', 'independently', 'such', 'as', 'cooking', 'and', 'cleaning', '.', 'she', 'lost', 'her', 'husband', 'in', '2005', 'and', 'has', 'three', 'adult', 'daughters', '.', 'she', 'previously', 'reported', 'some', 'concerns', 'that', 'her', 'children', 'wanted', 'her', 'to', 'move', 'into', 'assisted', 'living', ',', 'but', 'she', 'did', 'not', 'discuss', 'that', 'during', 'this', 'current', 'evaluation', '.', 'she', 'also', 'reported', 'number', 'of', 'other', 'family', 'members', 'who', 'had', 'recently', 'passed', 'away', '.', 'she', 'has', 'returned', 'to', 'activities', 'she', 'enjoys', 'such', 'as', 'qu', 'ire', ',', 'kn', 'itting', ',', 'and', 'cooking', 'and', 'plans', 'to', 'go', 'on', 'a', 'cruise', 'to', 'the', 'b', 'ah', 'amas', 'at', 'the', 'end', 'of', 'oc', 'to', 'ber', '.', ',', 'psychiatric', 'history', ':', ',', 'ms', '.', 'a', 'did', 'not', 'report', 'a', 'history', 'of', 'psychological', 'or', 'psychiatric', 'treatment', '.', 'she', 'reported', 'that', 'her', 'current', 'mood', 'was', 'good', ',', 'but', 'did', 'describe', 'some', 'anxiety', 'and', 'nervous', 'ness', 'about', 'various', 'issues', 'such', 'as', 'her', 'return', 'to', 'work', ',', 'her', 'upcoming', 'trip', ',', 'and', 'other', 'events', '.', 'she', 'reported', 'that', 'this', 'only', '\"', 'comes', 'and', 'goes', '.', '\"', ',', 'tasks', 'administered', ':', ',', 'clinical', 'interview', ',', 'adult', 'history', 'question', 'na', 'ire', ',', 'we', 'chs', 'ler', 'test', 'of', 'adult', 'reading', '(', 'w', 'tar', ')', ',', 'mini', 'mental', 'status', 'exam', '(', 'mm', 'se', ')', ',', 'cogn', 'ist', 'at', 'neuro', 'behavior', 'al', 'cognitive', 'status', 'examination', ',', 'repeat', 'able', 'battery', 'for', 'the', 'assessment', 'of', 'ne', 'u', 'rop', 'sy', 'ch', 'ological', 'status', '(', 'r', 'b', 'ans', ';', 'form', 'xx', ')', ',', 'm', 'att', 'is', 'de', 'ment', 'ia', 'rating', 'scale', ',', '2nd', 'edition', '(', 'dr', 's', '-', '2', ')', ',', 'ne', 'u', 'rop', 'sy', 'ch', 'ological', 'assessment', 'battery', '(', 'n', 'ab', ')', ',', 'we', 'chs', 'ler', 'adult', 'intelligence', 'scale', ',', 'third', 'edition', '(', 'wa', 'is', '-', 'ii', 'i', ')', ',', 'we', 'chs', 'ler', 'adult', 'intelligence', 'scale', ',', 'fourth', 'edition', '(', 'wa', 'is', '-', 'iv', ')', ',', 'we', 'chs', 'ler', 'abbreviated', 'scale', 'of', 'intelligence', '(', 'was', 'i', ')', ',', 'test', 'of', 'variables', 'of', 'attention', '(', 'to', 'va', ')', ',', 'aud', 'itory', 'conson', 'ant', 'tr', 'igr', 'ams', '(', 'act', ')', ',', 'paced', 'aud', 'itory', 'serial', 'addition', 'test', '(', 'p', 'as', 'at', ')', ',', 'ruff', '2', '&', '7', 'selective', 'attention', 'test', ',', 'symbol', 'digit', 'mod', 'al', 'ities', 'test', '(', 's', 'd', 'm', 't', ')', ',', 'mult', 'iling', 'ual', 'ap', 'has', 'ia', 'examination', ',', 'second', 'edition', '(', 'ma', 'e', '-', 'ii', ')', ',', 'to', 'ken', 'test', ',', 'sentence', 'repetition', ',', 'visual', 'naming', ',', 'controlled', 'oral', 'word', 'association', ',', 'spelling', 'test', ',', 'a', 'ural', 'compre', 'hen', 'sion', ',', 'reading', 'compre', 'hen', 'sion', ',', 'bo', 'ston', 'naming', 'test', ',', 'second', 'edition', '(', 'b', 'nt', '-', '2', ')', ',', 'animal', 'naming', 'test']\n"
     ]
    }
   ],
   "source": [
    "output = tokenizer.encode(medical_doc.lower())\n",
    "print(output.tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process without defining spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'reason for refer ral : , ms . a is a 60 - year - old af rican - amer ican female with 12 years of education who was referred for ne u rop sy ch ological evaluation by dr . x after she demonstrated mild cognitive def ic its on a ne u rop sy ch ological screening evaluation during a follow up appointment with him for stroke in ju ly . a comprehensive evaluation was requested to assess current cognitive functioning and assist with diag nostic decisions and treatment planning . , relevant background information : , historical information was obtained from a review of available medical records and clinical interview with ms . a . a summary of pert inent information is presented below . please refer to the patient \\' s medical chart for a more complete history . , history of presenting problem : , ms . a presented to the ab c hospital emergency department on 07 / 26 / 2009 reporting that after she had w ok en up that morning she noticed num b ness and weakness in her left hand , sl ur red speech , and left facial dro op . neuro logical evaluation with dr . x confirmed left hem ip ares is . brain ct showed no evidence of intr ac ran ial he mor rh age or mass effect and that she received t pa and had moderate improvement in left - sided weakness . these symptoms were thought to be due to a right middle cerebral ar tery stroke . she was transferred to the ic u for monitoring . ultr as ound of the car ot ids showed 20 % to 30 % sten osis of the right ica and 0 % to 19 % sten osis of the left ica . on 07 / 29 / 2009 , she was admitted for acute in patient rehabilitation for the treatment of resid ual functional def ic its of her acute is che mic right m ca / c va . at discharge on 08 / 06 / 2009 , she was mainly on supervision for all ad ls and walking with a rolling walk er , but toler ating increased amb ulation with a cane . she was discharged home with recommendations for out patient physical therapy . she returned to the s ina i er on 08 / 2009 / 2009 due to reported left arm pain , num b ness , and weakness , which lasted 10 to 15 minutes and she reported that it felt \" just like the stroke . \" brain ct on 08 / 2009 / 2009 was read as showing \" mild chronic micro v ascular is che mic change of deep white matter , \" but no acute or significant interval change compared to her previous scan . neuro logical examination with dr . y was within normal limits , but she was admitted for a more extensive work up . due to left arm pain an ultr as ound was completed on her left upper extre m ity , but it did not show deep vein throm b osis . , follow up ct on 08 / 10 / 2009 showed no significant interval change . m ri could not be completed due to the patient \\' s weight . she was discharged on 08 / 11 / 2009 in stable condition after it was determined that this event was not neuro logical in origin ; however , note that ms . a referred to this as a second stroke . , ms . a presented for a follow up out patient neuro logical evaluation with dr . x on 09 / 22 / 2009 , at which time a brief ne u rop sy ch ological screening was also conducted . she demonstrated significant impair ments in confrontation naming , abstract verbal reasoning , and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full - time work duty . during the current interview , ms . a reported that she noticed mild memory problems including some difficult ly remember ing conversations , events , and at times for getting to take her medications . she also reported mild difficulty finding words in conversation , solving novel problems and tasks ( e . g . difficulty learning to use her cam corder ) , but overall denied significant cognitive def ic its in attention , concentration , language or other areas of cognitive functioning . when asked about her return to work , she said that she was still on light duty due to limited physical activity because of resid ual left leg weakness . she reported that no one had indicated to her that she appeared less capable of performing her job duties , but said that she was also receiving fewer files to process and enter data into the computer at the social security agency that she works at . note also that she had some difficulty explaining exactly what her job involved . she also reported having problems falling asleep at work and that she is working full - time although on light duty . , other medical history : , as mentioned , ms . a continues to have some resid ual left leg weakness and continues to use a rolling walk er for amb ulation , but she reported that her motor functioning had improved significantly . she was diagnosed with sleep ap ne a approximately two years ago and was recently coun se led by dr . x on the need to use her c pap because she indicated she never used it at night . she reported that since her appointment with dr . x , she has been using it \" every other night . \" when asked about daytime fatigue , ms . a initially denied that she was having any difficulties , but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen . she reported at times \" s nor ing \" and for getting where she is at and said that a supervisor offered to give her coffee at one point . she receives approximately two to five hours of sleep per night . other current unt re ated risk factors include obes ity and hyper cho le ster ole m ia . her medical history is also significant for hyper tension , as th ma , abdominal aden ocar cin oma status post hy stere cto my with bil ateral sal ping o - o oph ore cto my , colon ic ben ign poly ps status post re section , ben ign les ions of the breast status post lum pe cto my , and deep vein throm b osis in the left lower extre m ity status post six months of antic o ag ulation ( which she had discontinued just prior to her stroke ) . , current medications : , aspir in 81 mg daily , col ace 100 mg b . i . d . , lip itor 80 mg daily , and al bu ter ol m di p . r . n . , substance use : , ms . a denied drinking alcohol or using ill icit drugs . she used to smoke a pack of cigaret tes per day , but quit five to six years ago . , family medical history : , ms . a had difficulty providing information on fam il ial medical history . she reported that her mother died three to four years ago from lung cancer . her father has g out and blood cl ots . siblings have reportedly been treated for as th ma and gi tum ors . she was unsure of fam il ial history of other conditions such as hyper tension , high cho le ster ol , stroke , etc . , social history : , ms . a completed high school degree . she reported that she primarily obtained b \\' s and c \\' s in school . she received some tut oring for al gebra in middle school , but denied ever having been held back a grade failing any classes or having any problems with attention or hyper activity . , she currently works for the social security administration in data processing . as mentioned , she has returned to full - time work , but continues to perform only light duties due to her physical condition . she is now living on her own . she has never driven . she reported that she continues to perform ad ls independently such as cooking and cleaning . she lost her husband in 2005 and has three adult daughters . she previously reported some concerns that her children wanted her to move into assisted living , but she did not discuss that during this current evaluation . she also reported number of other family members who had recently passed away . she has returned to activities she enjoys such as qu ire , kn itting , and cooking and plans to go on a cruise to the b ah amas at the end of oc to ber . , psychiatric history : , ms . a did not report a history of psychological or psychiatric treatment . she reported that her current mood was good , but did describe some anxiety and nervous ness about various issues such as her return to work , her upcoming trip , and other events . she reported that this only \" comes and goes . \" , tasks administered : , clinical interview , adult history question na ire , we chs ler test of adult reading ( w tar ) , mini mental status exam ( mm se ) , cogn ist at neuro behavior al cognitive status examination , repeat able battery for the assessment of ne u rop sy ch ological status ( r b ans ; form xx ) , m att is de ment ia rating scale , 2nd edition ( dr s - 2 ) , ne u rop sy ch ological assessment battery ( n ab ) , we chs ler adult intelligence scale , third edition ( wa is - ii i ) , we chs ler adult intelligence scale , fourth edition ( wa is - iv ) , we chs ler abbreviated scale of intelligence ( was i ) , test of variables of attention ( to va ) , aud itory conson ant tr igr ams ( act ) , paced aud itory serial addition test ( p as at ) , ruff 2 & 7 selective attention test , symbol digit mod al ities test ( s d m t ) , mult iling ual ap has ia examination , second edition ( ma e - ii ) , to ken test , sentence repetition , visual naming , controlled oral word association , spelling test , a ural compre hen sion , reading compre hen sion , bo ston naming test , second edition ( b nt - 2 ) , animal naming test '"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy.tokens import Doc\n",
    "\n",
    "nlp = spacy.blank(\"en\")\n",
    "words = output.tokens\n",
    "spaces = None\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "\n",
    "doc.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process when defining spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'reasonfor referral : , ms . a is a 60 - year - old african - american female with 12 years of education who was referred for neuropsychological evaluation by dr . x after she demonstrated mild cognitive deficits on a neuropsychological screening evaluation during a followup appointment with him for stroke in july . a comprehensive evaluation was requested to assess current cognitive functioning and assist with diagnostic decisions and treatment planning ., relevant background information :, historical information was obtained from a review of available medical records and clinical interview with ms . a . a summary of pertinent information is presented below . please refer to the patient \\' s medical chart for a more complete history ., history of presenting problem :, ms . a presented to the abc hospital emergency department on 07 / 26 / 2009 reporting that after she had woken up that morning she noticed numbness and weakness in her left hand , slurred speech , and left facial droop . neurological evaluation with dr . x confirmed left hemiparesis . brain ct showed no evidence of intracranial hemorrhage or mass effect and that she received tpa and had moderate improvement in left - sided weakness . these symptoms were thought to be due to a right middle cerebral artery stroke . she was transferred to the icu for monitoring . ultrasound of the carotids showed 20 % to 30 % stenosis of the right ica and 0 % to 19 % stenosis of the left ica . on 07 / 29 / 2009 , she was admitted for acute inpatient rehabilitation for the treatment of residual functional deficits of her acute ischemic right mca / cva . at discharge on 08 / 06 / 2009 , she was mainly on supervision for all adls and walking with a rolling walker , but tolerating increased ambulation with a cane . she was discharged home with recommendations for outpatient physical therapy . she returned to the sinai er on 08 / 2009 / 2009 due to reported left arm pain , numbness , and weakness , which lasted 10 to 15 minutes and she reported that it felt \" just like the stroke .\" brain ct on 08 / 2009 / 2009 was read as showing \" mild chronic microvascular ischemic change of deep white matter ,\" but no acute or significant interval change compared to her previous scan . neurological examination with dr . y was within normal limits , but she was admitted for a more extensive workup . due to left arm pain an ultrasound was completed on her left upper extremity , but it did not show deep vein thrombosis ., followup ct on 08 / 10 / 2009 showed no significant interval change . mri could not be completed due to the patient \\' s weight . she was discharged on 08 / 11 / 2009 in stable condition after it was determined that this event was not neurological in origin ; however , note that ms . a referred to this as a second stroke ., ms . a presented for a followup outpatient neurological evaluation with dr . x on 09 / 22 / 2009 , at which time a brief neuropsychological screening was also conducted . she demonstrated significant impairments in confrontation naming , abstract verbal reasoning , and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full - time work duty . during the current interview , ms . a reported that she noticed mild memory problems including some difficultly remembering conversations , events , and at times forgetting to take her medications . she also reported mild difficulty finding words in conversation , solving novel problems and tasks ( e . g . difficulty learning to use her camcorder ), but overall denied significant cognitive deficits in attention , concentration , language or other areas of cognitive functioning . when asked about her return to work , she said that she was still on light duty due to limited physical activity because of residual left leg weakness . she reported that no one had indicated to her that she appeared less capable of performing her job duties , but said that she was also receiving fewer files to process and enter data into the computer at the social security agency that she works at . note also that she had some difficulty explaining exactly what her job involved . she also reported having problems falling asleep at work and that she is working full - time although on light duty ., other medical history : , as mentioned , ms . a continues to have some residual left leg weakness and continues to use a rolling walker for ambulation , but she reported that her motor functioning had improved significantly . she was diagnosed with sleep apnea approximately two years ago and was recently counseled by dr . x on the need to use her cpap because she indicated she never used it at night . she reported that since her appointment with dr . x , she has been using it \" every other night .\" when asked about daytime fatigue , ms . a initially denied that she was having any difficulties , but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen . she reported at times \" snoring \" and forgetting where she is at and said that a supervisor offered to give her coffee at one point . she receives approximately two to five hours of sleep per night . other current untreated risk factors include obesity and hypercholesterolemia . her medical history is also significant for hypertension , asthma , abdominal adenocarcinoma status post hysterectomy with bilateral salpingo - oophorectomy , colonic benign polyps status post resection , benign lesions of the breast status post lumpectomy , and deep vein thrombosis in the left lower extremity status post six months of anticoagulation ( which she had discontinued just prior to her stroke )., current medications : , aspirin 81 mg daily , colace 100 mg b . i . d ., lipitor 80 mg daily , and albuterol mdi p . r . n ., substance use :, ms . a denied drinking alcohol or using illicit drugs . she used to smoke a pack of cigarettes per day , but quit five to six years ago ., family medical history : , ms . a had difficulty providing information on familial medical history . she reported that her mother died three to four years ago from lung cancer . her father has gout and blood clots . siblings have reportedly been treated for asthma and gi tumors . she was unsure of familial history of other conditions such as hypertension , high cholesterol , stroke , etc ., social history : , ms . a completed high school degree . she reported that she primarily obtained b \\' s and c \\' s in school . she received some tutoring for algebra in middle school , but denied ever having been held back a grade failing any classes or having any problems with attention or hyperactivity ., she currently works for the social security administration in data processing . as mentioned , she has returned to full - time work , but continues to perform only light duties due to her physical condition . she is now living on her own . she has never driven . she reported that she continues to perform adls independently such as cooking and cleaning . she lost her husband in 2005 and has three adult daughters . she previously reported some concerns that her children wanted her to move into assisted living , but she did not discuss that during this current evaluation . she also reported number of other family members who had recently passed away . she has returned to activities she enjoys such as quire , knitting , and cooking and plans to go on a cruise to the bahamas at the end of october ., psychiatric history : , ms . a did not report a history of psychological or psychiatric treatment . she reported that her current mood was good , but did describe some anxiety and nervousness about various issues such as her return to work , her upcoming trip , and other events . she reported that this only \" comes and goes .\", tasks administered :, clinical interview , adult history questionnaire , wechsler test of adult reading ( wtar ), mini mental status exam ( mmse ), cognistat neurobehavioral cognitive status examination , repeatable battery for the assessment of neuropsychological status ( rbans ; form xx ), mattis dementia rating scale , 2nd edition ( drs - 2 ), neuropsychological assessment battery ( nab ), wechsler adult intelligence scale , third edition ( wais - iii ), wechsler adult intelligence scale , fourth edition ( wais - iv ), wechsler abbreviated scale of intelligence ( wasi ), test of variables of attention ( tova ), auditory consonant trigrams ( act ), paced auditory serial addition test ( pasat ), ruff 2 & 7 selective attention test , symbol digit modalities test ( sdmt ), multilingual aphasia examination , second edition ( mae - ii ), token test , sentence repetition , visual naming , controlled oral word association , spelling test , aural comprehension , reading comprehension , boston naming test , second edition ( bnt - 2 ), animal naming test'"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find Spaces\n",
    "\n",
    "spaces = []\n",
    "\n",
    "for i in enumerate(output.words):\n",
    "    if (i[0]-1 >= 0) and (i[0]+1 < len(output.words)):\n",
    "        if output.words[i[0]] != output.words[i[0]+1]:\n",
    "            spaces.append(True)\n",
    "        else:\n",
    "            spaces.append(False)\n",
    "    else:\n",
    "        spaces.append(False)\n",
    "        \n",
    "# Re-Run Spacy\n",
    "nlp = spacy.blank(\"en\")\n",
    "words = output.tokens\n",
    "spaces = spaces\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "\n",
    "doc.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Example - Add Sectionizer\n",
    "\n",
    "Note that all sections are not present because the tokenizer did not detect every word properly. For instance, \"reason referral\" was tokenized as \"reason\", \"for\", \"refer\", \"ral\", \":\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">reasonfor referral : , ms . a is a 60 - year - old african - american female with 12 years of education who was referred for neuropsychological evaluation by dr . x after she demonstrated mild cognitive deficits on a neuropsychological screening evaluation during a followup appointment with him for stroke in july . a comprehensive evaluation was requested to assess current cognitive functioning and assist with diagnostic decisions and treatment planning ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    relevant background information :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< RELEVANT_BACKGROUND_INFORMATION >></span>\n",
       "</mark>\n",
       ", historical information was obtained from a review of available medical records and clinical interview with ms . a . a summary of pertinent information is presented below . please refer to the patient ' s medical chart for a more complete history ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    history of presenting problem :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< HISTORY_OF_PRESENTING_PROBLEM >></span>\n",
       "</mark>\n",
       ", ms . a presented to the abc hospital emergency department on 07 / 26 / 2009 reporting that after she had woken up that morning she noticed numbness and weakness in her left hand , slurred speech , and left facial droop . neurological evaluation with dr . x confirmed left hemiparesis . brain ct showed no evidence of intracranial hemorrhage or mass effect and that she received tpa and had moderate improvement in left - sided weakness . these symptoms were thought to be due to a right middle cerebral artery stroke . she was transferred to the icu for monitoring . ultrasound of the carotids showed 20 % to 30 % stenosis of the right ica and 0 % to 19 % stenosis of the left ica . on 07 / 29 / 2009 , she was admitted for acute inpatient rehabilitation for the treatment of residual functional deficits of her acute ischemic right mca / cva . at discharge on 08 / 06 / 2009 , she was mainly on supervision for all adls and walking with a rolling walker , but tolerating increased ambulation with a cane . she was discharged home with recommendations for outpatient physical therapy . she returned to the sinai er on 08 / 2009 / 2009 due to reported left arm pain , numbness , and weakness , which lasted 10 to 15 minutes and she reported that it felt &quot; just like the stroke .&quot; brain ct on 08 / 2009 / 2009 was read as showing &quot; mild chronic microvascular ischemic change of deep white matter ,&quot; but no acute or significant interval change compared to her previous scan . neurological examination with dr . y was within normal limits , but she was admitted for a more extensive workup . due to left arm pain an ultrasound was completed on her left upper extremity , but it did not show deep vein thrombosis ., followup ct on 08 / 10 / 2009 showed no significant interval change . mri could not be completed due to the patient ' s weight . she was discharged on 08 / 11 / 2009 in stable condition after it was determined that this event was not neurological in origin ; however , note that ms . a referred to this as a second stroke ., ms . a presented for a followup outpatient neurological evaluation with dr . x on 09 / 22 / 2009 , at which time a brief neuropsychological screening was also conducted . she demonstrated significant impairments in confrontation naming , abstract verbal reasoning , and visual and verbal memory and thus a more comprehensive evaluation was suggested due to her intent to return to her full - time work duty . during the current interview , ms . a reported that she noticed mild memory problems including some difficultly remembering conversations , events , and at times forgetting to take her medications . she also reported mild difficulty finding words in conversation , solving novel problems and tasks ( e . g . difficulty learning to use her camcorder ), but overall denied significant cognitive deficits in attention , concentration , language or other areas of cognitive functioning . when asked about her return to work , she said that she was still on light duty due to limited physical activity because of residual left leg weakness . she reported that no one had indicated to her that she appeared less capable of performing her job duties , but said that she was also receiving fewer files to process and enter data into the computer at the social security agency that she works at . note also that she had some difficulty explaining exactly what her job involved . she also reported having problems falling asleep at work and that she is working full - time although on light duty ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    other medical history :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< OTHER_MEDICAL_HISTORY >></span>\n",
       "</mark>\n",
       " , as mentioned , ms . a continues to have some residual left leg weakness and continues to use a rolling walker for ambulation , but she reported that her motor functioning had improved significantly . she was diagnosed with sleep apnea approximately two years ago and was recently counseled by dr . x on the need to use her cpap because she indicated she never used it at night . she reported that since her appointment with dr . x , she has been using it &quot; every other night .&quot; when asked about daytime fatigue , ms . a initially denied that she was having any difficulties , but repeatedly indicated that she was falling asleep at work and thought that it was due to looking at a computer screen . she reported at times &quot; snoring &quot; and forgetting where she is at and said that a supervisor offered to give her coffee at one point . she receives approximately two to five hours of sleep per night . other current untreated risk factors include obesity and hypercholesterolemia . her medical history is also significant for hypertension , asthma , abdominal adenocarcinoma status post hysterectomy with bilateral salpingo - oophorectomy , colonic benign polyps status post resection , benign lesions of the breast status post lumpectomy , and deep vein thrombosis in the left lower extremity status post six months of anticoagulation ( which she had discontinued just prior to her stroke )., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    current medications :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< CURRENT_MEDICATIONS >></span>\n",
       "</mark>\n",
       " , aspirin 81 mg daily , colace 100 mg b . i . d ., lipitor 80 mg daily , and albuterol mdi p . r . n ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    substance use :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< SUBSTANCE_USE >></span>\n",
       "</mark>\n",
       ", ms . a denied drinking alcohol or using illicit drugs . she used to smoke a pack of cigarettes per day , but quit five to six years ago ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    family medical history :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< FAMILY_MEDICAL_HISTORY >></span>\n",
       "</mark>\n",
       " , ms . a had difficulty providing information on familial medical history . she reported that her mother died three to four years ago from lung cancer . her father has gout and blood clots . siblings have reportedly been treated for asthma and gi tumors . she was unsure of familial history of other conditions such as hypertension , high cholesterol , stroke , etc ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    social history :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< SOCIAL_HISTORY >></span>\n",
       "</mark>\n",
       " , ms . a completed high school degree . she reported that she primarily obtained b ' s and c ' s in school . she received some tutoring for algebra in middle school , but denied ever having been held back a grade failing any classes or having any problems with attention or hyperactivity ., she currently works for the social security administration in data processing . as mentioned , she has returned to full - time work , but continues to perform only light duties due to her physical condition . she is now living on her own . she has never driven . she reported that she continues to perform adls independently such as cooking and cleaning . she lost her husband in 2005 and has three adult daughters . she previously reported some concerns that her children wanted her to move into assisted living , but she did not discuss that during this current evaluation . she also reported number of other family members who had recently passed away . she has returned to activities she enjoys such as quire , knitting , and cooking and plans to go on a cruise to the bahamas at the end of october ., \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    psychiatric history :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< PSYCHIATRIC_HISTORY >></span>\n",
       "</mark>\n",
       " , ms . a did not report a history of psychological or psychiatric treatment . she reported that her current mood was good , but did describe some anxiety and nervousness about various issues such as her return to work , her upcoming trip , and other events . she reported that this only &quot; comes and goes .&quot;, \n",
       "<mark class=\"entity\" style=\"background: #dee0e3; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    tasks administered :\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\"><< TASKS_ADMINISTERED >></span>\n",
       "</mark>\n",
       ", clinical interview , adult history questionnaire , wechsler test of adult reading ( wtar ), mini mental status exam ( mmse ), cognistat neurobehavioral cognitive status examination , repeatable battery for the assessment of neuropsychological status ( rbans ; form xx ), mattis dementia rating scale , 2nd edition ( drs - 2 ), neuropsychological assessment battery ( nab ), wechsler adult intelligence scale , third edition ( wais - iii ), wechsler adult intelligence scale , fourth edition ( wais - iv ), wechsler abbreviated scale of intelligence ( wasi ), test of variables of attention ( tova ), auditory consonant trigrams ( act ), paced auditory serial addition test ( pasat ), ruff 2 &amp; 7 selective attention test , symbol digit modalities test ( sdmt ), multilingual aphasia examination , second edition ( mae - ii ), token test , sentence repetition , visual naming , controlled oral word association , spelling test , aural comprehension , reading comprehension , boston naming test , second edition ( bnt - 2 ), animal naming test</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[Section(category=None at 0 : 0 in the doc with a body at 0 : 94 based on the rule None,\n",
       " Section(category=relevant_background_information at 94 : 98 in the doc with a body at 98 : 144 based on the rule SectionRule(literal=\"RELEVANT BACKGROUND INFORMATION:\", category=\"relevant_background_information\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=history_of_presenting_problem at 144 : 149 in the doc with a body at 149 : 885 based on the rule SectionRule(literal=\"HISTORY OF PRESENTING PROBLEM:\", category=\"history_of_presenting_problem\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=other_medical_history at 885 : 889 in the doc with a body at 889 : 1190 based on the rule SectionRule(literal=\"OTHER MEDICAL HISTORY:\", category=\"other_medical_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=current_medications at 1190 : 1193 in the doc with a body at 1193 : 1231 based on the rule SectionRule(literal=\"CURRENT MEDICATIONS:\", category=\"current_medications\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=substance_use at 1231 : 1234 in the doc with a body at 1234 : 1268 based on the rule SectionRule(literal=\"SUBSTANCE USE:\", category=\"substance_use\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=family_medical_history at 1268 : 1272 in the doc with a body at 1272 : 1353 based on the rule SectionRule(literal=\"FAMILY MEDICAL HISTORY:\", category=\"family_medical_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=social_history at 1353 : 1356 in the doc with a body at 1356 : 1573 based on the rule SectionRule(literal=\"SOCIAL HISTORY:\", category=\"social_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=psychiatric_history at 1573 : 1576 in the doc with a body at 1576 : 1638 based on the rule SectionRule(literal=\"PSYCHIATRIC HISTORY:\", category=\"psychiatric_history\", pattern=None, on_match=None, parents=None, parent_required=False),\n",
       " Section(category=tasks_administered at 1638 : 1641 in the doc with a body at 1641 : 1898 based on the rule SectionRule(literal=\"TASKS ADMINISTERED:\", category=\"tasks_administered\", pattern=None, on_match=None, parents=None, parent_required=False)]"
      ]
     },
     "execution_count": 259,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "import spacy\n",
    "from medspacy.section_detection import Sectionizer\n",
    "from medspacy.section_detection import SectionRule\n",
    "\n",
    "nlp = spacy.blank(\"en\")\n",
    "sectionizer = nlp.add_pipe(\"medspacy_sectionizer\", config={\"rules\":None})\n",
    "\n",
    "# Find new sections using regex patterns & clean up strings\n",
    "sections = [x.replace(\":\",\"\").replace(\",\",\"\") for x in re.findall('[A-Z, ]*:', medical_doc)]\n",
    "new_patterns = []\n",
    "for s in sections:\n",
    "    section_cat = s.lower().replace(\" \",\"_\").replace(\":\",\"\")\n",
    "    section_lit = \"{}:\".format(s)\n",
    "    new_patterns.append(SectionRule(category=section_cat, literal=section_lit))\n",
    "\n",
    "sectionizer.add(new_patterns)\n",
    "\n",
    "doc = Doc(nlp.vocab, words=words, spaces=spaces)\n",
    "sectionizer(doc)\n",
    "visualize_ent(doc)\n",
    "\n",
    "doc._.sections"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}