1_spacy_intro.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<H1>This is a very short introduction to SpaCy.</H1>\n",
    "for further reading please navigate to:\n",
    "https://alpha.spacy.io/docs/usage/spacy-101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>What is spaCy? </h4>\n",
    "<p>\n",
    "<ul>\n",
    "<li>spaCy is a free, open-source library for advanced Natural Language Processing (NLP) in Python.</li>\n",
    "<li>\n",
    "spaCy is designed specifically for <b><em>production use</em></b> and helps you build applications that process and \"understand\" large volumes of text. It can be used to build information extraction or natural language understanding systems, or to pre-process text for deep learning.</li>\n",
    "</ul></p>\n",
    "<h4>What does it do?</h4>\n",
    "<ul>\n",
    "<li>Tokenization<p>   Segmenting text into words, punctuations marks etc.</p></li>\n",
    "<li>Part-of-speech (POS) Tagging <p>Assigning word types to tokens, like verb or noun.</p>\t</li>\n",
    "<li>Dependency Parsing<p>\tAssigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.</p>\t</li>\n",
    "<li>Lemmatization<p>\tAssigning the base forms of words. For example, the lemma of \"was\" is \"be\", and the lemma of \"rats\" is \"rat\".</p>\t</li>\n",
    "<li>Sentence Boundary Detection (SBD)<p>\tFinding and segmenting individual sentences.</p>\t</li>\n",
    "<li>Named Entity Recongition (NER)\t<p>Labelling named \"real-world\" objects, like persons, companies or locations.</p>\t</li>\n",
    "<li>Similarity<p>\tComparing words, text spans and documents and how similar they are to each other.</p>\t</li>\n",
    "<li>Text classification\t<p>Assigning categories or labels to a whole document, or parts of a document.\t</p></li>\n",
    "<li>Rule-based Matching\t<p>Finding sequences of tokens based on their texts and linguistic annotations, similar to regular expressions.</p>\t</li>\n",
    "<li>Model Training\t<p>Updating and improving all statistical models.<p>\t</li>\n",
    "<li>Language Data<p> Comes packed with laguage models (and data) for various languages (English,German,Spanish,French)</p></li>\n",
    "</ul>\n",
    "<h4>Architecture Overview</h4>\n",
    "<img src='architecture.svg'></img>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Let's see it in action</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Load the English language model in SpaCy\n",
    "# this takes a little longer because there's a lot of data to load\n",
    "import spacy                        \n",
    "nlp = spacy.load('en')\n",
    "print('Language:',nlp.lang)\n",
    "print('Vocabulary size:',nlp.vocab.length)\n",
    "print(\"Default NLP Pipeline:\")\n",
    "for obj in nlp.pipeline:\n",
    "    print('\\t',obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Default Pipeline</h4>\n",
    "<img src = 'pipeline.svg'></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#Input text\n",
    "txt =u\"\"\"Prescribing sick days due to diagnosis of influenza.Jane complains about flu-like symptoms.Jane may be experiencing some sort of flu episode.Jane’s RIDT came back negative for influenza.\n",
    "Jane is at high risk for flu if she’s not vaccinated.Jane’s older brother had the flu last month.Jane had a severe case of flu last year.Joe expressed concerns about the risks of bird flu.\n",
    "Joe shows no signs of stroke, except for numbness.Nausea, vomiting and ankle swelling negative.Patient denies alcohol abuse. Allergies: Penicillin, Dust, Sneezing.\n",
    "There's an outbreak of happiness in New York organized by O'Reilly Media, today, September 26, 2017, involving thousands of people.\"\"\"\n",
    "\n",
    "#Call Spacy on the input text\n",
    "#This runs the standard NLP pipeline on the input text\n",
    "doc = nlp(txt) \n",
    "print(doc.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Sentence detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data=[]\n",
    "for sent in doc.sents:\n",
    "    data.append((sent.start,sent.end,sent.text.replace('\\n','')))\n",
    "#For display purposes only we put the sentence boundry information in a Pandas DataFrame\n",
    "import pandas as pd\n",
    "sents = pd.DataFrame(data=data,columns = ['Start','End','Sentence Text'])\n",
    "sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part of speech tagging and Named Entity extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data = []\n",
    "for sent in doc.sents:\n",
    "    for w in sent:\n",
    "        tmp=[]\n",
    "        tmp.append(w.idx)\n",
    "        tmp.append(w.text)\n",
    "        tmp.append(w.lex_id)\n",
    "        tmp.append(w.lemma_)\n",
    "        tmp.append(w.pos_)\n",
    "        tmp.append(w.head)\n",
    "        tmp.append(w.dep_)\n",
    "        tmp.append(w.ent_type_)\n",
    "        data.append(tmp)\n",
    "tokens = pd.DataFrame(data=data, columns = ['Index','Token','Id_in_vocab',\n",
    "        'Lemma','POS','Depends_on','Dependency_type','Entity_Type'])\n",
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the syntactic dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from spacy.symbols import nsubj, VERB\n",
    "# Finding a verb with a subject \n",
    "pairs = []\n",
    "for possible_subject in doc:\n",
    "    if possible_subject.dep == nsubj and possible_subject.head.pos == VERB:\n",
    "        pairs.append((possible_subject,possible_subject.head))\n",
    "\n",
    "for pair in pairs:\n",
    "    print('Subject: ',pair[0],' verb: ',pair[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Costum pipeline... Adding negation detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#loading NegEx and it's rules\n",
    "from negex import *\n",
    "rfile = open(r'negex_triggers.txt')\n",
    "irules = sortRules(rfile.readlines())\n",
    "\n",
    "#Define a new pipeline component (based on NegEx)\n",
    "#Every pipeline component gets a Doc object and needs to return one\n",
    "#To store custom data, spaCy currently has a document level variable: doc.user_data\n",
    "#To store our negated words we add the index of the word (i) to a set under the 'negated' key\n",
    "\n",
    "def negation_tag(doc):\n",
    "    doc.user_data['negated']=set()\n",
    "    for sent in doc.sents:\n",
    "        ph= set()\n",
    "        for word in sent:\n",
    "            if word.pos_!='ADP' and word.pos_!='PUNCT':\n",
    "                ph.add(word.text)\n",
    "        tagger = negTagger(sentence = sent.text, phrases = list(ph),rules = irules, negP=False)\n",
    "        scopes=  tagger.getScopes()\n",
    "        res = set()\n",
    "        for scope in scopes:\n",
    "            s = scope.replace('[NEGATED]','').replace('.','').replace(',','')\n",
    "            if ' ' in s:\n",
    "                for wd in s.split(' '):\n",
    "                    res.add(wd)\n",
    "            else:\n",
    "                res.add(s)\n",
    "                \n",
    "        for word in sent:\n",
    "            if word.text in res:\n",
    "                doc.user_data['negated'].add(word.i)\n",
    "    return doc\n",
    "\n",
    "#add the component to the pipeline\n",
    "nlp.add_pipe(negation_tag,first=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "doc2 = nlp(txt)\n",
    "for sent in doc2.sents:\n",
    "    negs = []\n",
    "    for word in sent:\n",
    "        if word.i in doc2.user_data['negated']:\n",
    "            negs.append(word)\n",
    "    if len(negs)>0:\n",
    "        print(sent)\n",
    "        print('Negated words: ',negs)\n",
    "        print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}