01-Visualizing-POS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "<a href='http://www.pieriandata.com'> <img src='../Pierian_Data_Logo.png' /></a>\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing Parts of Speech\n",
    "spaCy offers an outstanding visualizer called **displaCy**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform standard imports\n",
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Import the displaCy library\n",
    "from spacy import displacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a simple Doc object\n",
    "doc = nlp(u\"The quick brown fox jumped over the lazy dog's back.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" id=\"0\" class=\"displacy\" width=\"1260\" height=\"357.0\" style=\"max-width: none; height: 357.0px; color: #000000; background: #ffffff; font-family: Arial\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">The</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"160\">quick</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"160\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"270\">brown</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"270\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"380\">fox</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"380\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"490\">jumped</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"490\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"600\">over</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"600\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"710\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"710\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"820\">lazy</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"820\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"930\">dog</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"930\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1040\">'s</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1040\">PART</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"267.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1150\">back.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1150\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-0\" stroke-width=\"2px\" d=\"M70,222.0 C70,57.0 375.0,57.0 375.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-0\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,224.0 L62,212.0 78,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-1\" stroke-width=\"2px\" d=\"M180,222.0 C180,112.0 370.0,112.0 370.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-1\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M180,224.0 L172,212.0 188,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-2\" stroke-width=\"2px\" d=\"M290,222.0 C290,167.0 365.0,167.0 365.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-2\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M290,224.0 L282,212.0 298,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-3\" stroke-width=\"2px\" d=\"M400,222.0 C400,167.0 475.0,167.0 475.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-3\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M400,224.0 L392,212.0 408,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-4\" stroke-width=\"2px\" d=\"M510,222.0 C510,167.0 585.0,167.0 585.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-4\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M585.0,224.0 L593.0,212.0 577.0,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-5\" stroke-width=\"2px\" d=\"M730,222.0 C730,112.0 920.0,112.0 920.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-5\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M730,224.0 L722,212.0 738,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-6\" stroke-width=\"2px\" d=\"M840,222.0 C840,167.0 915.0,167.0 915.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-6\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M840,224.0 L832,212.0 848,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-7\" stroke-width=\"2px\" d=\"M950,222.0 C950,112.0 1140.0,112.0 1140.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-7\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">poss</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M950,224.0 L942,212.0 958,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-8\" stroke-width=\"2px\" d=\"M950,222.0 C950,167.0 1025.0,167.0 1025.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-8\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1025.0,224.0 L1033.0,212.0 1017.0,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-9\" stroke-width=\"2px\" d=\"M620,222.0 C620,2.0 1150.0,2.0 1150.0,222.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-9\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1150.0,224.0 L1158.0,212.0 1142.0,212.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Render the dependency parse immediately inside Jupyter:\n",
    "displacy.render(doc, style='dep', jupyter=True, options={'distance': 110})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dependency parse shows the coarse POS tag for each token, as well as the **dependency tag** if given:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The        DET     det     determiner\n",
      "quick      ADJ     amod    adjectival modifier\n",
      "brown      ADJ     amod    adjectival modifier\n",
      "fox        NOUN    nsubj   nominal subject\n",
      "jumped     VERB    ROOT    None\n",
      "over       ADP     prep    prepositional modifier\n",
      "the        DET     det     determiner\n",
      "lazy       ADJ     amod    adjectival modifier\n",
      "dog        NOUN    poss    possession modifier\n",
      "'s         PART    case    None\n",
      "back       NOUN    pobj    object of preposition\n",
      ".          PUNCT   punct   punctuation\n"
     ]
    }
   ],
   "source": [
    "for token in doc:\n",
    "    print(f'{token.text:{10}} {token.pos_:{7}} {token.dep_:{7}} {spacy.explain(token.dep_)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "# Creating Visualizations Outside of Jupyter\n",
    "If you're using another Python IDE or writing a script, you can choose to have spaCy serve up HTML separately.\n",
    "\n",
    "Instead of `displacy.render()`, use `displacy.serve()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Serving on port 5000...\n",
      "    Using the 'dep' visualizer\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [12/Oct/2018 16:54:07] \"GET / HTTP/1.1\" 200 8304\n",
      "127.0.0.1 - - [12/Oct/2018 16:54:07] \"GET /favicon.ico HTTP/1.1\" 200 8304\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Shutting down server on port 5000.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "displacy.serve(doc, style='dep', options={'distance': 110})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=blue>**After running the cell above, click the link below to view the dependency parse**:</font>\n",
    "\n",
    "http://127.0.0.1:5000\n",
    "<br><br>\n",
    "<font color=red>**To shut down the server and return to jupyter**, interrupt the kernel either through the **Kernel** menu above, by hitting the black square on the toolbar, or by typing the keyboard shortcut `Esc`, `I`, `I`</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=green>**NOTE**: We'll use this method moving forward because, at this time, several of the customizations we want to show don't work well in Jupyter.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Handling Large Text\n",
    "`displacy.serve()` accepts a single Doc or list of Doc objects. Since large texts are difficult to view in one line, you may want to pass a list of spans instead. Each span will appear on its own line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Serving on port 5000...\n",
      "    Using the 'dep' visualizer\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [12/Oct/2018 16:57:03] \"GET / HTTP/1.1\" 200 7328\n",
      "127.0.0.1 - - [12/Oct/2018 16:57:03] \"GET /favicon.ico HTTP/1.1\" 200 7328\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Shutting down server on port 5000.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "doc2 = nlp(u\"This is a sentence. This is another, possibly longer sentence.\")\n",
    "\n",
    "# Create spans from Doc.sents:\n",
    "spans = list(doc2.sents)\n",
    "\n",
    "displacy.serve(spans, style='dep', options={'distance': 110})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Click this link to view the dependency**: http://127.0.0.1:5000\n",
    "<br>Interrupt the kernel to return to jupyter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Customizing the Appearance\n",
    "Besides setting the distance between tokens, you can pass other arguments to the `options` parameter:\n",
    "\n",
    "<table>\n",
    "<tr><th>NAME</th><th>TYPE</th><th>DESCRIPTION</th><th>DEFAULT</th></tr>\n",
    "<tr><td>`compact`</td><td>bool</td><td>\"Compact mode\" with square arrows that takes up less space.</td><td>`False`</td></tr>\n",
    "<tr><td>`color`</td><td>unicode</td><td>Text color (HEX, RGB or color names).</td><td>`#000000`</td></tr>\n",
    "<tr><td>`bg`</td><td>unicode</td><td>Background color (HEX, RGB or color names).</td><td>`#ffffff`</td></tr>\n",
    "<tr><td>`font`</td><td>unicode</td><td>Font name or font family for all text.</td><td>`Arial`</td></tr>\n",
    "</table>\n",
    "\n",
    "For a full list of options visit https://spacy.io/api/top-level#displacy_options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Serving on port 5000...\n",
      "    Using the 'dep' visualizer\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [12/Oct/2018 17:02:17] \"GET / HTTP/1.1\" 200 8533\n",
      "127.0.0.1 - - [12/Oct/2018 17:02:17] \"GET /favicon.ico HTTP/1.1\" 200 8533\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Shutting down server on port 5000.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "options = {'distance': 110, 'compact': 'True', 'color': 'yellow', 'bg': '#09a3d5', 'font': 'Times'}\n",
    "\n",
    "displacy.serve(doc, style='dep', options=options)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Click this link to view the dependency**: http://127.0.0.1:5000\n",
    "<br>Interrupt the kernel to return to jupyter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "___\n",
    "Great! Now you should be familiar with visualizing spaCy's dependency parse. For more info on **displaCy** visit https://spacy.io/usage/visualizers\n",
    "<br>In the next section we'll look at Named Entity Recognition, followed by displaCy's NER visualizer.\n",
    "\n",
    "### Next Up: Named Entity Recognition"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}