wiki/president_place_of_birth_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Extracting the Place of Birth of American Presidents from their Wikipedia HTML Pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you already have the database pob_presidents in your postgresql, please uncomment the first line to drop and create it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#! dropdb --if-exists pob_presidents #Uncomment to remove databases created in previous runs\n",
    "! createdb pob_presidents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "We will walk through the process of using `Fonduer` to extract relations from [**richly formatted** data](https://hazyresearch.github.io/snorkel/blog/fonduer.html), where information is conveyed via combinations of textual, structural and tabular, and visual expressions, as seen in webpages, business reports, product specifications, and scientific literature.\n",
    "\n",
    "In this tutorial, we use `Fonduer` to identify the place of birth of presidents and vice presidents from their wikipedia articles. The HTML files are generated from Wikipedia's markup language Wikitext via [wtf_wikipedia](https://github.com/spencermountain/wtf_wikipedia).\n",
    "\n",
    "The tutorial is broken into several parts, each covering a phase of the `Fonduer` pipeline (as outlined in the [paper](https://arxiv.org/abs/1703.05028)), and the iterative KBC process:\n",
    "\n",
    "1. KBC Initialization\n",
    "2. Candidate Generation and Multimodal Featurization\n",
    "3. Probabilistic Relation Classification\n",
    "4. Error Analysis and Iterative KBC\n",
    "\n",
    "In addition, we show how users can iteratively improve labeling functions to improve relation extraction quality.\n",
    "\n",
    "# Phase 1: KBC Initialization\n",
    "\n",
    "In this first phase of `Fonduer`'s pipeline, `Fonduer` uses a user-specified _schema_ to initialize a relational database where the output KB will be stored. Furthermore, `Fonduer` iterates over its input _corpus_ and transforms each document into a unified data model, which captures the variability and multimodality of richly formatted data. This unified data model then servers as an intermediate representation used in the rest of the phases.\n",
    "\n",
    "This preprocessed data is saved to a database. The connection string to the database is provided to the `Meta` object, which will initialize a PostgreSQL database for parallel execution.\n",
    "\n",
    "We initialize several variables for convenience that define what the database should be called and what level of parallelization the `Fonduer` pipeline will be run with.\n",
    "\n",
    "Before you continue, please make sure that you have PostgreSQL installed and have created a new database named `pob_presidents`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "import sys\n",
    "import logging\n",
    "\n",
    "PARALLEL = 4 # assuming a quad-core machine\n",
    "ATTRIBUTE = \"pob_presidents\"\n",
    "conn_string = 'postgresql://localhost:5432/' + ATTRIBUTE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Parsing and Transforming the Input Documents into Unified Data Models\n",
    "\n",
    "We first initialize a `Meta` object, which manages the connection to the database automatically, and enables us to save intermediate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer import Meta, init_logging\n",
    "\n",
    "# Configure logging for Fonduer\n",
    "init_logging(log_dir=\"logs\")\n",
    "\n",
    "session = Meta.init(conn_string).Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the corpus of wikpedia articles and transform them into the unified data model. Each article is represented by an HTML file. The HTML file is parsed to create a robust unified data model with textual, structural, and tabular modality information. Note that since each document is independent of each other, we can parse the documents in parallel. We depend on PostgreSQL for this functionality.\n",
    "\n",
    "### Configuring an `HTMLDocPreprocessor`\n",
    "We start by setting the paths to where our documents are stored, and defining a `HTMLDocPreprocessor` to read in the documents found in the specified paths. `max_docs` specified the maximum number of documents to parse.\n",
    "\n",
    "**Note that you need to have run `download_data.sh` before executing these next steps or you won't have the documents needed for the tutorial.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.parser.preprocessors import HTMLDocPreprocessor\n",
    "from fonduer.parser import Parser\n",
    "\n",
    "docs_path = \"data/presidents/\"\n",
    "doc_preprocessor = HTMLDocPreprocessor(docs_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring a `Parser`\n",
    "Next, we configure a `Parser`, which serves as our `CorpusParser` for PDF documents. We use [spaCy](https://spacy.io/) as a preprocessing tool to split our documents into sentences and tokens.In addition, we can specify which modality information to include in the unified data model for each document. Below, we enable structural information, as well as lingual information, which uses [spaCy] to provide annotations such as part-of-speech tags and dependency parse structures for these sentences. \n",
    "Note that after the progress bar indicates the completion of the parsing process, some more time will pass until all objects have been inserted into the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_parser = Parser(session, structural=True, lingual=True)\n",
    "%time corpus_parser.apply(doc_preprocessor, parallelism=PARALLEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use simple database queries (written in the syntax of [SQLAlchemy](http://www.sqlalchemy.org/), which `Fonduer` uses) to check how many documents and sentences were parsed, or even check how many sentences are contained in each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.parser.models import Document, Sentence\n",
    "\n",
    "print(f\"Documents: {session.query(Document).count()}\")\n",
    "print(f\"Sentences: {session.query(Sentence).count()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Dividing the Corpus into Test and Train\n",
    "\n",
    "We'll split the documents 80/10/10 into train/dev/test splits. Note that here we do this in a non-random order to preverse the consistency in the tutorial, and we reference the splits by 0/1/2 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = session.query(Document).order_by(Document.name).all()\n",
    "ld = len(docs)\n",
    "\n",
    "train_docs = set()\n",
    "dev_docs = set()\n",
    "test_docs = set()\n",
    "splits = (0.7, 0.85)\n",
    "data = [(doc.name, doc) for doc in docs]\n",
    "data.sort(key=lambda x: x[0])\n",
    "for i, (doc_name, doc) in enumerate(data):\n",
    "    if i < splits[0] * ld:\n",
    "        train_docs.add(doc)\n",
    "    elif i < splits[1] * ld:\n",
    "        dev_docs.add(doc)\n",
    "    else:\n",
    "        test_docs.add(doc)\n",
    "from pprint import pprint\n",
    "\n",
    "pprint([x.name for x in train_docs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phase 2: Mention Extraction, Candidate Extraction Multimodal Featurization\n",
    "\n",
    "Given the unified data model from Phase 1, `Fonduer` extracts relation\n",
    "candidates based on user-provided **matchers** and **throttlers**. Then,\n",
    "`Fonduer` leverages the multimodality information captured in the unified data\n",
    "model to provide multimodal features for each candidate.\n",
    "\n",
    "## 2.1 Mention Extraction\n",
    "\n",
    "The first step is to extract **mentions** from our corpus. A `mention` is the\n",
    "type of object which makes up a `candidate`. For example, if we wanted to\n",
    "extract pairs of (vice) president names and their corresponding place of birth, the name would be one `mention` while\n",
    "the place of birth would be another. These `mention`s are then combined to\n",
    "create `candidates`, where our task is to predict which `candidates` are true\n",
    "in the associated document.\n",
    "\n",
    "We first start by defining and naming our two `mention`s:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.models import mention_subclass\n",
    "\n",
    "Presidentname = mention_subclass(\"Presidentname\")\n",
    "Placeofbirth = mention_subclass(\"Placeofbirth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we write **matchers** to define which spans of text in the corpus are\n",
    "instances of each entity. Matchers can leverage a variety of information from\n",
    "regular expressions, to dictionaries, to user-defined functions. Furthermore,\n",
    "different techniques can be combined to form higher quality matchers. In\n",
    "general, matchers should seek to be as precise as possible while maintaining\n",
    "complete recall. More documentation about Matchers can be found on [Read the Docs](https://fonduer.readthedocs.io/en/stable/user/candidates.html#matchers).\n",
    "\n",
    "In our case, we need to write a matcher that defines a string for the name of \n",
    "the president and another matcher to define the place of birth\n",
    "\n",
    "### Writing a simple name matcher\n",
    "\n",
    "Our name matcher makes use of the fact that the name of the president is \n",
    "indicated in the html file name. We define a function that compares a `span` of \n",
    "text with the file name to find mentions of the president.\n",
    "Because we previously converted the document into a unified data model, each\n",
    "`span` of text has a reference to its source document name, which makes comparison\n",
    "with the source file name easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mention_span_matches_file_name(mention):\n",
    "    president_name_string = mention.get_span()\n",
    "    file_name = mention.sentence.document.name.replace(\"_\", \" \")\n",
    "    if president_name_string == file_name:\n",
    "        return True\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a `LambdaFunctionMatcher` to wrap this function into a `Fonduer` mention matcher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.matchers import LambdaFunctionMatcher, Intersect, Union\n",
    "\n",
    "president_name_matcher = LambdaFunctionMatcher(func=mention_span_matches_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a place of birth matcher\n",
    "\n",
    "While the place of birth is also a regular string, we rather want to match it based on it's location in the document.\n",
    "By inspecting the html files, we can find out that each file contains a table with the most important information about the respective president. We observe that there is typically a label in the left column and the corresponding value in the right column.\n",
    "To capture this information, we define a function that checks whether a `span` is inside a table and its neighboring cells in the same row contain the label 'birth place'.\n",
    "Furthermore, we observe that birth places are typically split into a fine-grained location, followed by its respective state. As we want to gather more fine-grained information, we specify to extract only the first `spans` of text inside the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.utils.data_model_utils import get_row_ngrams\n",
    "\n",
    "\n",
    "def is_in_birthplace_table_row(mention):\n",
    "    if not mention.sentence.is_tabular():\n",
    "        return False\n",
    "    ngrams = get_row_ngrams(mention, lower=True)\n",
    "    birth_place_words = set([\"birth\", \"place\"])\n",
    "    if birth_place_words <= set(ngrams):\n",
    "        return True\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "\n",
    "def birthplace_left_aligned_to_punctuation(mention):\n",
    "    # Return false, if the cell containing the text candidate does not have any reference\n",
    "    # to `sentence` objects\n",
    "    for sentence in mention.sentence.cell.sentences:\n",
    "        sentence_parts = sentence.text.split(\",\")\n",
    "        for sentence_part in sentence_parts:\n",
    "            if sentence_part.startswith(mention.get_span()):\n",
    "                return True\n",
    "    return False\n",
    "\n",
    "\n",
    "# We only want one granularity of the birth place\n",
    "def no_commas_in_birth_place(mention):\n",
    "    if \",\" in mention.get_span():\n",
    "        return False\n",
    "    else:\n",
    "        return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After defining all functions to capture the properties of a birth place mention, we combine them via an `Intersect` matcher. This matcher will only select a `span`, if all three functions agree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "birth_place_in_labeled_row_matcher = LambdaFunctionMatcher(\n",
    "    func=is_in_birthplace_table_row\n",
    ")\n",
    "birth_place_in_labeled_row_matcher.longest_match_only = False\n",
    "birth_place_no_commas_matcher = LambdaFunctionMatcher(func=no_commas_in_birth_place)\n",
    "birth_place_left_aligned_matcher = LambdaFunctionMatcher(\n",
    "    func=birthplace_left_aligned_to_punctuation\n",
    ")\n",
    "\n",
    "place_of_birth_matcher = Intersect(\n",
    "    birth_place_in_labeled_row_matcher,\n",
    "    birth_place_no_commas_matcher,\n",
    "    birth_place_left_aligned_matcher,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two matchers define each entity in our relation schema."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a Mention's `MentionSpace`\n",
    "\n",
    "Next, in order to define the \"space\" of all mentions that are even considered\n",
    "from the document, we need to define a `MentionSpace` for each component of the\n",
    "relation we wish to extract. Fonduer provides a default `MentionSpace` for you\n",
    "to use, but you can also extend the default `MentionSpace` depending on your\n",
    "needs.\n",
    "\n",
    "In the case of names, the `MentionSpace` can be relatively simple: We know that\n",
    "each name will contain at least two words (first name, last name). Considering\n",
    "additional middle names, we expect a maximum of four words per name.\n",
    "\n",
    "Similarly, we expect the place of birth to be a `span` of one to three words.\n",
    "\n",
    "We use the default `Ngrams` class provided by `fonduer` to define these properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates import MentionNgrams\n",
    "\n",
    "presname_ngrams = MentionNgrams(n_max=4, n_min=2)\n",
    "placeofbirth_ngrams = MentionNgrams(n_max=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Mention Extraction \n",
    "\n",
    "Next, we create a `MentionExtractor` to extract the mentions from all of\n",
    "our documents based on the `MentionSpace` and matchers we defined above.\n",
    "\n",
    "View the API for the MentionExtractor on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/candidates.html#fonduer.candidates.MentionExtractor).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates import MentionExtractor\n",
    "\n",
    "mention_extractor = MentionExtractor(\n",
    "    session,\n",
    "    [Presidentname, Placeofbirth],\n",
    "    [presname_ngrams, placeofbirth_ngrams],\n",
    "    [president_name_matcher, place_of_birth_matcher],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we run the extractor on all of our documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.models import Mention\n",
    "\n",
    "mention_extractor.apply(docs, parallelism=PARALLEL)\n",
    "num_names = session.query(Presidentname).count()\n",
    "num_pobs = session.query(Placeofbirth).count()\n",
    "print(\n",
    "    f\"Total Mentions: {session.query(Mention).count()} ({num_names} names, {num_pobs} places of birth)\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Candidate Extraction\n",
    "\n",
    "Now that we have both defined and extracted the Mentions that can be used to compose Candidates, we are ready to move on to extracting Candidates. Like we did with the Mentions, we first define what each candidate schema looks like. In this example, we create a candidate that is composed of a `Presidentname` and a `Placeofbirth` mention as we defined above. We name this candidate \"PresidentnamePlaceofbirth\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.models import candidate_subclass\n",
    "\n",
    "PresidentnamePlaceofbirth = candidate_subclass(\n",
    "    \"PresidentnamePlaceofbirth\", [Presidentname, Placeofbirth]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining candidate `Throttlers`\n",
    "\n",
    "Next, we need to define **throttlers**, which allow us to further prune excess candidates and avoid unnecessarily materializing invalid candidates. Throttlers, like matchers, act as hard filters, and should be created to have high precision while maintaining complete recall, if possible.\n",
    "\n",
    "Here, we create a throttler that discards candidates if they are in the same table. \n",
    "NOTE: Currently, we don't use this throttler, as issues with `wtf_wikipedia` lead to the absence of the president name in the table. We therefore skip this stip for now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the `CandidateExtractor`\n",
    "\n",
    "Now, we have all the component necessary to perform candidate extraction. We have defined the Mentions that compose each candidate and a throttler to prunes away excess candidates. We now can define the `CandidateExtractor` with the candidate subclass and corresponding throttler to use.\n",
    "\n",
    "View the API for the CandidateExtractor on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/candidates.html#fonduer.candidates.CandidateExtractor)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates import CandidateExtractor\n",
    "\n",
    "\n",
    "candidate_extractor = CandidateExtractor(session, [PresidentnamePlaceofbirth])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we specified that these `Candidates` belong to the training set by specifying `split=0`; recall that we're referring to train/dev/test as splits 0/1/2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, docs in enumerate([train_docs, dev_docs, test_docs]):\n",
    "    candidate_extractor.apply(docs, split=i, parallelism=PARALLEL)\n",
    "    print(\n",
    "        f\"Number of Candidates in split={i}: {session.query(PresidentnamePlaceofbirth).filter(PresidentnamePlaceofbirth.split == i).count()}\"\n",
    "    )\n",
    "\n",
    "train_cands = candidate_extractor.get_candidates(split=0)\n",
    "dev_cands = candidate_extractor.get_candidates(split=1)\n",
    "test_cands = candidate_extractor.get_candidates(split=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Multimodal Featurization\n",
    "Unlike dealing with plain unstructured text, `Fonduer` deals with richly formatted data, and consequently featurizes each candidate with a baseline library of multimodal features. \n",
    "\n",
    "### Featurize with `Fonduer`'s optimized Postgres Featurizer\n",
    "We now annotate the candidates in our training, dev, and test sets with features. The `Featurizer` provided by `Fonduer` allows this to be done in parallel to improve performance.\n",
    "\n",
    "View the API provided by the `Featurizer` on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/features.html#fonduer.features.Featurizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.features import Featurizer\n",
    "\n",
    "featurizer = Featurizer(session, [PresidentnamePlaceofbirth])\n",
    "%time featurizer.apply(split=0, train=True, parallelism=PARALLEL)\n",
    "%time F_train = featurizer.get_feature_matrices(train_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(F_train[0].shape)\n",
    "%time featurizer.apply(split=1, parallelism=PARALLEL)\n",
    "%time F_dev = featurizer.get_feature_matrices(dev_cands)\n",
    "print(F_dev[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time featurizer.apply(split=2, parallelism=PARALLEL)\n",
    "%time F_test = featurizer.get_feature_matrices(test_cands)\n",
    "print(F_test[0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this phase, `Fonduer` has generated the set of candidates and the feature matrix. Note that Phase 1 and 2 are relatively static and typically are only executed once during the KBC process.\n",
    "\n",
    "# Phase 3: Probabilistic Relation Classification\n",
    "In this phase, `Fonduer` applies user-defined **labeling functions**, which express various heuristics, patterns, and [weak supervision](http://hazyresearch.github.io/snorkel/blog/weak_supervision.html) strategies to label our data, to each of the candidates to create a label matrix that is used by our data programming engine.\n",
    "\n",
    "In the wild, hand-labeled training data is rare and expensive. A common scenario is to have access to tons of unlabeled training data, and have some idea of how to label them programmatically. For example:\n",
    "* We may have knowledge about typical place constructs, such as combinations of strings with words like 'New','County' or 'City'.\n",
    "Our labeling functions will capture these types of strategies. We know that these labeling functions will not be perfect, and some may be quite low-quality, so we will model their accuracies with a generative model, which `Fonduer` will help us easily apply.\n",
    "\n",
    "Using data programming, we can then train machine learning models to learn which features are the most important in classifying candidates.\n",
    "\n",
    "### Loading Gold Data\n",
    "For convenience in error analysis and evaluation, we have already annotated the dev and test set for this tutorial, and we'll now load it using an externally-defined helper function. If you're interested in the example implementation details, please see the script we now load:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.supervision.models import GoldLabel\n",
    "from wiki_table_utils import gold\n",
    "\n",
    "from fonduer.supervision import Labeler\n",
    "\n",
    "docs = corpus_parser.get_documents()\n",
    "labeler = Labeler(session, [PresidentnamePlaceofbirth])\n",
    "%time labeler.apply(docs=docs, lfs=[[gold]], table=GoldLabel, train=True, parallelism=PARALLEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Labeling Functions\n",
    "\n",
    "In `Fonduer`, our primary interface through which we provide training signal to\n",
    "the end extraction model we are training is by writing labeling functions\n",
    "(**LFs**) (as opposed to hand-labeling massive training sets).\n",
    "\n",
    "A labeling function isn't anything special. It's just a Python function that\n",
    "accepts a `Candidate` as the input argument and returns `2` if it says the\n",
    "Candidate should be marked as true, `1` if it says the `Candidate` should be\n",
    "marked as false, and `0` if it doesn't know how to vote and abstains. In\n",
    "practice, many labeling functions are unipolar: it labels only 2s and 0s, or it\n",
    "labels only 1s and 0s.\n",
    "\n",
    "Recall that our goal is ultimately to train a high-performance classification\n",
    "model that predicts which of our Candidates are true mentions of president name to place of birth relations. It turns out that we can do this by writing\n",
    "potentially low-quality labeling functions!\n",
    "\n",
    "With `Fonduer`, labeling functions can be written using intuitive patterns\n",
    "discovered by inspecting the target corpus. A library of data model utilities\n",
    "which can be used to write labeling functions are outline in [Read the\n",
    "Docs](http://fonduer.readthedocs.io/en/stable/user/data_model_utils.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We express several of these simple patterns below as a set of labeling functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.labeling import labeling_function\n",
    "from fonduer.utils.data_model_utils import *\n",
    "import re\n",
    "\n",
    "ABSTAIN = -1\n",
    "FALSE = 0\n",
    "TRUE = 1\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_place_of_birth_has_link(c):\n",
    "    place = c.placeofbirth\n",
    "    ancestor_tag_names = get_ancestor_tag_names(place)\n",
    "    if len(ancestor_tag_names) > 1 and \"a\" in ancestor_tag_names:\n",
    "        return ABSTAIN\n",
    "    else:\n",
    "        return FALSE\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_place_of_birth_is_longest_ordered_span_before_comma(c):\n",
    "    place = c.placeofbirth\n",
    "    place_string = place.context.get_span()\n",
    "    place_sentence_string = place.context.sentence.text\n",
    "    left_aligned_first_span = place_sentence_string.split(\",\")[0]\n",
    "    if place_string == left_aligned_first_span:\n",
    "        return TRUE\n",
    "    else:\n",
    "        return FALSE\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_place_in_first_sentence_of_cell(c):\n",
    "    place = c.placeofbirth\n",
    "    place_sentence = place.context.sentence\n",
    "    place_cell = place_sentence.cell\n",
    "    if place_sentence == place_cell.sentences[0]:\n",
    "        return TRUE\n",
    "    else:\n",
    "        return FALSE\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_place_is_full_sentence(c):\n",
    "    place = c.placeofbirth\n",
    "    place_sentence = place.context.sentence\n",
    "    if place.context.get_span() == place_sentence.text:\n",
    "        return ABSTAIN\n",
    "    else:\n",
    "        return FALSE\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_place_not_a_US_state(c):\n",
    "    place = c.placeofbirth\n",
    "    place_string = place.context.get_span().lower()\n",
    "    if place_string is None:\n",
    "        return FALSE\n",
    "    state_dictionary = set(\n",
    "        x.lower()\n",
    "        for x in [\n",
    "            \"Alabama\",\n",
    "            \"Alaska\",\n",
    "            \"Arizona\",\n",
    "            \"Arkansas\",\n",
    "            \"California\",\n",
    "            \"Colorado\",\n",
    "            \"Connecticut\",\n",
    "            \"Delaware\",\n",
    "            \"Florida\",\n",
    "            \"Georgia\",\n",
    "            \"Hawaii\",\n",
    "            \"Idaho\",\n",
    "            \"Illinois\",\n",
    "            \"Indiana\",\n",
    "            \"Iowa\",\n",
    "            \"Kansas\",\n",
    "            \"Kentucky\",\n",
    "            \"Louisiana\",\n",
    "            \"Maine\",\n",
    "            \"Maryland\",\n",
    "            \"Massachusetts\",\n",
    "            \"Michigan\",\n",
    "            \"Minnesota\",\n",
    "            \"Mississippi\",\n",
    "            \"Missouri\",\n",
    "            \"Montana\",\n",
    "            \"Nebraska\",\n",
    "            \"Nevada\",\n",
    "            \"New Hampshire\",\n",
    "            \"New Jersey\",\n",
    "            \"New Mexico\",\n",
    "            \"New York\",\n",
    "            \"North Carolina\",\n",
    "            \"North Dakota\",\n",
    "            \"Ohio\",\n",
    "            \"Oklahoma\",\n",
    "            \"Oregon\",\n",
    "            \"Pennsylvania\",\n",
    "            \"Rhode Island\",\n",
    "            \"South Carolina\",\n",
    "            \"South Dakota\",\n",
    "            \"Tennessee\",\n",
    "            \"Texas\",\n",
    "            \"Utah\",\n",
    "            \"Vermont\",\n",
    "            \"Virginia\",\n",
    "            \"Washington\",\n",
    "            \"West Virginia\",\n",
    "            \"Wisconsin\",\n",
    "            \"Wyoming\",\n",
    "        ]\n",
    "    )\n",
    "    if place_string == \"new york city\":  # exception\n",
    "        return TRUE\n",
    "    if place_string in state_dictionary:\n",
    "        return FALSE\n",
    "    elif any(x in place_string for x in state_dictionary):\n",
    "        return FALSE\n",
    "    else:\n",
    "        return ABSTAIN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we collect all of the labeling function we would like to use into a single list, which is provided as input to the `Labeler`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "president_name_pob_lfs = [\n",
    "    LF_place_of_birth_has_link,\n",
    "    LF_place_of_birth_is_longest_ordered_span_before_comma,\n",
    "    LF_place_not_a_US_state,\n",
    "    LF_place_in_first_sentence_of_cell,\n",
    "    LF_place_is_full_sentence,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying the Labeling Functions\n",
    "\n",
    "Next, we need to actually run the LFs over all of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database. Note that this will delete any existing `Labels` and `LabelKeys` for this candidate set.\n",
    "\n",
    "View the API provided by the `Labeler` on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/supervision.html#fonduer.supervision.Labeler)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time labeler.apply(split=0, lfs=[president_name_pob_lfs], train=True, parallelism=PARALLEL)\n",
    "%time L_train = labeler.get_label_matrices(train_cands)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also view statistics about the resulting label matrix.\n",
    "* **Coverage** is the fraction of candidates that the labeling function emits a non-zero label for.\n",
    "* **Overlap** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a non-zero label for.\n",
    "* **Conflict** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a conflicting non-zero label for.\n",
    "\n",
    "In addition, because we have already loaded the gold labels, we can view the emperical accuracy of these labeling functions when compared to our gold labels using the `LFAnalysis` module of [Snorkel](https://github.com/snorkel-team/snorkel)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L_gold_train = labeler.get_gold_labels(train_cands, annotator=\"gold\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.labeling import LFAnalysis\n",
    "\n",
    "LFAnalysis(\n",
    "    L_train[0],\n",
    "    lfs=sorted(president_name_pob_lfs, key=lambda lf: lf.name)\n",
    ").lf_summary(Y=L_gold_train[0].reshape(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting the Generative Model\n",
    "\n",
    "Now, we'll train a model of the LFs to estimate their accuracies. Once the model is trained, we can combine the outputs of the LFs into a single, noise-aware training label set for our extractor. Intuitively, we'll model the LFs by observing how they overlap and conflict with each other. To do so, we use [Snorkel](https://github.com/snorkel-team/snorkel)'s single-task label model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.labeling.model import LabelModel\n",
    "\n",
    "gen_model = LabelModel(cardinality=2)\n",
    "%time gen_model.fit(L_train[0], n_epochs=500, log_freq=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now apply the generative model to the training candidates to get the noise-aware training label set. We'll refer to these as the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_marginals = gen_model.predict_proba(L_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at the distribution of the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.hist(train_marginals[:, TRUE], bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Model to Iterate on Labeling Functions\n",
    "\n",
    "Now that we have learned the generative model, we can stop here and use this to potentially debug and/or improve our labeling function set. First, we apply the LFs to our development set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labeler.apply(split=1, lfs=[president_name_pob_lfs], parallelism=PARALLEL)\n",
    "%time L_dev = labeler.get_label_matrices(dev_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L_dev[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Generative Model Performance\n",
    "\n",
    "At this point, we should be getting an F1 score of around 0.6 to 0.7 on the development set, which is pretty good! However, we should be very careful in interpreting this. Since we developed our labeling functions using this development set as a guide, and our generative model is composed of these labeling functions, we expect it to score very well here!\n",
    "\n",
    "In fact, it is probably somewhat overfit to this set. However this is fine, since in the next, we'll train a more powerful end extraction model which will generalize beyond the development set, and which we will evaluate on a blind test set (i.e. one we never looked at during development).\n",
    "\n",
    "\n",
    "### Training the Discriminative Model\n",
    "\n",
    "Now, we'll use the noisy training labels we generated in the last part to train our end extraction model. For this tutorial, we will be training a simple--but fairly effective--logistic regression model.\n",
    "\n",
    "We use the training marginals to train a discriminative model that classifies each Candidate as a true or false mention.\n",
    "\n",
    "In `Fonduer`, we use a new machine learning framework [Emmental](https://github.com/SenWu/emmental) to support all model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import emmental\n",
    "\n",
    "# Setup training config\n",
    "config = {\n",
    "    \"meta_config\": {\"verbose\": True},\n",
    "    \"model_config\": {\"model_path\": None, \"device\": 0, \"dataparallel\": False},\n",
    "    \"learner_config\": {\n",
    "        \"n_epochs\": 30,\n",
    "        \"optimizer_config\": {\"lr\": 0.001, \"l2\": 0.0},\n",
    "        \"task_scheduler\": \"round_robin\",\n",
    "    },\n",
    "    \"logging_config\": {\n",
    "        \"evaluation_freq\": 1,\n",
    "        \"counter_unit\": \"epoch\",\n",
    "        \"checkpointing\": False,\n",
    "        \"checkpointer_config\": {\n",
    "            \"checkpoint_metric\": {f\"{ATTRIBUTE}/{ATTRIBUTE}/train/loss\": \"min\"},\n",
    "            \"checkpoint_freq\": 1,\n",
    "            \"checkpoint_runway\": 2,\n",
    "            \"clear_intermediate_checkpoints\": True,\n",
    "            \"clear_all_checkpoints\": True,\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "emmental.init(Meta.log_path)\n",
    "emmental.Meta.update_config(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collect word counter from training data\n",
    "from fonduer.learning.utils import collect_word_counter\n",
    "\n",
    "word_counter = collect_word_counter(train_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate word embedding module for LSTM model\n",
    "# (in Logistic Regression, we generate it since Fonduer dataset requires word2id dict)\n",
    "from emmental.modules.embedding_module import EmbeddingModule\n",
    "\n",
    "arity = 2\n",
    "\n",
    "# Geneate special tokens\n",
    "specials = []\n",
    "for i in range(arity):\n",
    "    specials += [f\"~~[[{i}\", f\"{i}]]~~\"]\n",
    "\n",
    "emb_layer = EmbeddingModule(\n",
    "    word_counter=word_counter, word_dim=300, specials=specials\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate dataloader for training set\n",
    "from emmental.data import EmmentalDataLoader\n",
    "from fonduer.learning.dataset import FonduerDataset\n",
    "import numpy as np\n",
    "\n",
    "# Filter out noise samples\n",
    "diffs = train_marginals.max(axis=1) - train_marginals.min(axis=1)\n",
    "train_idxs = np.where(diffs > 1e-6)[0]\n",
    "\n",
    "train_dataloader = EmmentalDataLoader(\n",
    "    task_to_label_dict={ATTRIBUTE: \"labels\"},\n",
    "    dataset=FonduerDataset(\n",
    "        ATTRIBUTE,\n",
    "        train_cands[0],\n",
    "        F_train[0],\n",
    "        emb_layer.word2id,\n",
    "        train_marginals,\n",
    "        train_idxs,\n",
    "    ),\n",
    "    split=\"train\",\n",
    "    batch_size=100,\n",
    "    shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creaet task and model, and perform training\n",
    "from emmental.model import EmmentalModel\n",
    "from fonduer.learning.task import create_task\n",
    "from emmental.learner import EmmentalLearner\n",
    "\n",
    "tasks = create_task(\n",
    "    ATTRIBUTE, 2, F_train[0].shape[1], 2, emb_layer, model=\"LogisticRegression\"\n",
    ")\n",
    "\n",
    "model = EmmentalModel(name=f\"{ATTRIBUTE}_task\")\n",
    "\n",
    "for task in tasks:\n",
    "    model.add_task(task)\n",
    "\n",
    "emmental_learner = EmmentalLearner()\n",
    "emmental_learner.learn(model, [train_dataloader])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating on the Test Set\n",
    "In this final section, we'll get the score we've been after: the performance of the extraction model on the blind test set (split 2). First, we load the test set labels and gold candidates from earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from wiki_table_utils import entity_level_f1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we score using the discriminitive model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate dataloader for test data\n",
    "test_dataloader = EmmentalDataLoader(\n",
    "    task_to_label_dict={ATTRIBUTE: \"labels\"},\n",
    "    dataset=FonduerDataset(\n",
    "        ATTRIBUTE, test_cands[0], F_test[0], emb_layer.word2id, 2\n",
    "    ),\n",
    "    split=\"test\",\n",
    "    batch_size=100,\n",
    "    shuffle=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gold_file = \"data/president_tutorial_gold.csv\"\n",
    "\n",
    "test_preds = model.predict(test_dataloader, return_preds=True)\n",
    "positive = np.where(np.array(test_preds[\"probs\"][ATTRIBUTE])[:, TRUE] > 0.6)\n",
    "true_pred = [test_cands[0][_] for _ in positive[0]]\n",
    "%time (TP, FP, FN) = entity_level_f1(true_pred, gold_file, test_docs)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}