COMP6714-2018s2-Proj-Stage1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COMP6714 18s2 Project\n",
    "\n",
    "# Stage 1: Implement a hyponymy classification model using BiLSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective\n",
    "In this project, you need to build a system that can extract [hyponym and hypernym](https://en.wikipedia.org/wiki/Hyponymy_and_hypernymy) from a sentence. \n",
    "\n",
    "For example, in sentence *Stephen Hawking is a physicist .*, phrase *Stephen Hawking* is the **hyponym** of *physicist*, and *physicist* is the **hypernym** of *Stephen Hawking*.\n",
    "\n",
    "We formulate the problem as a sequence tagging task where\n",
    " * Input is a sentence formed as a sequence of words with length $l$.\n",
    " * output is a tag sequence with length $l$. \n",
    " \n",
    "We use [IOB2](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)) scheme to represent the results (i.e., encode the tags), as both hypernyms and hyponyms can be phrases.\n",
    "\n",
    "Thus, in the above example,\n",
    " * `[Stephen, Hawking, is, a, physicist, .]` is the input word list \n",
    " * `[B-TAR, I-TAR, O, O, B-HYP, O]` is the corresponding output tag sequence, where `TAR` corresponds to hyponyms and `HYP` corresponds to hypernyms.\n",
    " \n",
    "You need to build and train a neural network model for this task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "### Description of given files\n",
    "+ `tags.txt` lists all the tags.\n",
    "+ `train.txt`, `dev.txt`, `test.txt` refers to training set, development set, and test set, individually. Each of them is in the CoNLL format, with each line having two columns for word and tag. Sentences are separated by blank lines.\n",
    "+ `word_embeddings.txt` is the pretrained word embedding file, with word in the first column and embeddings (a vector with 50-dimensions) in the rest columns.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate batches\n",
    "+ In order to speed up the training process, instead of pass the training sentences one by one to the model, we pass a batch of sentences at each iteration. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate tensors\n",
    "+ The first step is to convert each sentence into a list of word embeddings. In order to do that, we need two additional *words*, they are\n",
    "    - `<PAD>` is used to pad words to the end of sentences, such that all the sentence in the same batch have the same length. For example, if there are 5 sentences in a batch, with length `[3, 4, 4, 5, 5]`, then we need to pad two `<PAD>` for the first sentence and one `<PAD>` for the second and third sentences.\n",
    "    - `<UNK_WORD>` is used for those words that can not be found in word embedding.\n",
    "    - we offered the embedding of `<UNK_WORD>` in the embedding file. And `<PAD>` can be randomly initialled. It will not contribute to the loss function or the prediction, thus will not be updated anyway.\n",
    "+ Then for each batch, we can generate the corresponding **tensor** with size `[batch_size, max_sent_len, word_embedding_dim]`. Where `max_sent_len` is the maximum length among sentences in the batch.\n",
    "+ For tag sequence, We generate a **tensor** with size `[batch_size, max_sent_len]`, in which each element represents a tag_id. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Baseline BiLSTM Model\n",
    "\n",
    "+ In this project, you are required to use Bidirectional LSTM (BiLSTM) and SoftMax in your model.\n",
    "+ The input tensor will go through a *BiLSTM layer*, followed by a *softmax* function to determine the output tags.\n",
    "+ An example of the model is shown as follows:\n",
    "<img src=\"./fig/workflow.png\" width=\"25%\">."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "In the training process, in each epoch, you should feed the training data to the model batch by batch, and update the paramters accordingly. \n",
    "\n",
    "You should use ADAM as the optimizer for your model.\n",
    "\n",
    "If your implementation is correct, then your loss should converge within 80 epochs. Therefore the maximum number of epochs is set to 80.\n",
    "\n",
    "You can either take the model from the last epoch or the one with highest F1 score on the development set as the final model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helpful Resources\n",
    "\n",
    "* [PyTorch Documents](https://pytorch.org/docs/stable/index.html)\n",
    "* [Sequence Modelling Tutorial](https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 for the project\n",
    "\n",
    "In the next stage of the project, you will be given a baseline model based on the above description. You will need to firstly read and understand the implementation, and then finish the following 3 request:\n",
    "\n",
    "* implement a special LSTM cell and replace it with the default one\n",
    "* implement character embedding and plug it into the model\n",
    "* implement the evaluation function (i.e., compute F1 score) to help the training process with model selection.\n",
    "\n",
    "You are also expected to write a short report with the implementation details.\n",
    "\n",
    "Detailed specification of stage 2 will be posted later."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}