NER-BiLSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition with PyTorch\n",
    "\n",
    "In this notebook we'll explore how we can use Deep Learning for sequence labelling tasks such as part-of-speech tagging or named entity recognition. We won't focus on getting state-of-the-art accuracy, but rather, implement a first neural network to get the main concepts across."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "For our experiments we'll reuse the NER data we've already used for our CRF experiments. The Dutch CoNLL-2002 data has four kinds of named entities (people, locations, organizations and miscellaneous entities) and comes split into a training, development and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "train_sents = list(nltk.corpus.conll2002.iob_sents('ned.train'))\n",
    "dev_sents = list(nltk.corpus.conll2002.iob_sents('ned.testa'))\n",
    "test_sents = list(nltk.corpus.conll2002.iob_sents('ned.testb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15806, 2895, 5195)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_sents), len(dev_sents), len(test_sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('De', 'Art', 'O'),\n",
       " ('tekst', 'N', 'O'),\n",
       " ('van', 'Prep', 'O'),\n",
       " ('het', 'Art', 'O'),\n",
       " ('arrest', 'N', 'O'),\n",
       " ('is', 'V', 'O'),\n",
       " ('nog', 'Adv', 'O'),\n",
       " ('niet', 'Adv', 'O'),\n",
       " ('schriftelijk', 'Adj', 'O'),\n",
       " ('beschikbaar', 'Adj', 'O'),\n",
       " ('maar', 'Conj', 'O'),\n",
       " ('het', 'Art', 'O'),\n",
       " ('bericht', 'N', 'O'),\n",
       " ('werd', 'V', 'O'),\n",
       " ('alvast', 'Adv', 'O'),\n",
       " ('bekendgemaakt', 'V', 'O'),\n",
       " ('door', 'Prep', 'O'),\n",
       " ('een', 'Art', 'O'),\n",
       " ('communicatiebureau', 'N', 'O'),\n",
       " ('dat', 'Conj', 'O'),\n",
       " ('Floralux', 'N', 'B-ORG'),\n",
       " ('inhuurde', 'V', 'O'),\n",
       " ('.', 'Punc', 'O')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_sents[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to preprocess the data. For this we use the torchtext Python library, which has a number of handy utilities for preprocessing natural language. We process our data to a Dataset that consists of Examples. Each of these examples has two fields: a text field and a label field. Both contain sequential information (the sequence of tokens, and the sequence of labels). We don't have to tokenize this information anymore, as the CONLL data has already been tokenized for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data import Example\n",
    "from torchtext.data import Field, Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = Field(sequential= True, tokenize=lambda x:x, include_lengths=True)\n",
    "LABEL = Field(sequential=True, tokenize=lambda x:x, is_target=True)\n",
    "\n",
    "def read_data(sents):\n",
    "    examples = []\n",
    "    fields = [('labels', LABEL),('text', TEXT)]\n",
    "    \n",
    "    for sent in sents:\n",
    "        tokens = [t[0] for t in sent]\n",
    "        labels = [t[2] for t in sent]\n",
    "        examples.append(Example.fromlist([labels, tokens], fields))\n",
    "        \n",
    "    return Dataset(examples, fields)\n",
    "\n",
    "train_data = read_data(train_sents)\n",
    "dev_data = read_data(dev_sents)\n",
    "test_data = read_data(test_sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'labels': <torchtext.data.field.Field object at 0x124134898>, 'text': <torchtext.data.field.Field object at 0x124134198>}\n",
      "['De', 'tekst', 'van', 'het', 'arrest', 'is', 'nog', 'niet', 'schriftelijk', 'beschikbaar', 'maar', 'het', 'bericht', 'werd', 'alvast', 'bekendgemaakt', 'door', 'een', 'communicatiebureau', 'dat', 'Floralux', 'inhuurde', '.']\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-ORG', 'O', 'O']\n",
      "Train: 15806\n",
      "Dev: 2895\n",
      "Test: 5195\n"
     ]
    }
   ],
   "source": [
    "print(train_data.fields)\n",
    "print(train_data[0].text)\n",
    "print(train_data[0].labels)\n",
    "\n",
    "print(\"Train:\", len(train_data))\n",
    "print(\"Dev:\", len(dev_data))\n",
    "print(\"Test:\", len(test_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = 2000\n",
    "\n",
    "TEXT.build_vocab(train_data, max_size=VOCAB_SIZE)\n",
    "LABEL.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data import BucketIterator\n",
    "\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "train_iter = BucketIterator(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True, \n",
    "                            sort_key=lambda x: len(x.text), sort_within_batch=True)\n",
    "dev_iter = BucketIterator(dataset=dev_data, batch_size=BATCH_SIZE, \n",
    "                          sort_key=lambda x: len(x.text), sort_within_batch=True)\n",
    "test_iter = BucketIterator(dataset=test_data, batch_size=BATCH_SIZE, \n",
    "                           sort_key=lambda x: len(x.text), sort_within_batch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained embeddings\n",
    "\n",
    "Pre-trained embeddings embeddings are generally an easy way of improving the performance of your model, particularly if you have little training data. Thanks to these embeddings, you'll be able to make use of knowledge about the meaning and use of the words in your dataset that was learned from another, typically larger data set. In this way, your model will be able to generalize better between semantically related words.\n",
    "In this example, we make use of the popular FastText embeddings. These are high-quality pre-trained word embeddings that are available for a wide variety of languages. After downloading the vec file with the embeddings, we use them to initialize our embedding matrix. We do this by creating a matrix filled with zeros whose number of rows equals the number of words in our vocabulary and whose number of columns equals the number of dimensions in the FastText vectors (300). We have to take care that we insert the FastText embedding for a particular word in the correct row. This is the row whose index corresponds to the index of the word in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import os\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing embedding matrix\n"
     ]
    }
   ],
   "source": [
    "EMBEDDING_PATH = \"data/embeddings/wiki-news-300d-1M.vec\"\n",
    "\n",
    "def load_embeddings(path):\n",
    "    \"\"\" Load the FastText embeddings from the embedding file. \"\"\"\n",
    "    print(\"Loading pre-trained embeddings\")\n",
    "    \n",
    "    embedding_size = None\n",
    "    embeddings = {}\n",
    "    with open(path) as f:\n",
    "        for line in f:\n",
    "            if len(line) > 2:\n",
    "                line = line.strip().split()\n",
    "                word = line[0]\n",
    "                embedding_vec = np.array(line[1:])\n",
    "                embeddings[word] = embedding_vec\n",
    "    \n",
    "    embedding_size = embedding_vec.shape[0]\n",
    "    return embeddings, embedding_size\n",
    "\n",
    "def initialize_embeddings(embeddings, vocabs, embedding_size):\n",
    "    \"\"\" Use the pre-trained embeddings to initialize an embedding matrix. \"\"\"\n",
    "    print(\"Initializing embedding matrix\")\n",
    "    \n",
    "    embedding_matrix = np.zeros((len(vocabs), embedding_size))\n",
    "    \n",
    "    for idx, word in enumerate(vocabs.itos):\n",
    "        if word in embeddings:\n",
    "            embedding_matrix[idx,:] = embeddings[word]\n",
    "    \n",
    "    return embedding_matrix\n",
    "            \n",
    "    \n",
    "embeddings,embedding_size = load_embeddings(EMBEDDING_PATH)\n",
    "embedding_matrix = initialize_embeddings(embeddings, TEXT.vocab, embedding_size)\n",
    "embedding_matrix = torch.from_numpy(embedding_matrix).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2002, 300])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "Next, we create our BiLSTM model. It consists of four layers:\n",
    "\n",
    "- An embedding layer that maps one-hot word vectors to dense word embeddings. These embeddings are either pretrained or trained from scratch.\n",
    "- A bidirectional LSTM layer that reads the text both front to back and back to front. For each word, this LSTM produces two output vectors of dimensionality hidden_dim, which are concatenated to a vector of 2*hidden_dim.\n",
    "- A dropout layer that helps us avoid overfitting by dropping a certain percentage of the items in the LSTM output.\n",
    "- A dense layer that projects the LSTM output to an output vector with a dimensionality equal to the number of labels.\n",
    "\n",
    "We initialize these layers in the __init__ method, and put them together in the forward method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiLSTMTagger(nn.Module):\n",
    "    def __init__(self, embedding_dim, hidden_dim, vocab_size, output_size, embeddings=None):\n",
    "        super(BiLSTMTagger, self).__init__()\n",
    "        \n",
    "        if not embeddings:\n",
    "            self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        else:\n",
    "            self.embeddings = nn.Embedding(embeddings)\n",
    "        \n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True, num_layers=1)\n",
    "        self.dropout_layer = nn.Dropout(p=0.5)\n",
    "        self.hidden2tag = nn.Linear(2*hidden_dim, output_size)\n",
    "    \n",
    "    def forward(self, batch_text, batch_lengths):\n",
    "        embeddings = self.embeddings(batch_text)\n",
    "        packed_seqs = pack_padded_sequence(embeddings, batch_lengths)\n",
    "        lstm_output, _ = self.lstm(packed_seqs)\n",
    "        # (seq_len, batch, num_directions * hidden_size)\n",
    "        lstm_output, _ = pad_packed_sequence(lstm_output)\n",
    "        logits = self.hidden2tag(lstm_output)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "Then we need to train this model. This involves taking a number of decisions:\n",
    "\n",
    "- We pick a loss function (or criterion) to quantify how far away the model predictions are from the correct output. For multiclass tasks such as Named Entity Recognition, a standard loss function is the Cross-Entropy Loss, which here measures the difference between two multinomial probability distributions. PyTorch's CrossEntropyLoss does this by first applying a softmax to the last layer of the model to transform the output scores to probabilities, and then computing the cross-entropy between the predicted and correct probability distributions. The ignore_index parameter allows us to mask the padding items in the training data, so that these do not contribute to the loss. We also remove these masked items from the output afterwards, so they are not taken into account when we evaluate the model output.\n",
    "\n",
    "- Next, we need to choose an optimizer. For many NLP problems, the Adam optimizer is a good first choice. Adam is a variation of Stochastic Gradient Descent with several advantages: it maintains per-parameter learning rates and adapts these learning rates based on how quickly the values of a specific parameter are changing (or, how large its average gradient is).\n",
    "\n",
    "Then the actual training starts. This happens in several epochs. During each epoch, we show all of the training data to the network, in the batches produced by the BucketIterators we created above. Before we show the model a new batch, we set the gradients of the model to zero to avoid accumulating gradients across batches. Then we let the model make its predictions for the batch. We do this by taking the output, and finding out what label received the highest score, using the torch.max method. We then compute the loss with respect to the correct labels. loss.backward() then computes the gradients for all model parameters; optimizer.step() performs an optimization step.\n",
    "\n",
    "When we have shown all the training data in an epoch, we perform the precision, recall and F-score on the training data and development data. Note that we compute the loss for the development data, but we do not optimize the model with it. Whenever the F-score on the development data is better than before, we save the model. If the F-score is lower than the minimum F-score we've seen in the past few epochs (we call this number the patience), we stop training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(<bound method Vocab._default_unk_index of <torchtext.vocab.Vocab object at 0x123fb99b0>>,\n",
       "            {'<unk>': 0,\n",
       "             '<pad>': 1,\n",
       "             '.': 2,\n",
       "             'de': 3,\n",
       "             ',': 4,\n",
       "             'van': 5,\n",
       "             'een': 6,\n",
       "             'het': 7,\n",
       "             'en': 8,\n",
       "             'in': 9,\n",
       "             'dat': 10,\n",
       "             'is': 11,\n",
       "             '\"': 12,\n",
       "             'op': 13,\n",
       "             'De': 14,\n",
       "             'te': 15,\n",
       "             'zijn': 16,\n",
       "             ')': 17,\n",
       "             '(': 18,\n",
       "             'die': 19,\n",
       "             'met': 20,\n",
       "             \"'\": 21,\n",
       "             'voor': 22,\n",
       "             'niet': 23,\n",
       "             ':': 24,\n",
       "             'er': 25,\n",
       "             'aan': 26,\n",
       "             'om': 27,\n",
       "             'als': 28,\n",
       "             '--': 29,\n",
       "             'Het': 30,\n",
       "             'ook': 31,\n",
       "             'hij': 32,\n",
       "             'maar': 33,\n",
       "             '-': 34,\n",
       "             'was': 35,\n",
       "             'ik': 36,\n",
       "             'dan': 37,\n",
       "             'ze': 38,\n",
       "             'naar': 39,\n",
       "             'door': 40,\n",
       "             'uit': 41,\n",
       "             'nog': 42,\n",
       "             'of': 43,\n",
       "             'over': 44,\n",
       "             'bij': 45,\n",
       "             'worden': 46,\n",
       "             'je': 47,\n",
       "             'heeft': 48,\n",
       "             '1': 49,\n",
       "             'al': 50,\n",
       "             'tot': 51,\n",
       "             'jaar': 52,\n",
       "             '?': 53,\n",
       "             '6': 54,\n",
       "             'wordt': 55,\n",
       "             'hebben': 56,\n",
       "             'geen': 57,\n",
       "             'we': 58,\n",
       "             'meer': 59,\n",
       "             'zich': 60,\n",
       "             'In': 61,\n",
       "             'wat': 62,\n",
       "             'werd': 63,\n",
       "             '2000': 64,\n",
       "             '/': 65,\n",
       "             'wel': 66,\n",
       "             '2': 67,\n",
       "             'Morgen': 68,\n",
       "             'Ik': 69,\n",
       "             'hun': 70,\n",
       "             'Maar': 71,\n",
       "             'kan': 72,\n",
       "             'Algemeen': 73,\n",
       "             'Een': 74,\n",
       "             'zo': 75,\n",
       "             'auteur': 76,\n",
       "             'pagina': 77,\n",
       "             'editie': 78,\n",
       "             'publicatie': 79,\n",
       "             'Dat': 80,\n",
       "             'publicatiedatum': 81,\n",
       "             'sectie': 82,\n",
       "             'kunnen': 83,\n",
       "             'nu': 84,\n",
       "             'zou': 85,\n",
       "             'moet': 86,\n",
       "             '3': 87,\n",
       "             'veel': 88,\n",
       "             'deze': 89,\n",
       "             'En': 90,\n",
       "             'twee': 91,\n",
       "             'zal': 92,\n",
       "             'had': 93,\n",
       "             'andere': 94,\n",
       "             'dit': 95,\n",
       "             'haar': 96,\n",
       "             'tegen': 97,\n",
       "             'na': 98,\n",
       "             'juli': 99,\n",
       "             'onder': 100,\n",
       "             'Van': 101,\n",
       "             'mijn': 102,\n",
       "             'nieuwe': 103,\n",
       "             'waar': 104,\n",
       "             'hem': 105,\n",
       "             'Fra': 106,\n",
       "             'zaterdag': 107,\n",
       "             'mensen': 108,\n",
       "             'moeten': 109,\n",
       "             'eerste': 110,\n",
       "             '7': 111,\n",
       "             'ons': 112,\n",
       "             'goed': 113,\n",
       "             'wil': 114,\n",
       "             'grote': 115,\n",
       "             'Hij': 116,\n",
       "             'toch': 117,\n",
       "             'daar': 118,\n",
       "             'gaan': 119,\n",
       "             'We': 120,\n",
       "             'maken': 121,\n",
       "             'werden': 122,\n",
       "             'gaat': 123,\n",
       "             'doen': 124,\n",
       "             'heb': 125,\n",
       "             'u': 126,\n",
       "             'tussen': 127,\n",
       "             'dus': 128,\n",
       "             'juni': 129,\n",
       "             'alle': 130,\n",
       "             'waren': 131,\n",
       "             'Op': 132,\n",
       "             'zegt': 133,\n",
       "             '4': 134,\n",
       "             'zoals': 135,\n",
       "             'procent': 136,\n",
       "             'zelf': 137,\n",
       "             'Brussel': 138,\n",
       "             'omdat': 139,\n",
       "             'onze': 140,\n",
       "             'Als': 141,\n",
       "             'vooral': 142,\n",
       "             'zelfs': 143,\n",
       "             'frank': 144,\n",
       "             'Er': 145,\n",
       "             'heel': 146,\n",
       "             'toen': 147,\n",
       "             'Ita': 148,\n",
       "             'drie': 149,\n",
       "             'alleen': 150,\n",
       "             'één': 151,\n",
       "             'vrijdag': 152,\n",
       "             'Ze': 153,\n",
       "             'weer': 154,\n",
       "             'staat': 155,\n",
       "             'altijd': 156,\n",
       "             'iets': 157,\n",
       "             'jaren': 158,\n",
       "             'komen': 159,\n",
       "             'me': 160,\n",
       "             'enkele': 161,\n",
       "             '...': 162,\n",
       "             'Ook': 163,\n",
       "             'Voor': 164,\n",
       "             'hele': 165,\n",
       "             'mij': 166,\n",
       "             'Spa': 167,\n",
       "             'Met': 168,\n",
       "             'af': 169,\n",
       "             'eens': 170,\n",
       "             'even': 171,\n",
       "             'komt': 172,\n",
       "             '5': 173,\n",
       "             'Die': 174,\n",
       "             'kinderen': 175,\n",
       "             'kon': 176,\n",
       "             'eigen': 177,\n",
       "             'krijgen': 178,\n",
       "             'zullen': 179,\n",
       "             'mee': 180,\n",
       "             'tijdens': 181,\n",
       "             'werk': 182,\n",
       "             'leven': 183,\n",
       "             'plaats': 184,\n",
       "             'hadden': 185,\n",
       "             'toe': 186,\n",
       "             'Wat': 187,\n",
       "             'keer': 188,\n",
       "             'lang': 189,\n",
       "             'miljoen': 190,\n",
       "             'willen': 191,\n",
       "             'laatste': 192,\n",
       "             'zij': 193,\n",
       "             'België': 194,\n",
       "             'tijd': 195,\n",
       "             'tweede': 196,\n",
       "             'Vlaamse': 197,\n",
       "             'hoe': 198,\n",
       "             'man': 199,\n",
       "             'hier': 200,\n",
       "             '!': 201,\n",
       "             'nooit': 202,\n",
       "             'wereld': 203,\n",
       "             \"zo'n\": 204,\n",
       "             'wie': 205,\n",
       "             'men': 206,\n",
       "             'minder': 207,\n",
       "             'steeds': 208,\n",
       "             'niets': 209,\n",
       "             'zouden': 210,\n",
       "             'gisteren': 211,\n",
       "             'kwam': 212,\n",
       "             'land': 213,\n",
       "             'zeggen': 214,\n",
       "             'achter': 215,\n",
       "             'per': 216,\n",
       "             'uur': 217,\n",
       "             'Belgische': 218,\n",
       "             'alles': 219,\n",
       "             'blijft': 220,\n",
       "             'echt': 221,\n",
       "             'staan': 222,\n",
       "             'zonder': 223,\n",
       "             'ben': 224,\n",
       "             'binnen': 225,\n",
       "             'want': 226,\n",
       "             'dag': 227,\n",
       "             'hen': 228,\n",
       "             'maakt': 229,\n",
       "             'manier': 230,\n",
       "             'Zo': 231,\n",
       "             'blijven': 232,\n",
       "             'erg': 233,\n",
       "             'geleden': 234,\n",
       "             'beter': 235,\n",
       "             'geval': 236,\n",
       "             'laat': 237,\n",
       "             'zien': 238,\n",
       "             'groot': 239,\n",
       "             'weet': 240,\n",
       "             'Bij': 241,\n",
       "             'net': 242,\n",
       "             'week': 243,\n",
       "             '10': 244,\n",
       "             'Franse': 245,\n",
       "             'Zijn': 246,\n",
       "             'anders': 247,\n",
       "             'later': 248,\n",
       "             'mag': 249,\n",
       "             'terug': 250,\n",
       "             'zeker': 251,\n",
       "             'rond': 252,\n",
       "             'Volgens': 253,\n",
       "             'geld': 254,\n",
       "             'laten': 255,\n",
       "             'samen': 256,\n",
       "             'volgens': 257,\n",
       "             'waarin': 258,\n",
       "             'werken': 259,\n",
       "             '15': 260,\n",
       "             'ging': 261,\n",
       "             'krijgt': 262,\n",
       "             'eigenlijk': 263,\n",
       "             'enige': 264,\n",
       "             'gemaakt': 265,\n",
       "             'weg': 266,\n",
       "             'aantal': 267,\n",
       "             'bijvoorbeeld': 268,\n",
       "             'Frankrijk': 269,\n",
       "             'Ned': 270,\n",
       "             'goede': 271,\n",
       "             'regering': 272,\n",
       "             'Europese': 273,\n",
       "             'Wie': 274,\n",
       "             'doet': 275,\n",
       "             'nemen': 276,\n",
       "             'oude': 277,\n",
       "             'ten': 278,\n",
       "             'ter': 279,\n",
       "             'zeer': 280,\n",
       "             'kleine': 281,\n",
       "             'moest': 282,\n",
       "             'snel': 283,\n",
       "             'vier': 284,\n",
       "             'vorig': 285,\n",
       "             'Je': 286,\n",
       "             'Toen': 287,\n",
       "             'bijna': 288,\n",
       "             'elkaar': 289,\n",
       "             'kreeg': 290,\n",
       "             'maanden': 291,\n",
       "             'opnieuw': 292,\n",
       "             'vijf': 293,\n",
       "             'zei': 294,\n",
       "             'echter': 295,\n",
       "             'jonge': 296,\n",
       "             'nodig': 297,\n",
       "             'vaak': 298,\n",
       "             'vraag': 299,\n",
       "             'dagen': 300,\n",
       "             'geven': 301,\n",
       "             'meteen': 302,\n",
       "             'minister': 303,\n",
       "             'via': 304,\n",
       "             'weinig': 305,\n",
       "             'Antwerpen': 306,\n",
       "             'bedrijf': 307,\n",
       "             'eerst': 308,\n",
       "             'houden': 309,\n",
       "             'lange': 310,\n",
       "             'zit': 311,\n",
       "             'duidelijk': 312,\n",
       "             'fr.': 313,\n",
       "             'ligt': 314,\n",
       "             'stad': 315,\n",
       "             'verschillende': 316,\n",
       "             'geweest': 317,\n",
       "             'misschien': 318,\n",
       "             'mogelijk': 319,\n",
       "             'Amerikaanse': 320,\n",
       "             'Sport': 321,\n",
       "             'gedaan': 322,\n",
       "             'hand': 323,\n",
       "             'kans': 324,\n",
       "             'Nederland': 325,\n",
       "             'allemaal': 326,\n",
       "             '&': 327,\n",
       "             '20': 328,\n",
       "             'Tour': 329,\n",
       "             'enkel': 330,\n",
       "             'huis': 331,\n",
       "             'wij': 332,\n",
       "             'Eigen': 333,\n",
       "             'deel': 334,\n",
       "             'elke': 335,\n",
       "             'kunt': 336,\n",
       "             'probleem': 337,\n",
       "             'tel.': 338,\n",
       "             'Euro': 339,\n",
       "             'Vlaams': 340,\n",
       "             'gewoon': 341,\n",
       "             'lijkt': 342,\n",
       "             'onderzoek': 343,\n",
       "             'weken': 344,\n",
       "             '0': 345,\n",
       "             '07': 346,\n",
       "             'Dit': 347,\n",
       "             'Dui': 348,\n",
       "             'Vlaanderen': 349,\n",
       "             'beetje': 350,\n",
       "             'elk': 351,\n",
       "             'geeft': 352,\n",
       "             'vinden': 353,\n",
       "             'weten': 354,\n",
       "             'Dan': 355,\n",
       "             'Deze': 356,\n",
       "             'Leven': 357,\n",
       "             'berichtgeving': 358,\n",
       "             'brengen': 359,\n",
       "             'euro': 360,\n",
       "             'finale': 361,\n",
       "             'hoofd': 362,\n",
       "             'liet': 363,\n",
       "             'maakte': 364,\n",
       "             'waarop': 365,\n",
       "             'beste': 366,\n",
       "             'kijken': 367,\n",
       "             'soort': 368,\n",
       "             'stuk': 369,\n",
       "             'tien': 370,\n",
       "             'vandaag': 371,\n",
       "             'wedstrijd': 372,\n",
       "             'Hoe': 373,\n",
       "             'immers': 374,\n",
       "             'miljard': 375,\n",
       "             'moment': 376,\n",
       "             'naam': 377,\n",
       "             'paar': 378,\n",
       "             'pas': 379,\n",
       "             'vele': 380,\n",
       "             'vrouwen': 381,\n",
       "             '8': 382,\n",
       "             'Bovendien': 383,\n",
       "             'Europa': 384,\n",
       "             'derde': 385,\n",
       "             'groep': 386,\n",
       "             'vorige': 387,\n",
       "             'vrij': 388,\n",
       "             'zon': 389,\n",
       "             'zoveel': 390,\n",
       "             'Na': 391,\n",
       "             'Om': 392,\n",
       "             'der': 393,\n",
       "             'politieke': 394,\n",
       "             'volgende': 395,\n",
       "             'Brusselse': 396,\n",
       "             'Daar': 397,\n",
       "             'Nederlandse': 398,\n",
       "             'iedereen': 399,\n",
       "             'vragen': 400,\n",
       "             'wanneer': 401,\n",
       "             'zaak': 402,\n",
       "             'Radio': 403,\n",
       "             'ander': 404,\n",
       "             'bekend': 405,\n",
       "             'best': 406,\n",
       "             'graag': 407,\n",
       "             'km': 408,\n",
       "             'problemen': 409,\n",
       "             'stond': 410,\n",
       "             'verder': 411,\n",
       "             'wilde': 412,\n",
       "             'zodat': 413,\n",
       "             'Jan': 414,\n",
       "             'genoeg': 415,\n",
       "             'helemaal': 416,\n",
       "             'moeilijk': 417,\n",
       "             'sinds': 418,\n",
       "             'slechts': 419,\n",
       "             '14': 420,\n",
       "             '9': 421,\n",
       "             'Britse': 422,\n",
       "             'familie': 423,\n",
       "             'gaf': 424,\n",
       "             'konden': 425,\n",
       "             'naast': 426,\n",
       "             'natuurlijk': 427,\n",
       "             'stellen': 428,\n",
       "             '12': 429,\n",
       "             'Frank': 430,\n",
       "             'Italië': 431,\n",
       "             'New': 432,\n",
       "             'Niet': 433,\n",
       "             'Nu': 434,\n",
       "             'VS': 435,\n",
       "             'begin': 436,\n",
       "             'eerder': 437,\n",
       "             'gebruikt': 438,\n",
       "             'informatie': 439,\n",
       "             'muziek': 440,\n",
       "             'ooit': 441,\n",
       "             'spelen': 442,\n",
       "             'terwijl': 443,\n",
       "             'vast': 444,\n",
       "             'water': 445,\n",
       "             '18': 446,\n",
       "             'Michel': 447,\n",
       "             'bedrijven': 448,\n",
       "             'bestaat': 449,\n",
       "             'boek': 450,\n",
       "             'geworden': 451,\n",
       "             'idee': 452,\n",
       "             'Armstrong': 453,\n",
       "             'Door': 454,\n",
       "             'Gent': 455,\n",
       "             'Marc': 456,\n",
       "             'begon': 457,\n",
       "             'blijkt': 458,\n",
       "             'grootste': 459,\n",
       "             'open': 460,\n",
       "             'zoek': 461,\n",
       "             'zowel': 462,\n",
       "             '13': 463,\n",
       "             'Bart': 464,\n",
       "             'Ronde': 465,\n",
       "             'Zij': 466,\n",
       "             'beeld': 467,\n",
       "             'beide': 468,\n",
       "             'bestaan': 469,\n",
       "             'bleek': 470,\n",
       "             'boven': 471,\n",
       "             'houdt': 472,\n",
       "             'iemand': 473,\n",
       "             'internationale': 474,\n",
       "             'meter': 475,\n",
       "             'mogen': 476,\n",
       "             'ogen': 477,\n",
       "             'ploeg': 478,\n",
       "             'ronde': 479,\n",
       "             'valt': 480,\n",
       "             'vindt': 481,\n",
       "             'zitten': 482,\n",
       "             '16': 483,\n",
       "             'Financiële': 484,\n",
       "             'The': 485,\n",
       "             'Toch': 486,\n",
       "             'gezegd': 487,\n",
       "             'hoge': 488,\n",
       "             'landen': 489,\n",
       "             'meeste': 490,\n",
       "             'moesten': 491,\n",
       "             'publiek': 492,\n",
       "             'vlak': 493,\n",
       "             'vol': 494,\n",
       "             'waardoor': 495,\n",
       "             'zes': 496,\n",
       "             'ziet': 497,\n",
       "             \"'s\": 498,\n",
       "             '17': 499,\n",
       "             '19': 500,\n",
       "             'Of': 501,\n",
       "             'VLD': 502,\n",
       "             'deed': 503,\n",
       "             'feit': 504,\n",
       "             'langer': 505,\n",
       "             'niemand': 506,\n",
       "             'nummer': 507,\n",
       "             'politie': 508,\n",
       "             'programma': 509,\n",
       "             'spelers': 510,\n",
       "             'spreken': 511,\n",
       "             'terecht': 512,\n",
       "             'vorm': 513,\n",
       "             'vrouw': 514,\n",
       "             'woensdag': 515,\n",
       "             'Paul': 516,\n",
       "             'einde': 517,\n",
       "             'film': 518,\n",
       "             'gevoel': 519,\n",
       "             'keuze': 520,\n",
       "             'kwamen': 521,\n",
       "             'nadat': 522,\n",
       "             'partij': 523,\n",
       "             'periode': 524,\n",
       "             'wegens': 525,\n",
       "             '11': 526,\n",
       "             '30': 527,\n",
       "             'Fuller': 528,\n",
       "             'Geen': 529,\n",
       "             'Grauwe': 530,\n",
       "             'Italiaanse': 531,\n",
       "             'Over': 532,\n",
       "             'Want': 533,\n",
       "             'aandacht': 534,\n",
       "             'acht': 535,\n",
       "             'eten': 536,\n",
       "             'licht': 537,\n",
       "             'minuten': 538,\n",
       "             'overheid': 539,\n",
       "             'sommige': 540,\n",
       "             'toekomst': 541,\n",
       "             'verhaal': 542,\n",
       "             'vond': 543,\n",
       "             'winnen': 544,\n",
       "             '25': 545,\n",
       "             'Aan': 546,\n",
       "             'Kongo': 547,\n",
       "             'aldus': 548,\n",
       "             'anderen': 549,\n",
       "             'basis': 550,\n",
       "             'bijzonder': 551,\n",
       "             'dingen': 552,\n",
       "             'echte': 553,\n",
       "             'gebouw': 554,\n",
       "             'gehad': 555,\n",
       "             'genomen': 556,\n",
       "             'gezien': 557,\n",
       "             'kilometer': 558,\n",
       "             'kregen': 559,\n",
       "             'moeder': 560,\n",
       "             'sterk': 561,\n",
       "             'tentoonstelling': 562,\n",
       "             'thuis': 563,\n",
       "             'vanuit': 564,\n",
       "             'volgen': 565,\n",
       "             'vroeg': 566,\n",
       "             'woord': 567,\n",
       "             'Al': 568,\n",
       "             'Barragán': 569,\n",
       "             'Duitsland': 570,\n",
       "             'Televisie': 571,\n",
       "             'Wij': 572,\n",
       "             'Zwi': 573,\n",
       "             'gevonden': 574,\n",
       "             'kant': 575,\n",
       "             'maatregelen': 576,\n",
       "             'nauwelijks': 577,\n",
       "             'onderwijs': 578,\n",
       "             'proberen': 579,\n",
       "             'vallen': 580,\n",
       "             'volledig': 581,\n",
       "             'werkt': 582,\n",
       "             'zaken': 583,\n",
       "             'Misschien': 584,\n",
       "             'begint': 585,\n",
       "             'beroep': 586,\n",
       "             'beslissing': 587,\n",
       "             'bleef': 588,\n",
       "             'brengt': 589,\n",
       "             'den': 590,\n",
       "             'denk': 591,\n",
       "             'des': 592,\n",
       "             'dezelfde': 593,\n",
       "             'hoog': 594,\n",
       "             'jongeren': 595,\n",
       "             'kun': 596,\n",
       "             'momenteel': 597,\n",
       "             'neemt': 598,\n",
       "             'ongeveer': 599,\n",
       "             'president': 600,\n",
       "             'rol': 601,\n",
       "             'soms': 602,\n",
       "             'stelt': 603,\n",
       "             'vroeger': 604,\n",
       "             'wist': 605,\n",
       "             'zag': 606,\n",
       "             'zoeken': 607,\n",
       "             '+': 608,\n",
       "             'Brugge': 609,\n",
       "             'Duitse': 610,\n",
       "             'Luc': 611,\n",
       "             'Tijdens': 612,\n",
       "             'Vandenbroucke': 613,\n",
       "             'Vriens': 614,\n",
       "             'burgemeester': 615,\n",
       "             'eeuw': 616,\n",
       "             'ervan': 617,\n",
       "             'gebruiken': 618,\n",
       "             'grond': 619,\n",
       "             'kind': 620,\n",
       "             'klein': 621,\n",
       "             'maand': 622,\n",
       "             'plaatsen': 623,\n",
       "             'precies': 624,\n",
       "             'rit': 625,\n",
       "             'trouwens': 626,\n",
       "             'waarmee': 627,\n",
       "             'waarom': 628,\n",
       "             'waarvan': 629,\n",
       "             'zetten': 630,\n",
       "             'zie': 631,\n",
       "             '1996': 632,\n",
       "             'Bruns': 633,\n",
       "             'IMF': 634,\n",
       "             'Nog': 635,\n",
       "             'Vorig': 636,\n",
       "             'Vrije': 637,\n",
       "             'York': 638,\n",
       "             'belangrijk': 639,\n",
       "             'belangrijkste': 640,\n",
       "             'buiten': 641,\n",
       "             'daarom': 642,\n",
       "             'druk': 643,\n",
       "             'eind': 644,\n",
       "             'extra': 645,\n",
       "             'gemeenten': 646,\n",
       "             'hard': 647,\n",
       "             'huidige': 648,\n",
       "             'komende': 649,\n",
       "             'liggen': 650,\n",
       "             'markt': 651,\n",
       "             'meestal': 652,\n",
       "             'nationale': 653,\n",
       "             'openbare': 654,\n",
       "             'oud': 655,\n",
       "             'ouders': 656,\n",
       "             'project': 657,\n",
       "             'totaal': 658,\n",
       "             'vader': 659,\n",
       "             'vanaf': 660,\n",
       "             'ver': 661,\n",
       "             'verleden': 662,\n",
       "             'vertrek': 663,\n",
       "             'witte': 664,\n",
       "             'zichzelf': 665,\n",
       "             'zorgen': 666,\n",
       "             'zwarte': 667,\n",
       "             '1999': 668,\n",
       "             '22': 669,\n",
       "             'Col': 670,\n",
       "             'La': 671,\n",
       "             'Martin': 672,\n",
       "             'Onze': 673,\n",
       "             'Strupar': 674,\n",
       "             'Tot': 675,\n",
       "             'Valentino': 676,\n",
       "             'Wimbledon': 677,\n",
       "             'augustus': 678,\n",
       "             'belangrijke': 679,\n",
       "             'betalen': 680,\n",
       "             'bezoek': 681,\n",
       "             'financiële': 682,\n",
       "             'gekregen': 683,\n",
       "             'hebt': 684,\n",
       "             'hetzelfde': 685,\n",
       "             'langs': 686,\n",
       "             'meest': 687,\n",
       "             'muur': 688,\n",
       "             'nieuw': 689,\n",
       "             'niveau': 690,\n",
       "             'regeling': 691,\n",
       "             'ruim': 692,\n",
       "             'stelde': 693,\n",
       "             'uiteindelijk': 694,\n",
       "             'verschil': 695,\n",
       "             'voetbal': 696,\n",
       "             'vormen': 697,\n",
       "             'werknemers': 698,\n",
       "             'woorden': 699,\n",
       "             'zeven': 700,\n",
       "             'zwaar': 701,\n",
       "             'John': 702,\n",
       "             'Laurent': 703,\n",
       "             'Museum': 704,\n",
       "             'Patrick': 705,\n",
       "             'Twee': 706,\n",
       "             'algemeen': 707,\n",
       "             'allerlei': 708,\n",
       "             'antwoord': 709,\n",
       "             'auto': 710,\n",
       "             'gelijk': 711,\n",
       "             'halen': 712,\n",
       "             'halve': 713,\n",
       "             'horen': 714,\n",
       "             'juist': 715,\n",
       "             'klanten': 716,\n",
       "             'mannen': 717,\n",
       "             'mooi': 718,\n",
       "             'ondertussen': 719,\n",
       "             'sociale': 720,\n",
       "             'systeem': 721,\n",
       "             'vertelt': 722,\n",
       "             'waarbij': 723,\n",
       "             'zat': 724,\n",
       "             'zin': 725,\n",
       "             'zomer': 726,\n",
       "             '24': 727,\n",
       "             '31': 728,\n",
       "             '50': 729,\n",
       "             'Alleen': 730,\n",
       "             'Frans': 731,\n",
       "             'Gucht': 732,\n",
       "             'Is': 733,\n",
       "             'Mexicaanse': 734,\n",
       "             'Mijn': 735,\n",
       "             'Nederlands': 736,\n",
       "             'VT4': 737,\n",
       "             'Veel': 738,\n",
       "             'amper': 739,\n",
       "             'beginnen': 740,\n",
       "             'bereiken': 741,\n",
       "             'betere': 742,\n",
       "             'bevolking': 743,\n",
       "             'bezig': 744,\n",
       "             'daarvoor': 745,\n",
       "             'groene': 746,\n",
       "             'hield': 747,\n",
       "             'kiezen': 748,\n",
       "             'leggen': 749,\n",
       "             'lokale': 750,\n",
       "             'm': 751,\n",
       "             'minstens': 752,\n",
       "             'namen': 753,\n",
       "             'oog': 754,\n",
       "             'oorlog': 755,\n",
       "             'pater': 756,\n",
       "             'reeks': 757,\n",
       "             'rest': 758,\n",
       "             'resultaat': 759,\n",
       "             'ruimte': 760,\n",
       "             'uw': 761,\n",
       "             'vakantie': 762,\n",
       "             'verloren': 763,\n",
       "             '%': 764,\n",
       "             '1998': 765,\n",
       "             'Bové': 766,\n",
       "             'Elke': 767,\n",
       "             'Geert': 768,\n",
       "             'Tent': 769,\n",
       "             'betrokken': 770,\n",
       "             'daarvan': 771,\n",
       "             'denken': 772,\n",
       "             'doel': 773,\n",
       "             'dollar': 774,\n",
       "             'donderdag': 775,\n",
       "             'duur': 776,\n",
       "             'ervaring': 777,\n",
       "             'genoemd': 778,\n",
       "             'honderd': 779,\n",
       "             'hoogte': 780,\n",
       "             'ieder': 781,\n",
       "             'kent': 782,\n",
       "             'kritiek': 783,\n",
       "             'kunstenaars': 784,\n",
       "             'la': 785,\n",
       "             'loopt': 786,\n",
       "             'raad': 787,\n",
       "             'recht': 788,\n",
       "             'rekenen': 789,\n",
       "             'rode': 790,\n",
       "             'rozen': 791,\n",
       "             'rust': 792,\n",
       "             'slecht': 793,\n",
       "             'sprake': 794,\n",
       "             'start': 795,\n",
       "             'tegenover': 796,\n",
       "             'vervolgens': 797,\n",
       "             'verwacht': 798,\n",
       "             'vind': 799,\n",
       "             'vrienden': 800,\n",
       "             'welke': 801,\n",
       "             'zet': 802,\n",
       "             'zondag': 803,\n",
       "             '1997': 804,\n",
       "             'Antwerpse': 805,\n",
       "             'Aula': 806,\n",
       "             'Bode': 807,\n",
       "             'Den': 808,\n",
       "             'Leo': 809,\n",
       "             'Lucien': 810,\n",
       "             'Omdat': 811,\n",
       "             'Rode': 812,\n",
       "             'Seghers': 813,\n",
       "             'Turkse': 814,\n",
       "             'architect': 815,\n",
       "             'bepaalde': 816,\n",
       "             'betekent': 817,\n",
       "             'betreft': 818,\n",
       "             'bondscoach': 819,\n",
       "             'bovendien': 820,\n",
       "             'buurt': 821,\n",
       "             'dood': 822,\n",
       "             'gebeurt': 823,\n",
       "             'gebrek': 824,\n",
       "             'gebruik': 825,\n",
       "             'gegeven': 826,\n",
       "             'gezet': 827,\n",
       "             'gingen': 828,\n",
       "             'handen': 829,\n",
       "             'heen': 830,\n",
       "             'indruk': 831,\n",
       "             'kennen': 832,\n",
       "             'leek': 833,\n",
       "             'leren': 834,\n",
       "             'liever': 835,\n",
       "             'lopen': 836,\n",
       "             'omgeving': 837,\n",
       "             'premier': 838,\n",
       "             'professor': 839,\n",
       "             'projecten': 840,\n",
       "             'reden': 841,\n",
       "             'rijden': 842,\n",
       "             'telkens': 843,\n",
       "             'twintig': 844,\n",
       "             'vijftig': 845,\n",
       "             'voldoende': 846,\n",
       "             'wijze': 847,\n",
       "             'zogenaamde': 848,\n",
       "             'zware': 849,\n",
       "             '100': 850,\n",
       "             'Belgen': 851,\n",
       "             'EK': 852,\n",
       "             'Eric': 853,\n",
       "             'Sportdirecteur': 854,\n",
       "             'Vooral': 855,\n",
       "             'Zaken': 856,\n",
       "             'Zoals': 857,\n",
       "             'afgelopen': 858,\n",
       "             'akkoord': 859,\n",
       "             'avond': 860,\n",
       "             'daarbij': 861,\n",
       "             'daarna': 862,\n",
       "             'delen': 863,\n",
       "             'dossier': 864,\n",
       "             'erop': 865,\n",
       "             'genieten': 866,\n",
       "             'geschiedenis': 867,\n",
       "             'gewone': 868,\n",
       "             'hart': 869,\n",
       "             'hoop': 870,\n",
       "             'jongen': 871,\n",
       "             'kerk': 872,\n",
       "             'kosten': 873,\n",
       "             'kunstenaar': 874,\n",
       "             'kwaliteit': 875,\n",
       "             'lezen': 876,\n",
       "             'lucht': 877,\n",
       "             'macht': 878,\n",
       "             'mond': 879,\n",
       "             'nogal': 880,\n",
       "             'perfect': 881,\n",
       "             'politici': 882,\n",
       "             'schreef': 883,\n",
       "             'stem': 884,\n",
       "             'the': 885,\n",
       "             'viel': 886,\n",
       "             'voorbije': 887,\n",
       "             'voorstel': 888,\n",
       "             'voortdurend': 889,\n",
       "             'vrije': 890,\n",
       "             'China': 891,\n",
       "             'Duivels': 892,\n",
       "             'Ja': 893,\n",
       "             'Johan': 894,\n",
       "             'Londen': 895,\n",
       "             'Lux': 896,\n",
       "             'Meer': 897,\n",
       "             'Robert': 898,\n",
       "             'Seattle': 899,\n",
       "             'Sommige': 900,\n",
       "             'Terwijl': 901,\n",
       "             'Zwe': 902,\n",
       "             'allochtonen': 903,\n",
       "             'alternatieve': 904,\n",
       "             'bloed': 905,\n",
       "             'buitenland': 906,\n",
       "             'contract': 907,\n",
       "             'crisis': 908,\n",
       "             'dankzij': 909,\n",
       "             'ene': 910,\n",
       "             'ervoor': 911,\n",
       "             'ga': 912,\n",
       "             'inderdaad': 913,\n",
       "             'internet': 914,\n",
       "             'ja': 915,\n",
       "             'klinkt': 916,\n",
       "             'lijst': 917,\n",
       "             'mooie': 918,\n",
       "             'negen': 919,\n",
       "             'politiek': 920,\n",
       "             'prijs': 921,\n",
       "             'radio': 922,\n",
       "             'samenleving': 923,\n",
       "             'sfeer': 924,\n",
       "             'speelde': 925,\n",
       "             'spel': 926,\n",
       "             'strafschoppen': 927,\n",
       "             'termijn': 928,\n",
       "             'trekken': 929,\n",
       "             'vergeten': 930,\n",
       "             'verlies': 931,\n",
       "             'vertrouwen': 932,\n",
       "             'volgend': 933,\n",
       "             'wou': 934,\n",
       "             'zeg': 935,\n",
       "             '02': 936,\n",
       "             'Arg': 937,\n",
       "             'Buitenland': 938,\n",
       "             'Cultuur': 939,\n",
       "             'Daarom': 940,\n",
       "             'Dus': 941,\n",
       "             'Eddy': 942,\n",
       "             'Elián': 943,\n",
       "             'Freddy': 944,\n",
       "             'Louis': 945,\n",
       "             'Rus': 946,\n",
       "             'VSt': 947,\n",
       "             'Verenigde': 948,\n",
       "             'WK': 949,\n",
       "             'Wel': 950,\n",
       "             'acteur': 951,\n",
       "             'bereikt': 952,\n",
       "             'commerciële': 953,\n",
       "             'cultuur': 954,\n",
       "             'dokter': 955,\n",
       "             'eveneens': 956,\n",
       "             'gasten': 957,\n",
       "             'gebied': 958,\n",
       "             'gehaald': 959,\n",
       "             'gehouden': 960,\n",
       "             'heet': 961,\n",
       "             'hoeft': 962,\n",
       "             'hof': 963,\n",
       "             'invloed': 964,\n",
       "             'kader': 965,\n",
       "             'koning': 966,\n",
       "             'leeftijd': 967,\n",
       "             'lichaam': 968,\n",
       "             'media': 969,\n",
       "             'mens': 970,\n",
       "             'nacht': 971,\n",
       "             'oplossing': 972,\n",
       "             'speciale': 973,\n",
       "             'televisie': 974,\n",
       "             'theater': 975,\n",
       "             'verkocht': 976,\n",
       "             'verliezen': 977,\n",
       "             'voordeel': 978,\n",
       "             'voorstellen': 979,\n",
       "             'waarschijnlijk': 980,\n",
       "             'ware': 981,\n",
       "             '2001': 982,\n",
       "             '27': 983,\n",
       "             'Agalev': 984,\n",
       "             'Buitenlandse': 985,\n",
       "             'Chinese': 986,\n",
       "             'Engeland': 987,\n",
       "             'Futuroscope': 988,\n",
       "             'Italianen': 989,\n",
       "             'Journaal': 990,\n",
       "             'Pas': 991,\n",
       "             'Staten': 992,\n",
       "             'Waseige': 993,\n",
       "             'Zelfs': 994,\n",
       "             'Zoff': 995,\n",
       "             'aandeel': 996,\n",
       "             'alsof': 997,\n",
       "             'as': 998,\n",
       "             'beleid': 999,\n",
       "             ...})"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEXT.vocab.stoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "from sklearn.metrics import precision_recall_fscore_support, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_predictions_for_masked_items(predicted_labels, correct_labels):\n",
    "    predicted_labels_without_mask = []\n",
    "    correct_labels_without_mask = []\n",
    "    for p, c in zip(predicted_labels, correct_labels):\n",
    "        if c > 1:\n",
    "            predicted_labels_without_mask.append(p)\n",
    "            correct_labels_without_mask.append(c)\n",
    "    return predicted_labels_without_mask, correct_labels_without_mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[torchtext.data.batch.Batch of size 32]\n",
      "\t[.labels]:[torch.LongTensor of size 28x32]\n",
      "\t[.text]:('[torch.LongTensor of size 28x32]', '[torch.LongTensor of size 32]')\n"
     ]
    }
   ],
   "source": [
    "for batch in train_iter:\n",
    "    print(batch)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([28, 32])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch.labels.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[  14,    0,   61,    0, 1957,    0, 1957,   90,    0,   74,    0,    0,\n",
       "           187,   80, 1583,  163,  120,   12,   90,  466,   80,  153,   30,   14,\n",
       "            71,   14,  116, 1552,   14,  116,  286,   12],\n",
       "         [   0,    0,    7,    4,    0,    0,    4,  399,  129,    0,  842,  206,\n",
       "            86,   35,    3,   10,   56,  116,   37,   56,  342,    0,    0,    0,\n",
       "            31,    0,  249,  172,  192,    0,  581,  572],\n",
       "         [ 521,   20,    0, 1245,    3,    0,   28,   19,  336,   24,   11,    3,\n",
       "            47,    6,    0,   48,    3,   48,   11,    6,    7,  130,    3,  212,\n",
       "             3,   63,   57,   25,  188,    3,    0,   56],\n",
       "         [ 284,    0,   55,    5, 1368,   85,   36,   28,  126,    0,   57, 1478,\n",
       "             4, 1051,  122,    5,  389, 2001,    7,    0,  366,  175,  199,  211,\n",
       "             0,    0,  177,  564,   35,  272,   26,   23],\n",
       "         [1540,  445,  335,    7,    3,  564,    9,    0,   39,    0,    0,    5,\n",
       "           268,    0, 1081,  569,  297,  427,  195,    4,    0,    8,   19,   24,\n",
       "             0,    4, 1354,    7,   10,    3,    3,    0],\n",
       "         [ 293,   18,    0,    0,    0,    3,  455,  169,    6,   29, 1855,    0,\n",
       "             4,   24,    0,    3,   22, 1713,  451,  974,   15,  595,  351,    0,\n",
       "            13,  139,   56,    0,    9,    0,  440,    0],\n",
       "         [ 217,   51,    0,   22,    8, 1476,   43,    8,  239,   14,    8,   84,\n",
       "           401,   42,    0, 1070,    3,    3,   22,    8,   16,  127, 1004,   40,\n",
       "          1034,   38,    8,    6,    0,    9,    4,   27],\n",
       "         [  98,    7,   20,    0,    3,    9,   75,  186,    0,    0,    7,    0,\n",
       "            47,  202,   29,  815,    0,    0,    3,    0,  522,  496,   20,    3,\n",
       "             8,    0,   50,  395,   97,    3, 1193,    7],\n",
       "         [   7,  242,    6,    5,    0,    0, 1352,  215,   20,    5,    0,    4,\n",
       "             9,  437,    3,  265,    5,  683,  110,    8,    3,    8,    3,    0,\n",
       "             3,   15,   16,    0,    3,    0,  189,  254],\n",
       "         [1468,  471, 1995, 1646,   41,    6,    4,    3,    0,    0,  283,   37,\n",
       "             7,   63,    0,   19,  112,   22,    0,   56,  474,    0,  969,    0,\n",
       "           720,   88,    0,   13,    0,   23, 1148,   22],\n",
       "         [   5,    3,    8,    8,   19, 1364,  591,    0,    4, 1568,    4,   85,\n",
       "           213,    6,  122,   32,    0,   16,   19,  820,    0,   52,    0, 1581,\n",
       "           643,  254,   46,  112,    0,    0,    8,    0],\n",
       "         [   3,    0,    6,    0,   31, 1096,   36,  155,   10,    9,   33,   38,\n",
       "           393, 1898,    0,   35,    8,    0,  112,   42,    0,   41,    4,    0,\n",
       "            27,   85,    0,  169,    4,   15,  208,    8],\n",
       "         [   0,  172,    0,    4,  414,   19,    0,    4,  275,    0,    7,  980,\n",
       "           197,    0,   20,   24,   22,    4,  564,    6,   65,   27,   16,    0,\n",
       "           505,  873,   40,    8,   33,    0,   59,    0],\n",
       "         [   4,   17,   19,  363,    0,  257,    4,  352,    0,    0,    0,    6,\n",
       "             0,    9,    0,   23,    3,    8,    0, 1181,    0,  334,  177,  522,\n",
       "            13,    4,    3,   19, 1572,   40,    0,   84],\n",
       "         [  62,    4,   45,    7,  182,    7,   11,    7,   26, 1942,  117,    5,\n",
       "             6,    0,    4,    0,    0,    7,    4,    9,   96,   15,    0, 1944,\n",
       "            15,   75,  272,    0,    0,   70,    9,    9],\n",
       "         [ 257,   20,    0,   25,   28,    0,    7,  186,    3,    8,    3,    3,\n",
       "             0,   25,  723,   40,    5,   11,    0,  914,    0,  276,    0, 1233,\n",
       "           232,  294,    4,    9, 1652,    3,    0,  829],\n",
       "         [   3,    0,   13,   23,  951, 1494,  200,   24,    0,    3, 1357,  640,\n",
       "          1390,    9,    3,    6,  140,    6,    0,  304,    0,   26,    0, 1028,\n",
       "            83,  340,   19,    3,   16,    0,    8,    5],\n",
       "         [   0,    0,    3,   45,   18,    0,  455,   12,   41,    0,    0,    4,\n",
       "             0,  155,  849,    0, 1624,  504,    4,    0,    5,    6,    8,   20,\n",
       "           911,  303,   13,    0,  478,    5,    6,    3],\n",
       "         [  15,    4,  103,  482,  100,    0,   43,   30,    3,    5,    4,   75,\n",
       "            22,    5,    0,  176,   18,   10,    3,   18,    6,   21,  142,   54,\n",
       "           666,    5,   96,   43,    9,    3,    0,    0],\n",
       "         [ 189,    0,    0,    8,   59,    0,   11,   11,    0,    3,    8,   23,\n",
       "           531,    0,    9,   32,    3,   32,    0,    3,  343,    0,  407,   34,\n",
       "            10,    0, 1984,  860,    3,    0,    0,   15],\n",
       "         [  11,   20,  843,    0,   45,    4,    7,   88,    8, 1581,   19,    3,\n",
       "          1095, 1086, 1081,   16,  553,    6,    5,    0,   45,    0,    9,  345,\n",
       "           175,    0,    0,  292,    0,    5,   11,  301],\n",
       "         [  27,    0,  292,  797,    7,  462,  200, 1638,    0,    0,  249,    0,\n",
       "             5,   22,    0,  177,    0,  369,    0,    0,    0,  279,    3,    4,\n",
       "           248,    0,    0,    6,    0,    3,    6,   12],\n",
       "         [  15,    8,   86,   31,  805,    0,    6, 1095,  616,    9,   13,    0,\n",
       "             3,    0,   97,    0,  507,  207,    4,    9,    0,    0, 1508,  134,\n",
       "           119,   18,   11,  267,    0,    0,  146,    4],\n",
       "         [ 511,  260,   46,  137,    0,    4,    0,   37,   26,    3,   57,    0,\n",
       "             0,    0,    7,  255,   49,    0,   39,    0,    9,    5,    5,   34,\n",
       "             0,  984,   26,    0,   39,  186, 1786,  548],\n",
       "         [   5,  538, 1670,    9,    0,    0,    9,    7,    3,    0,  330,   16,\n",
       "             8,  181,    0,    0,  961,   11,    0,    4,  919,    3,   16,   49,\n",
       "             4,   17,    3,   43,    3,   15,  777,    3],\n",
       "         [   6,  255,    8,    3,   17,   28, 1762,  342,  273,    5,  376,    5,\n",
       "             0,   16,   63,    8,    0,   37,   92,    0,  273,    0,  510,    0,\n",
       "           133,  211,    0,    0,  361,  932,    2,    0],\n",
       "         [ 868,    0,    0,    0,    0,    0,   53,    2,    0,  138,    0,  384,\n",
       "           616,    0,    0,    0,   17, 1436,  359,   17,  489,   21,  582,   93,\n",
       "            32,    2,    2,    2,    2,    2,   12,    2],\n",
       "         [   0,    2,    2,    2,    2,    2,   12,   12,    2,    2,    2,    2,\n",
       "            53,    2,    2,    2,    2,    2,    2,    2,    2,    2,    2,    2,\n",
       "             2,    1,    1,    1,    1,    1,    1,    1]]),\n",
       " tensor([28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28,\n",
       "         28, 28, 28, 28, 28, 28, 28, 27, 27, 27, 27, 27, 27, 27]))"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch.text[0], batch.text[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, train_iter, dev_iter, batch_size, max_epochs, num_batches, patience, output_path):\n",
    "    criterion = nn.CrossEntropyLoss(ignore_index=1)  # we mask the <pad> labels\n",
    "    optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "    train_f_score_history = []\n",
    "    dev_f_score_history = []\n",
    "    no_improvement = 0\n",
    "    for epoch in range(max_epochs):\n",
    "\n",
    "        total_loss = 0\n",
    "        predictions, correct = [], []\n",
    "        for batch in tqdm(train_iter, total=num_batches, desc=f\"Epoch {epoch}\"):\n",
    "            optimizer.zero_grad()\n",
    "            \n",
    "            text_length, cur_batch_size = batch.text[0].shape\n",
    "            \n",
    "            pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size*text_length, NUM_CLASSES)\n",
    "            gold = batch.labels.to(device).view(cur_batch_size*text_length)\n",
    "            \n",
    "            loss = criterion(pred, gold)\n",
    "            \n",
    "            total_loss += loss.item()\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            _, pred_indices = torch.max(pred, 1)\n",
    "            \n",
    "            predicted_labels = list(pred_indices.cpu().numpy())\n",
    "            correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())\n",
    "            \n",
    "            predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, \n",
    "                                                                                   correct_labels)\n",
    "            \n",
    "            predictions += predicted_labels\n",
    "            correct += correct_labels\n",
    "\n",
    "        train_scores = precision_recall_fscore_support(correct, predictions, average=\"micro\")\n",
    "        train_f_score_history.append(train_scores[2])\n",
    "            \n",
    "        print(\"Total training loss:\", total_loss)\n",
    "        print(\"Training performance:\", train_scores)\n",
    "        \n",
    "        total_loss = 0\n",
    "        predictions, correct = [], []\n",
    "        for batch in dev_iter:\n",
    "\n",
    "            text_length, cur_batch_size = batch.text[0].shape\n",
    "\n",
    "            pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size * text_length, NUM_CLASSES)\n",
    "            gold = batch.labels.to(device).view(cur_batch_size * text_length)\n",
    "            loss = criterion(pred, gold)\n",
    "            total_loss += loss.item()\n",
    "\n",
    "            _, pred_indices = torch.max(pred, 1)\n",
    "            predicted_labels = list(pred_indices.cpu().numpy())\n",
    "            correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())\n",
    "            \n",
    "            predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, \n",
    "                                                                                   correct_labels)\n",
    "            \n",
    "            predictions += predicted_labels\n",
    "            correct += correct_labels\n",
    "\n",
    "        dev_scores = precision_recall_fscore_support(correct, predictions, average=\"micro\")\n",
    "            \n",
    "        print(\"Total development loss:\", total_loss)\n",
    "        print(\"Development performance:\", dev_scores)\n",
    "        \n",
    "        dev_f = dev_scores[2]\n",
    "        if len(dev_f_score_history) > patience and dev_f < max(dev_f_score_history):\n",
    "            no_improvement += 1\n",
    "\n",
    "        elif len(dev_f_score_history) == 0 or dev_f > max(dev_f_score_history):\n",
    "            print(\"Saving model.\")\n",
    "            torch.save(model, output_path)\n",
    "            no_improvement = 0\n",
    "            \n",
    "        if no_improvement > patience:\n",
    "            print(\"Development F-score does not improve anymore. Stop training.\")\n",
    "            dev_f_score_history.append(dev_f)\n",
    "            break\n",
    "            \n",
    "        dev_f_score_history.append(dev_f)\n",
    "        \n",
    "    return train_f_score_history, dev_f_score_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we test the model, we basically take the same steps as in the evaluation on the development data above: we get the predictions, remove the masked items and print a classification report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, test_iter, batch_size, labels, target_names): \n",
    "    \n",
    "    total_loss = 0\n",
    "    predictions, correct = [], []\n",
    "    for batch in test_iter:\n",
    "\n",
    "        text_length, cur_batch_size = batch.text[0].shape\n",
    "\n",
    "        pred = model(batch.text[0].to(device), batch.text[1].to(device)).view(cur_batch_size * text_length, NUM_CLASSES)\n",
    "        gold = batch.labels.to(device).view(cur_batch_size * text_length)\n",
    "\n",
    "        _, pred_indices = torch.max(pred, 1)\n",
    "        predicted_labels = list(pred_indices.cpu().numpy())\n",
    "        correct_labels = list(batch.labels.view(cur_batch_size*text_length).numpy())\n",
    "\n",
    "        predicted_labels, correct_labels = remove_predictions_for_masked_items(predicted_labels, \n",
    "                                                                               correct_labels)\n",
    "\n",
    "        predictions += predicted_labels\n",
    "        correct += correct_labels\n",
    "    \n",
    "    print(classification_report(correct, predictions, labels=labels, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the actual training. We set the embedding dimension to 300 (the dimensionality of the FastText embeddings), and pick a hidden dimensionality for each component of the BiLSTM (which will therefore output 512-dimensional vectors). The number of classes (the length of the vocabulary of the label field) will become the dimensionality of the output layer. Finally, we also compute the number of batches in an epoch, so that we can show a progress bar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "EMBEDDING_DIM = 300\n",
    "HIDDEN_DIM = 256\n",
    "NUM_CLASSES = len(label_field.vocab)\n",
    "MAX_EPOCHS = 50\n",
    "PATIENCE = 3\n",
    "OUTPUT_PATH = \"/tmp/bilstmtagger\"\n",
    "num_batches = math.ceil(len(train_data) / BATCH_SIZE)\n",
    "\n",
    "tagger = BiLSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, VOCAB_SIZE+2, NUM_CLASSES, embeddings=embedding_matrix)  \n",
    "\n",
    "train_f, dev_f = train(tagger.to(device), train_iter, dev_iter, BATCH_SIZE, MAX_EPOCHS, \n",
    "                       num_batches, PATIENCE, OUTPUT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now plot the evolution of the F-score on our training and development set, to visually evaluate if training went well. If it did, the training F-score should first increase suddenly, then more gradually. The development F-score will increase during the first few epochs, but at some point it will start to decrease again. That's when the model starts overfitting. This is where we abandon training, and why we only save the model when we have reached an optimal F-score on the development data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "# Data\n",
    "df = pd.DataFrame({'epochs': range(0,len(train_f)), \n",
    "                  'train_f': train_f, \n",
    "                   'dev_f': dev_f})\n",
    " \n",
    "# multiple line plot\n",
    "plt.plot('epochs', 'train_f', data=df, color='blue', linewidth=2)\n",
    "plt.plot('epochs', 'dev_f', data=df, color='green', linewidth=2)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we test our model on the test data, we have to run its eval() method. This will put the model in eval mode, and deactivate dropout layers and other functionality that is only useful in training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger = torch.load(OUTPUT_PATH)\n",
    "tagger.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = label_field.vocab.itos[3:]\n",
    "labels = sorted(labels, key=lambda x: x.split(\"-\")[-1])\n",
    "label_idxs = [label_field.vocab.stoi[l] for l in labels]\n",
    "\n",
    "test(tagger, test_iter, BATCH_SIZE, labels = label_idxs, target_names = labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this notebook we've trained a simple bidirectional LSTM for named entity recognition. Far from achieving state-of-the-art performance, our aim was to understand how neural networks can be implemented and trained in PyTorch. To improve our performance, one of the things that is typically done is to add an additional CRF layer to the neural network. This layer helps us optimize the complete label sequence, and not the labels individually. We leave that for future work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "- https://github.com/nlptown/nlp-notebooks/blob/master/Sequence%20Labelling%20with%20a%20BiLSTM%20in%20PyTorch.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tryit",
   "language": "python",
   "name": "tryit"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}