2_context_vectors.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2-context-vectors.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dpressel/dlss-tutorial/blob/master/2_context_vectors.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGiURKhLmONv",
        "colab_type": "text"
      },
      "source": [
        "# Part II: Contextualized embeddings\n",
        "\n",
        "\n",
        "In this section, we are going to learn how to train an LSTM-based word-level language model.  Then we will take load a pre-trained langage model checkpoint and use everything below the output layers as the lower layers of our previously defined classification model.  We dont really need to change anything else, we just need to pass this whole network as the `embedding` parameter to the model.\n",
        "\n",
        "## LSTM Language Models\n",
        "\n",
        "We are going to quickly build an LSTM language model so that we can see how the training works.  For both our objectives and our metrics, we are interested in the perplexity, which is the exponentiated cross-entropy loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PvcXdbNZq8NG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "78355358-4f8d-4a9a-a156-292a451e04d5"
      },
      "source": [
        "!wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip\n",
        "!unzip wikitext-2-v1.zip"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-06-30 19:10:48--  https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip\n",
            "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.134.253\n",
            "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.134.253|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4475746 (4.3M) [application/zip]\n",
            "Saving to: ‘wikitext-2-v1.zip.5’\n",
            "\n",
            "wikitext-2-v1.zip.5 100%[===================>]   4.27M  18.2MB/s    in 0.2s    \n",
            "\n",
            "2019-06-30 19:10:49 (18.2 MB/s) - ‘wikitext-2-v1.zip.5’ saved [4475746/4475746]\n",
            "\n",
            "Archive:  wikitext-2-v1.zip\n",
            "replace wikitext-2/wiki.test.tokens? [y]es, [n]o, [A]ll, [N]one, [r]ename: "
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNUhkQC_dzmR",
        "colab_type": "text"
      },
      "source": [
        "Our LSTM model will be a word-based model.  We will have a randomly trained embedding to start and we will put each output timestep through our LSTM blocks and then project to the output vocabulary size. At every step of training, we will detach our hidden states, preventing full backpropagation, but we will initialize the new batch from our old hidden state.  We will also create a function that resets the hidden state, which we will use at the start of each epoch to zero out the hidden states."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wl5CfAsAuaTu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from typing import List, Tuple\n",
        "import os\n",
        "import io\n",
        "import re\n",
        "import codecs\n",
        "import numpy as np\n",
        "from collections import Counter\n",
        "import math\n",
        "import time\n",
        "\n",
        "class LSTMLanguageModel(nn.Module):\n",
        "\n",
        "    def __init__(self, vocab_size, embed_dim, hidden_dim, dropout=0.5, layers=2):\n",
        "        super().__init__()\n",
        "        self.layers = layers\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.embed = nn.Embedding(vocab_size, embed_dim)\n",
        "        self.rnn = torch.nn.LSTM(embed_dim,\n",
        "                                 hidden_dim,\n",
        "                                 layers,\n",
        "                                 dropout=dropout,\n",
        "                                 bidirectional=False,\n",
        "                                 batch_first=True)\n",
        "        self.proj = nn.Linear(embed_dim, vocab_size)\n",
        "        self.proj.bias.data.zero_()\n",
        "\n",
        "        # Tie weights\n",
        "        self.proj.weight = self.embed.weight\n",
        "\n",
        "    def forward(self, x, hidden):\n",
        "        emb = self.embed(x)\n",
        "        decoded, hidden = self.rnn(emb, hidden)\n",
        "        return self.proj(decoded), hidden\n",
        "        \n",
        "    def init_hidden(self, batchsz):\n",
        "        weight = next(self.parameters()).data\n",
        "        return (torch.autograd.Variable(weight.new(self.layers, batchsz, self.hidden_dim).zero_()),\n",
        "                torch.autograd.Variable(weight.new(self.layers, batchsz, self.hidden_dim).zero_()))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_fJp1BpIe3gu",
        "colab_type": "text"
      },
      "source": [
        "Our dataset reader will read in a sequence of words and vectorize them.  We would like this to be a long sequence of text (like maybe a book), and we will read this in contiguously.  Our task is to learn to predict the next word, so we will end up using this sequence for input and output"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_C28CdeCrsj5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "class WordDatasetReader(object):\n",
        "    \"\"\"Provide a base-class to do operations to read words to tensors\n",
        "    \"\"\"\n",
        "    def __init__(self, nctx, vectorizer=None):\n",
        "        self.nctx = nctx\n",
        "        self.num_words = {}\n",
        "        self.vectorizer = vectorizer if vectorizer else self._vectorizer\n",
        "\n",
        "    def build_vocab(self, files, min_freq=0):\n",
        "        x = Counter()\n",
        "\n",
        "        for file in files:\n",
        "            if file is None:\n",
        "                continue\n",
        "            self.num_words[file] = 0\n",
        "            with codecs.open(file, encoding='utf-8', mode='r') as f:\n",
        "                sentences = []\n",
        "                for line in f:\n",
        "                    split_sentence = line.split() + ['<EOS>']\n",
        "                    self.num_words[file] += len(split_sentence)\n",
        "                    sentences += split_sentence\n",
        "                x.update(Counter(sentences))\n",
        "        x = dict(filter(lambda cnt: cnt[1] >= min_freq, x.items()))\n",
        "        alpha = list(x.keys())\n",
        "        alpha.sort()\n",
        "        self.vocab = {w: i+1 for i, w in enumerate(alpha)}\n",
        "        self.vocab['[PAD]'] = 0\n",
        "    \n",
        "    def _vectorizer(self, words: List[str]) -> List[int]:\n",
        "        return [self.vocab.get(w, 0) for w in words]\n",
        "\n",
        "      \n",
        "    def load_features(self, filename):\n",
        "\n",
        "        with codecs.open(filename, encoding='utf-8', mode='r') as f:\n",
        "            sentences = []\n",
        "            for line in f:\n",
        "                sentences += line.strip().split() + ['<EOS>']\n",
        "            return torch.tensor(self.vectorizer(sentences), dtype=torch.long)\n",
        "\n",
        "    def load(self, filename, batch_size):\n",
        "        x_tensor = self.load_features(filename)\n",
        "        rest = x_tensor.shape[0]//batch_size\n",
        "        num_steps = rest // self.nctx\n",
        "        # if num_examples is divisible by batchsz * nctx (equivalent to rest is divisible by nctx), we\n",
        "        # have a problem. reduce rest in that case.\n",
        "\n",
        "        if rest % self.nctx == 0:\n",
        "            rest = rest-1\n",
        "        trunc = batch_size * rest\n",
        "        \n",
        "        x_tensor = x_tensor.narrow(0, 0, trunc)\n",
        "        x_tensor = x_tensor.view(batch_size, -1).contiguous()\n",
        "        return x_tensor\n",
        "     \n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EEw2NNZxfp9U",
        "colab_type": "text"
      },
      "source": [
        "This class will keep track of our running average as we go so we dont have to remember to average things in our loops"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pyuDxweqgD01",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Average(object):\n",
        "    def __init__(self, name, fmt=':f'):\n",
        "        self.name = name\n",
        "        self.fmt = fmt\n",
        "        self.val = 0\n",
        "        self.avg = 0\n",
        "        self.sum = 0\n",
        "        self.count = 0\n",
        "\n",
        "    def update(self, val, n=1):\n",
        "        self.val = val\n",
        "        self.sum += val * n\n",
        "        self.count += n\n",
        "        self.avg = self.sum / self.count\n",
        "\n",
        "    def __str__(self):\n",
        "        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'\n",
        "        return fmtstr.format(**self.__dict__)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JnLnechhgIqM",
        "colab_type": "text"
      },
      "source": [
        "We are going to train on batches of contiguous text. Our batches will have been pre-created by the loader.  Each batch will be `BxT` where `B` is the batch size we specified, and `T` is the number of backprop steps through time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z-BDzx2hwIoS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SequenceCriterion(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.crit = nn.CrossEntropyLoss(ignore_index=0, size_average=True)\n",
        "          \n",
        "    def forward(self, inputs, targets):\n",
        "        \"\"\"Evaluate some loss over a sequence.\n",
        "\n",
        "        :param inputs: torch.FloatTensor, [B, .., C] The scores from the model. Batch First\n",
        "        :param targets: torch.LongTensor, The labels.\n",
        "\n",
        "        :returns: torch.FloatTensor, The loss.\n",
        "        \"\"\"\n",
        "        total_sz = targets.nelement()\n",
        "        loss = self.crit(inputs.view(total_sz, -1), targets.view(total_sz))\n",
        "        return loss\n",
        "\n",
        "class LMTrainer:\n",
        "  \n",
        "    def __init__(self, optimizer: torch.optim.Optimizer, nctx):\n",
        "        self.optimizer = optimizer\n",
        "        self.nctx = nctx\n",
        "    \n",
        "    def run(self, model, train_data, loss_function, batch_size=20, clip=0.25):\n",
        "        avg_loss = Average('average_train_loss')\n",
        "        metrics = {}\n",
        "        self.optimizer.zero_grad()\n",
        "        start = time.time()\n",
        "        model.train()\n",
        "        hidden = model.init_hidden(batch_size)\n",
        "        num_steps = train_data.shape[1]//self.nctx\n",
        "        for i in range(num_steps):\n",
        "            x = train_data[:,i*self.nctx:(i + 1) * self.nctx]\n",
        "            y = train_data[:, i*self.nctx+1:(i + 1)*self.nctx + 1]\n",
        "            labels = y.to('cuda:0').transpose(0, 1).contiguous()\n",
        "            inputs = x.to('cuda:0')\n",
        "            logits, (h, c) = model(inputs, hidden)\n",
        "            hidden = (h.detach(), c.detach())\n",
        "            logits = logits.transpose(0, 1).contiguous()\n",
        "            loss = loss_function(logits, labels)\n",
        "            loss.backward()\n",
        "\n",
        "            avg_loss.update(loss.item())\n",
        "\n",
        "            torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
        "            self.optimizer.step()\n",
        "            self.optimizer.zero_grad()\n",
        "            if (i + 1) % 100 == 0:\n",
        "                print(avg_loss)\n",
        "\n",
        "        # How much time elapsed in minutes\n",
        "        elapsed = (time.time() - start)/60\n",
        "        train_token_loss = avg_loss.avg\n",
        "        train_token_ppl = math.exp(train_token_loss)\n",
        "        metrics['train_elapsed_min'] = elapsed\n",
        "        metrics['average_train_loss'] = train_token_loss\n",
        "        metrics['train_ppl'] = train_token_ppl\n",
        "        return metrics\n",
        "\n",
        "class LMEvaluator:\n",
        "    def __init__(self, nctx):\n",
        "        self.nctx = nctx\n",
        "        \n",
        "    def run(self, model, valid_data, loss_function, batch_size=20):\n",
        "        avg_valid_loss = Average('average_valid_loss')\n",
        "        start = time.time()\n",
        "        model.eval()\n",
        "        hidden = model.init_hidden(batch_size)\n",
        "        metrics = {}\n",
        "        num_steps = valid_data.shape[1]//self.nctx\n",
        "        for i in range(num_steps):\n",
        "\n",
        "            with torch.no_grad():\n",
        "                x = valid_data[:,i*self.nctx:(i + 1) * self.nctx]\n",
        "                y = valid_data[:, i*self.nctx+1:(i + 1)*self.nctx + 1]\n",
        "                labels = y.to('cuda:0').transpose(0, 1).contiguous()\n",
        "                inputs = x.to('cuda:0')\n",
        "                \n",
        "                logits, hidden = model(inputs, hidden)\n",
        "                logits = logits.transpose(0, 1).contiguous()\n",
        "                loss = loss_function(logits, labels)\n",
        "                avg_valid_loss.update(loss.item())\n",
        "\n",
        "        valid_token_loss = avg_valid_loss.avg\n",
        "        valid_token_ppl = math.exp(valid_token_loss)\n",
        "\n",
        "        elapsed = (time.time() - start)/60\n",
        "        metrics['valid_elapsed_min'] = elapsed\n",
        "\n",
        "        metrics['average_valid_loss'] = valid_token_loss\n",
        "        metrics['average_valid_word_ppl'] = valid_token_ppl\n",
        "        return metrics\n",
        "      \n",
        "def fit_lm(model, optimizer, epochs, batch_size, nctx, train_data, valid_data):\n",
        "\n",
        "    loss = SequenceCriterion()\n",
        "    trainer = LMTrainer(optimizer, nctx)\n",
        "    evaluator = LMEvaluator(nctx)\n",
        "    best_acc = 0.0\n",
        "\n",
        "    metrics = evaluator.run(model, valid_data, loss, batch_size)\n",
        "\n",
        "    for epoch in range(epochs):\n",
        "\n",
        "        print('EPOCH {}'.format(epoch + 1))\n",
        "        print('=================================')\n",
        "        print('Training Results')\n",
        "        metrics = trainer.run(model, train_data, loss, batch_size)\n",
        "        print(metrics)\n",
        "        print('Validation Results')\n",
        "        metrics = evaluator.run(model, valid_data, loss, batch_size)\n",
        "        print(metrics)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iW1dxycFoihz",
        "colab_type": "text"
      },
      "source": [
        "Now we will train it on [Wikitext-2, Merity et al. 2016](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/).  We will use 35 steps of backprop."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aceEL3kxyJcq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BASE = 'wikitext-2'\n",
        "TRAIN = os.path.join(BASE, 'wiki.train.tokens')\n",
        "VALID = os.path.join(BASE, 'wiki.valid.tokens')\n",
        "\n",
        "batch_size = 20\n",
        "nctx = 35\n",
        "reader = WordDatasetReader(nctx)\n",
        "reader.build_vocab((TRAIN,))\n",
        "\n",
        "train_set = reader.load(TRAIN, batch_size)\n",
        "valid_set = reader.load(VALID, batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UX5AqOMfo6_I",
        "colab_type": "text"
      },
      "source": [
        "Lets start with 1 epoch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sKQuguge7fMY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 663
        },
        "outputId": "06a1ad3d-4fd4-4a6c-ed02-e0c3798f1e8c"
      },
      "source": [
        "\n",
        "model = LSTMLanguageModel(len(reader.vocab), 512, 512)\n",
        "model.to('cuda:0')\n",
        "\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adam(learnable_params, lr=0.001)\n",
        "fit_lm(model, optimizer, 1, batch_size, nctx, train_set, valid_set)\n"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model has 21274623 parameters\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "average_train_loss 7.130287 (7.630262)\n",
            "average_train_loss 6.948112 (7.174242)\n",
            "average_train_loss 6.429612 (6.957250)\n",
            "average_train_loss 6.706582 (6.817735)\n",
            "average_train_loss 6.480259 (6.716661)\n",
            "average_train_loss 6.253604 (6.639940)\n",
            "average_train_loss 6.250593 (6.584427)\n",
            "average_train_loss 6.086081 (6.535446)\n",
            "average_train_loss 6.046218 (6.491021)\n",
            "average_train_loss 5.840661 (6.455732)\n",
            "average_train_loss 6.127773 (6.425025)\n",
            "average_train_loss 5.766460 (6.398616)\n",
            "average_train_loss 6.137995 (6.376816)\n",
            "average_train_loss 6.115303 (6.351095)\n",
            "average_train_loss 6.203366 (6.333509)\n",
            "average_train_loss 6.009459 (6.318195)\n",
            "average_train_loss 6.126120 (6.297565)\n",
            "average_train_loss 5.796104 (6.276726)\n",
            "average_train_loss 5.737082 (6.260670)\n",
            "average_train_loss 5.954897 (6.243683)\n",
            "average_train_loss 5.674878 (6.226430)\n",
            "average_train_loss 5.613625 (6.207307)\n",
            "average_train_loss 5.878324 (6.189868)\n",
            "average_train_loss 5.824322 (6.178013)\n",
            "average_train_loss 5.932457 (6.164326)\n",
            "average_train_loss 5.771354 (6.153274)\n",
            "average_train_loss 5.401644 (6.139012)\n",
            "average_train_loss 5.825085 (6.124258)\n",
            "average_train_loss 5.493943 (6.110777)\n",
            "{'train_elapsed_min': 2.2767986059188843, 'average_train_loss': 6.0969437521813985, 'train_ppl': 444.4971984264256}\n",
            "Validation Results\n",
            "{'valid_elapsed_min': 0.07908193667729696, 'average_valid_loss': 5.534342344345585, 'average_valid_word_ppl': 253.24118736148796}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvywGcj9pJcA",
        "colab_type": "text"
      },
      "source": [
        "We can sample out of our language model using the code below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DWOgGxLbaefT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e9d1f2f6-cc1c-4b8a-9469-6fdb042ebebb"
      },
      "source": [
        "def sample(model, index2word, start_word='the', maxlen=20):\n",
        "  \n",
        "\n",
        "    model.eval() \n",
        "    words = [start_word]\n",
        "    x = torch.tensor(reader.vocab.get(start_word)).long().reshape(1, 1).to('cuda:0')\n",
        "    hidden = model.init_hidden(1)\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for i in range(20):\n",
        "            output, hidden = model(x, hidden)\n",
        "            word_softmax = output.squeeze().exp().cpu()\n",
        "            selected = torch.multinomial(word_softmax, 1)[0]\n",
        "            x.fill_(selected)\n",
        "            word = index2word[selected.item()]\n",
        "            words.append(word)\n",
        "    words.append('...')\n",
        "    return words\n",
        "\n",
        "index2word = {i: w for w, i in reader.vocab.items()}\n",
        "words = sample(model, index2word)\n",
        "print(' '.join(words))\n"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "the latter story pass that would be in Park or Ireland . Like Liam Stuart illustrator , NC apologize and livestock ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aXjMYTo7paIh",
        "colab_type": "text"
      },
      "source": [
        "Lets train a few more epochs and try again"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "in_ftVIipSAS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "b10e9154-2410-477e-8801-8c556b9492b5"
      },
      "source": [
        "fit_lm(model, optimizer, 3, batch_size, 35, train_set, valid_set)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "average_train_loss 5.888901 (5.598958)\n",
            "average_train_loss 5.818975 (5.584962)\n",
            "average_train_loss 5.503317 (5.580568)\n",
            "average_train_loss 5.877174 (5.584086)\n",
            "average_train_loss 5.637257 (5.563949)\n",
            "average_train_loss 5.447403 (5.540543)\n",
            "average_train_loss 5.460862 (5.531339)\n",
            "average_train_loss 5.488514 (5.525078)\n",
            "average_train_loss 5.359737 (5.517013)\n",
            "average_train_loss 5.121772 (5.509718)\n",
            "average_train_loss 5.441720 (5.503375)\n",
            "average_train_loss 5.280029 (5.499962)\n",
            "average_train_loss 5.543726 (5.500008)\n",
            "average_train_loss 5.556562 (5.494267)\n",
            "average_train_loss 5.593565 (5.495319)\n",
            "average_train_loss 5.347257 (5.496266)\n",
            "average_train_loss 5.519910 (5.489749)\n",
            "average_train_loss 5.264927 (5.483263)\n",
            "average_train_loss 5.207999 (5.481013)\n",
            "average_train_loss 5.434073 (5.476722)\n",
            "average_train_loss 5.112748 (5.471222)\n",
            "average_train_loss 5.142471 (5.463090)\n",
            "average_train_loss 5.362827 (5.455768)\n",
            "average_train_loss 5.287307 (5.454580)\n",
            "average_train_loss 5.420770 (5.449693)\n",
            "average_train_loss 5.358116 (5.448896)\n",
            "average_train_loss 5.019379 (5.443910)\n",
            "average_train_loss 5.375151 (5.437743)\n",
            "average_train_loss 5.061219 (5.432098)\n",
            "{'train_elapsed_min': 2.3279770851135253, 'average_train_loss': 5.425043830046748, 'train_ppl': 227.0212964512552}\n",
            "Validation Results\n",
            "{'valid_elapsed_min': 0.0792834202448527, 'average_valid_loss': 5.2854782812057, 'average_valid_word_ppl': 197.4485968212379}\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "average_train_loss 5.594471 (5.241230)\n",
            "average_train_loss 5.464106 (5.238882)\n",
            "average_train_loss 5.156283 (5.241025)\n",
            "average_train_loss 5.523200 (5.251309)\n",
            "average_train_loss 5.250049 (5.232446)\n",
            "average_train_loss 5.129644 (5.205283)\n",
            "average_train_loss 5.083561 (5.195768)\n",
            "average_train_loss 5.166030 (5.192733)\n",
            "average_train_loss 5.086248 (5.188236)\n",
            "average_train_loss 4.777071 (5.182954)\n",
            "average_train_loss 5.161200 (5.178061)\n",
            "average_train_loss 5.008852 (5.176982)\n",
            "average_train_loss 5.142172 (5.180397)\n",
            "average_train_loss 5.281511 (5.176891)\n",
            "average_train_loss 5.325745 (5.180915)\n",
            "average_train_loss 5.139209 (5.184646)\n",
            "average_train_loss 5.210761 (5.179825)\n",
            "average_train_loss 5.041038 (5.176068)\n",
            "average_train_loss 4.949179 (5.175644)\n",
            "average_train_loss 5.189332 (5.173076)\n",
            "average_train_loss 4.856432 (5.168824)\n",
            "average_train_loss 4.836321 (5.162272)\n",
            "average_train_loss 5.064670 (5.156098)\n",
            "average_train_loss 4.960176 (5.156356)\n",
            "average_train_loss 5.104852 (5.152806)\n",
            "average_train_loss 5.087171 (5.153634)\n",
            "average_train_loss 4.799379 (5.150230)\n",
            "average_train_loss 5.135460 (5.145286)\n",
            "average_train_loss 4.837797 (5.140926)\n",
            "{'train_elapsed_min': 2.3252848744392396, 'average_train_loss': 5.135120418061357, 'train_ppl': 169.88477583838508}\n",
            "Validation Results\n",
            "{'valid_elapsed_min': 0.0793849547704061, 'average_valid_loss': 5.172104538640668, 'average_valid_word_ppl': 176.2854469011485}\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "average_train_loss 5.397759 (5.021339)\n",
            "average_train_loss 5.223657 (5.019342)\n",
            "average_train_loss 4.937864 (5.024831)\n",
            "average_train_loss 5.331026 (5.037493)\n",
            "average_train_loss 5.013303 (5.017465)\n",
            "average_train_loss 4.930802 (4.989496)\n",
            "average_train_loss 4.877040 (4.980525)\n",
            "average_train_loss 5.008083 (4.979450)\n",
            "average_train_loss 4.872459 (4.977045)\n",
            "average_train_loss 4.537238 (4.972334)\n",
            "average_train_loss 4.895549 (4.967794)\n",
            "average_train_loss 4.778692 (4.967996)\n",
            "average_train_loss 4.913031 (4.972698)\n",
            "average_train_loss 4.998916 (4.970212)\n",
            "average_train_loss 5.101923 (4.975337)\n",
            "average_train_loss 4.911569 (4.980182)\n",
            "average_train_loss 5.020348 (4.975942)\n",
            "average_train_loss 4.840934 (4.973322)\n",
            "average_train_loss 4.820501 (4.974319)\n",
            "average_train_loss 4.994349 (4.972502)\n",
            "average_train_loss 4.695219 (4.968739)\n",
            "average_train_loss 4.709369 (4.962705)\n",
            "average_train_loss 4.882851 (4.957035)\n",
            "average_train_loss 4.744756 (4.957699)\n",
            "average_train_loss 4.943813 (4.954420)\n",
            "average_train_loss 4.885959 (4.955989)\n",
            "average_train_loss 4.626930 (4.953493)\n",
            "average_train_loss 4.934981 (4.949243)\n",
            "average_train_loss 4.673239 (4.945310)\n",
            "{'train_elapsed_min': 2.333573551972707, 'average_train_loss': 4.939903614627328, 'train_ppl': 139.75677840163237}\n",
            "Validation Results\n",
            "{'valid_elapsed_min': 0.07996076345443726, 'average_valid_loss': 5.096776191649899, 'average_valid_word_ppl': 163.49398352530844}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K-9gBJE9pu2G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "34eb8d38-1eff-4fff-dec4-458cc9367b80"
      },
      "source": [
        "index2word = {i: w for w, i in reader.vocab.items()}\n",
        "words = sample(model, index2word)\n",
        "print(' '.join(words))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "the Supreme Court did not introduce any contact with its way and Grosser Davies of the chance of the country . ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HKbIXgu6rJs8",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## ELMo\n",
        "\n",
        "For the rest of this section, we will focus on ELMo ([Peters et al 2018](https://export.arxiv.org/pdf/1802.05365)), a language model with an embedding layer and 2 subsequent LSTM layers.  Actually, at training time, ELMo is basically two LMs -- one working in the forward direction and one working in the backward direction.  The losses for the forward and reverse directions are averaged.  At inference time, the forward and backward layers are aggregated into a single bidirectional representation at each layer.\n",
        "\n",
        "In our example, we created a word-based LM.  You might have been wondering what to do about words that we havent seen yet -- and that is a valid concern!   Instead of using a word embedding layer like our example above, what if we had a model that used a character compositional approach, taking each character in a word and applying a pooling operation to yield a word representation.  This would mean that the model can handle words that its never seen in the input before.\n",
        "\n",
        "This is exactly what ELMo does -- its based on the research of [Kim et al. 2015](https://arxiv.org/abs/1508.06615).  \n",
        "\n",
        "There is a nice [slide deck by the authors here](http://www.people.fas.harvard.edu/~yoonkim/data/char-nlm-slides.pdf), but the key high-level points are listed here:\n",
        "\n",
        "### Kim Language Model\n",
        "\n",
        "* **Goal**: predict the next word in the sentence (causal LM) but account for unseen words by using a character compositional approach that relies on letters within the pre-segmented words.  This also has the important impact of reducing the number of parameters required in the model drastically over word-level models.\n",
        "\n",
        "* **Using**: LSTM layers that take in a word representation for each position.  Each word is put in and used to predict the next word over a context\n",
        "\n",
        "* **The Twist**: use embeddings approach from [dos Santos & Zadrozny 2014](http://proceedings.mlr.press/v32/santos14.pdf) to represent words, but add parallel filters as in [Kim 2014](https://www.aclweb.org/anthology/D14-1181).  Also, add highway layers on top of the base model\n",
        "\n",
        "\n",
        "\n",
        "### ELMo Language Model\n",
        "\n",
        "* **Goal**: predict the next word in the sentence (causal LM) on the forward sequence **and** predict the previous word on the sentence conditioned on the following context.  \n",
        "\n",
        "* **Using**: LSTM layers as before, but bidirectional, sum the forward and backward loss to make one big loss\n",
        "\n",
        "* **The Twist** Potentially use all layers of the model (except we dont need head with the big softmax at the end over the words). After the fact, we can freeze our biLM embeddings but still provide useful information by learning a linear combination of the layers during downstream training.  During the biLM training, these scalars dont exist\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_19LSBBrq-fD",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "### ELMo with AllenNLP\n",
        "\n",
        "Even though ELMo is just a network like described above, there are a lot of details to getting it set up and reloading the pre-trained checkpoints that are provided, and these details are not really important for demonstration purposes.  So, we will just install [AllenNLP](https://github.com/allenai/allennlp) and use it as a contextual embedding layer.\n",
        "\n",
        "If you are interested in learning more about using ELMo with AllenNLP, they have provided a [tutorial here](https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md)\n",
        "\n",
        "#### TensorFlow and ELMo\n",
        "\n",
        "ELMo was originally trained with TensorFlow.  You can find the code to train and use it in the [bilm-tf repository](https://github.com/allenai/bilm-tf/tree/master/bilm)\n",
        "\n",
        "TF-Hub contains the [pre-trained ELMo model](https://tfhub.dev/google/elmo/2) and is very easy to integrate if you are using TensorFlow already.  The model takes a sequence of words (mixed-case) as inputs and can just be \"glued\" in to your existing models as a sub-graph of your own.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ziMb2Iphr4vJ",
        "colab_type": "code",
        "outputId": "53b96581-c495-41ca-c255-b1e69b321971",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install allennlp"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting allennlp\n",
            "  Using cached https://files.pythonhosted.org/packages/30/8c/72b14d20c9cbb0306939ea41109fc599302634fd5c59ccba1a659b7d0360/allennlp-0.8.4-py3-none-any.whl\n",
            "Collecting jsonnet>=0.10.0; sys_platform != \"win32\" (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a9/a8/adba6cd0f84ee6ab064e7f70cd03a2836cefd2e063fd565180ec13beae93/jsonnet-0.13.0.tar.gz (255kB)\n",
            "\u001b[K     |████████████████████████████████| 256kB 3.4MB/s \n",
            "\u001b[?25hCollecting numpydoc>=0.8.0 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/6a/f3/7cfe4c616e4b9fe05540256cc9c6661c052c8a4cec2915732793b36e1843/numpydoc-0.9.1.tar.gz\n",
            "Collecting pytorch-pretrained-bert>=0.6.0 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 49.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: sqlparse>=0.2.4 in /usr/local/lib/python3.6/dist-packages (from allennlp) (0.3.0)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from allennlp) (3.6.4)\n",
            "Collecting parsimonious>=0.8.0 (from allennlp)\n",
            "  Using cached https://files.pythonhosted.org/packages/02/fc/067a3f89869a41009e1a7cdfb14725f8ddd246f30f63c645e8ef8a1c56f4/parsimonious-0.8.1.tar.gz\n",
            "Collecting conllu==0.11 (from allennlp)\n",
            "  Using cached https://files.pythonhosted.org/packages/d4/2c/856344d9b69baf5b374c395b4286626181a80f0c2b2f704914d18a1cea47/conllu-0.11-py2.py3-none-any.whl\n",
            "Collecting overrides (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/de/55/3100c6d14c1ed177492fcf8f07c4a7d2d6c996c0a7fc6a9a0a41308e7eec/overrides-1.9.tar.gz\n",
            "Collecting awscli>=1.11.91 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/20/fa/f4b6207d59267da0be60be3df32682d2c7479122c7cb87556bd4412675fe/awscli-1.16.190-py2.py3-none-any.whl (1.7MB)\n",
            "\u001b[K     |████████████████████████████████| 1.7MB 51.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy<2.2,>=2.0.18 in /usr/local/lib/python3.6/dist-packages (from allennlp) (2.1.4)\n",
            "Collecting tensorboardX>=1.2 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a2/57/2f0a46538295b8e7f09625da6dd24c23f9d0d7ef119ca1c33528660130d5/tensorboardX-1.7-py2.py3-none-any.whl (238kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 52.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.9.175)\n",
            "Requirement already satisfied: tqdm>=4.19 in /usr/local/lib/python3.6/dist-packages (from allennlp) (4.28.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.3.0)\n",
            "Requirement already satisfied: requests>=2.18 in /usr/local/lib/python3.6/dist-packages (from allennlp) (2.21.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from allennlp) (2.8.0)\n",
            "Requirement already satisfied: gevent>=1.3.6 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.4.0)\n",
            "Collecting jsonpickle (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/07/07/c157520a3ebd166c8c24c6ae0ecae7c3968eb4653ff0e5af369bb82f004d/jsonpickle-1.2-py2.py3-none-any.whl\n",
            "Collecting flask-cors>=3.0.7 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/78/38/e68b11daa5d613e3a91e4bf3da76c94ac9ee0d9cd515af9c1ab80d36f709/Flask_Cors-3.0.8-py2.py3-none-any.whl\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.16.4)\n",
            "Requirement already satisfied: flask>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.0.3)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.6/dist-packages (from allennlp) (2018.9)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from allennlp) (3.2.5)\n",
            "Collecting ftfy (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/8f/86/df789c5834f15ae1ca53a8d4c1fc4788676c2e32112f6a786f2625d9c6e6/ftfy-5.5.1-py3-none-any.whl (43kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 26.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: editdistance in /usr/local/lib/python3.6/dist-packages (from allennlp) (0.5.3)\n",
            "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.1.0)\n",
            "Collecting word2number>=1.1 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/4a/29/a31940c848521f0725f0df6b25dca8917f13a2025b0e8fcbe5d0457e45e6/word2number-1.1.zip\n",
            "Collecting responses>=0.7 (from allennlp)\n",
            "  Using cached https://files.pythonhosted.org/packages/d1/5a/b887e89925f1de7890ef298a74438371ed4ed29b33def9e6d02dc6036fd8/responses-0.10.6-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.6/dist-packages (from allennlp) (3.0.3)\n",
            "Collecting flaky (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/ae/09/94d623dda1adacd51722f3e3e0f88ba08dd030ac2b2662bfb4383096340d/flaky-3.6.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from allennlp) (0.21.2)\n",
            "Collecting unidecode (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d0/42/d9edfed04228bacea2d824904cae367ee9efd05e6cce7ceaaedd0b0ad964/Unidecode-1.1.1-py2.py3-none-any.whl (238kB)\n",
            "\u001b[K     |████████████████████████████████| 245kB 27.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: sphinx>=1.6.5 in /usr/local/lib/python3.6/dist-packages (from numpydoc>=0.8.0->allennlp) (1.8.5)\n",
            "Requirement already satisfied: Jinja2>=2.3 in /usr/local/lib/python3.6/dist-packages (from numpydoc>=0.8.0->allennlp) (2.10.1)\n",
            "Collecting regex (from pytorch-pretrained-bert>=0.6.0->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6f/4e/1b178c38c9a1a184288f72065a65ca01f3154df43c6ad898624149b8b4e0/regex-2019.06.08.tar.gz (651kB)\n",
            "\u001b[K     |████████████████████████████████| 655kB 50.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (1.12.0)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (1.8.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (7.0.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (41.0.1)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (1.3.0)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (19.1.0)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (0.7.1)\n",
            "Requirement already satisfied: PyYAML<=5.1,>=3.10; python_version != \"2.6\" in /usr/local/lib/python3.6/dist-packages (from awscli>=1.11.91->allennlp) (3.13)\n",
            "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from awscli>=1.11.91->allennlp) (0.2.1)\n",
            "Collecting botocore==1.12.180 (from awscli>=1.11.91->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3b/27/fa7da6feb20d1dfc0ab562226061b20da2d27ea18ca32dc764fe86704a99/botocore-1.12.180-py2.py3-none-any.whl (5.6MB)\n",
            "\u001b[K     |████████████████████████████████| 5.6MB 35.1MB/s \n",
            "\u001b[?25hCollecting rsa<=3.5.0,>=3.1.2 (from awscli>=1.11.91->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 26.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: docutils>=0.10 in /usr/local/lib/python3.6/dist-packages (from awscli>=1.11.91->allennlp) (0.14)\n",
            "Collecting colorama<=0.3.9,>=0.2.5 (from awscli>=1.11.91->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl\n",
            "Requirement already satisfied: thinc<7.1.0,>=7.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (7.0.4)\n",
            "Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (0.2.4)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (2.0.2)\n",
            "Requirement already satisfied: jsonschema<3.1.0,>=2.6.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (2.6.0)\n",
            "Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (2.0.1)\n",
            "Requirement already satisfied: srsly<1.1.0,>=0.0.5 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (0.0.7)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (0.2.2)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (1.0.2)\n",
            "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy<2.2,>=2.0.18->allennlp) (0.9.6)\n",
            "Requirement already satisfied: protobuf>=3.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorboardX>=1.2->allennlp) (3.7.1)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->allennlp) (0.9.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (2.8)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (2019.6.16)\n",
            "Requirement already satisfied: greenlet>=0.4.14; platform_python_implementation == \"CPython\" in /usr/local/lib/python3.6/dist-packages (from gevent>=1.3.6->allennlp) (0.4.15)\n",
            "Requirement already satisfied: Werkzeug>=0.14 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp) (0.15.4)\n",
            "Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp) (7.0)\n",
            "Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask>=1.0.2->allennlp) (1.1.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from ftfy->allennlp) (0.1.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp) (0.10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp) (2.5.3)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->allennlp) (2.4.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->allennlp) (0.13.2)\n",
            "Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (2.1.3)\n",
            "Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (1.2.1)\n",
            "Requirement already satisfied: sphinxcontrib-websupport in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (1.1.2)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (19.0)\n",
            "Requirement already satisfied: imagesize in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (1.1.0)\n",
            "Requirement already satisfied: babel!=2.0,>=1.3 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (2.7.0)\n",
            "Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.6.5->numpydoc>=0.8.0->allennlp) (0.7.12)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.3->numpydoc>=0.8.0->allennlp) (1.1.1)\n",
            "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<=3.5.0,>=3.1.2->awscli>=1.11.91->allennlp) (0.4.5)\n",
            "Building wheels for collected packages: jsonnet, numpydoc, parsimonious, overrides, word2number, regex\n",
            "  Building wheel for jsonnet (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/1a/30/ab/ae4a57b1df44fa20a531edb9601b27603da8f5336225691f3f\n",
            "  Building wheel for numpydoc (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/51/30/d1/92a39ba40f21cb70e53f8af96eb98f002a781843c065406500\n",
            "  Building wheel for parsimonious (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/b7/8d/e7/a0e74217da5caeb3c1c7689639b6d28ddbf9985b840bc96a9a\n",
            "  Building wheel for overrides (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/8d/52/86/e5a83b1797e7d263b458d2334edd2704c78508b3eea9323718\n",
            "  Building wheel for word2number (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/46/2f/53/5f5c1d275492f2fce1cdab9a9bb12d49286dead829a4078e0e\n",
            "  Building wheel for regex (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Stored in directory: /root/.cache/pip/wheels/35/e4/80/abf3b33ba89cf65cd262af8a22a5a999cc28fbfabea6b38473\n",
            "Successfully built jsonnet numpydoc parsimonious overrides word2number regex\n",
            "Installing collected packages: jsonnet, numpydoc, regex, pytorch-pretrained-bert, parsimonious, conllu, overrides, botocore, rsa, colorama, awscli, tensorboardX, jsonpickle, flask-cors, ftfy, word2number, responses, flaky, unidecode, allennlp\n",
            "  Found existing installation: botocore 1.12.175\n",
            "    Uninstalling botocore-1.12.175:\n",
            "      Successfully uninstalled botocore-1.12.175\n",
            "  Found existing installation: rsa 4.0\n",
            "    Uninstalling rsa-4.0:\n",
            "      Successfully uninstalled rsa-4.0\n",
            "Successfully installed allennlp-0.8.4 awscli-1.16.190 botocore-1.12.180 colorama-0.3.9 conllu-0.11 flaky-3.6.0 flask-cors-3.0.8 ftfy-5.5.1 jsonnet-0.13.0 jsonpickle-1.2 numpydoc-0.9.1 overrides-1.9 parsimonious-0.8.1 pytorch-pretrained-bert-0.6.2 regex-2019.6.8 responses-0.10.6 rsa-3.4.2 tensorboardX-1.7 unidecode-1.1.1 word2number-1.1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "rsa"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usE8HacRrOfK",
        "colab_type": "text"
      },
      "source": [
        "### Approach\n",
        "\n",
        "We will use mostly the same code as in our previous classification experiments.  For brevity, I have compacted it all here and omitted parts that arent required for this section.  For more information, see the previous section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AN93oU4plXgA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from typing import List, Tuple\n",
        "import os\n",
        "import io\n",
        "import re\n",
        "import codecs\n",
        "import numpy as np\n",
        "from collections import Counter\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "\n",
        "class LSTMClassifier(nn.Module):\n",
        "\n",
        "    def __init__(self, embeddings, num_classes, embed_dims, rnn_units, rnn_layers=1, dropout=0.5, hidden_units=[]):\n",
        "        super().__init__()\n",
        "        self.embeddings = embeddings\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.rnn = torch.nn.LSTM(embed_dims,\n",
        "                                 rnn_units,\n",
        "                                 rnn_layers,\n",
        "                                 dropout=dropout,\n",
        "                                 bidirectional=False,\n",
        "                                 batch_first=False)\n",
        "        nn.init.orthogonal_(self.rnn.weight_hh_l0)\n",
        "        nn.init.orthogonal_(self.rnn.weight_ih_l0)\n",
        "        sequence = []\n",
        "        input_units = rnn_units\n",
        "        output_units = rnn_units\n",
        "        for h in hidden_units:\n",
        "            sequence.append(nn.Linear(input_units, h))\n",
        "            input_units = h\n",
        "            output_units = h\n",
        "            \n",
        "        sequence.append(nn.Linear(output_units, num_classes))\n",
        "        self.outputs = nn.Sequential(*sequence)\n",
        "        \n",
        "        \n",
        "    def forward(self, inputs):\n",
        "        one_hots, lengths = inputs\n",
        "        embed = self.dropout(self.embeddings(one_hots))\n",
        "        embed = embed.transpose(0, 1)\n",
        "        packed = torch.nn.utils.rnn.pack_padded_sequence(embed, lengths.tolist())\n",
        "        _, hidden = self.rnn(packed)\n",
        "        hidden = hidden[0].view(hidden[0].shape[1:])\n",
        "        linear = self.outputs(hidden)\n",
        "        return F.log_softmax(linear, dim=-1)\n",
        "\n",
        "class ConfusionMatrix:\n",
        "    \"\"\"Confusion matrix with metrics\n",
        "\n",
        "    This class accumulates classification output, and tracks it in a confusion matrix.\n",
        "    Metrics are available that use the confusion matrix\n",
        "    \"\"\"\n",
        "    def __init__(self, labels):\n",
        "        \"\"\"Constructor with input labels\n",
        "\n",
        "        :param labels: Either a dictionary (`k=int,v=str`) or an array of labels\n",
        "        \"\"\"\n",
        "        if type(labels) is dict:\n",
        "            self.labels = []\n",
        "            for i in range(len(labels)):\n",
        "                self.labels.append(labels[i])\n",
        "        else:\n",
        "            self.labels = labels\n",
        "        nc = len(self.labels)\n",
        "        self._cm = np.zeros((nc, nc), dtype=np.int)\n",
        "\n",
        "    def add(self, truth, guess):\n",
        "        \"\"\"Add a single value to the confusion matrix based off `truth` and `guess`\n",
        "\n",
        "        :param truth: The real `y` value (or ground truth label)\n",
        "        :param guess: The guess for `y` value (or assertion)\n",
        "        \"\"\"\n",
        "\n",
        "        self._cm[truth, guess] += 1\n",
        "\n",
        "    def __str__(self):\n",
        "        values = []\n",
        "        width = max(8, max(len(x) for x in self.labels) + 1)\n",
        "        for i, label in enumerate([''] + self.labels):\n",
        "            values += [\"{:>{width}}\".format(label, width=width+1)]\n",
        "        values += ['\\n']\n",
        "        for i, label in enumerate(self.labels):\n",
        "            values += [\"{:>{width}}\".format(label, width=width+1)]\n",
        "            for j in range(len(self.labels)):\n",
        "                values += [\"{:{width}d}\".format(self._cm[i, j], width=width + 1)]\n",
        "            values += ['\\n']\n",
        "        values += ['\\n']\n",
        "        return ''.join(values)\n",
        "\n",
        "    def save(self, outfile):\n",
        "        ordered_fieldnames = OrderedDict([(\"labels\", None)] + [(l, None) for l in self.labels])\n",
        "        with open(outfile, 'w') as f:\n",
        "            dw = csv.DictWriter(f, delimiter=',', fieldnames=ordered_fieldnames)\n",
        "            dw.writeheader()\n",
        "            for index, row in enumerate(self._cm):\n",
        "                row_dict = {l: row[i] for i, l in enumerate(self.labels)}\n",
        "                row_dict.update({\"labels\": self.labels[index]})\n",
        "                dw.writerow(row_dict)\n",
        "\n",
        "    def reset(self):\n",
        "        \"\"\"Reset the matrix\n",
        "        \"\"\"\n",
        "        self._cm *= 0\n",
        "\n",
        "    def get_correct(self):\n",
        "        \"\"\"Get the diagonals of the confusion matrix\n",
        "\n",
        "        :return: (``int``) Number of correct classifications\n",
        "        \"\"\"\n",
        "        return self._cm.diagonal().sum()\n",
        "\n",
        "    def get_total(self):\n",
        "        \"\"\"Get total classifications\n",
        "\n",
        "        :return: (``int``) total classifications\n",
        "        \"\"\"\n",
        "        return self._cm.sum()\n",
        "\n",
        "    def get_acc(self):\n",
        "        \"\"\"Get the accuracy\n",
        "\n",
        "        :return: (``float``) accuracy\n",
        "        \"\"\"\n",
        "        return float(self.get_correct())/self.get_total()\n",
        "\n",
        "    def get_recall(self):\n",
        "        \"\"\"Get the recall\n",
        "\n",
        "        :return: (``float``) recall\n",
        "        \"\"\"\n",
        "        total = np.sum(self._cm, axis=1)\n",
        "        total = (total == 0) + total\n",
        "        return np.diag(self._cm) / total.astype(float)\n",
        "\n",
        "    def get_support(self):\n",
        "        return np.sum(self._cm, axis=1)\n",
        "\n",
        "    def get_precision(self):\n",
        "        \"\"\"Get the precision\n",
        "        :return: (``float``) precision\n",
        "        \"\"\"\n",
        "\n",
        "        total = np.sum(self._cm, axis=0)\n",
        "        total = (total == 0) + total\n",
        "        return np.diag(self._cm) / total.astype(float)\n",
        "\n",
        "    def get_mean_precision(self):\n",
        "        \"\"\"Get the mean precision across labels\n",
        "\n",
        "        :return: (``float``) mean precision\n",
        "        \"\"\"\n",
        "        return np.mean(self.get_precision())\n",
        "\n",
        "    def get_weighted_precision(self):\n",
        "        return np.sum(self.get_precision() * self.get_support())/float(self.get_total())\n",
        "\n",
        "    def get_mean_recall(self):\n",
        "        \"\"\"Get the mean recall across labels\n",
        "\n",
        "        :return: (``float``) mean recall\n",
        "        \"\"\"\n",
        "        return np.mean(self.get_recall())\n",
        "\n",
        "    def get_weighted_recall(self):\n",
        "        return np.sum(self.get_recall() * self.get_support())/float(self.get_total())\n",
        "\n",
        "    def get_weighted_f(self, beta=1):\n",
        "        return np.sum(self.get_class_f(beta) * self.get_support())/float(self.get_total())\n",
        "\n",
        "    def get_macro_f(self, beta=1):\n",
        "        \"\"\"Get the macro F_b, with adjustable beta (defaulting to F1)\n",
        "\n",
        "        :param beta: (``float``) defaults to 1 (F1)\n",
        "        :return: (``float``) macro F_b\n",
        "        \"\"\"\n",
        "        if beta < 0:\n",
        "            raise Exception('Beta must be greater than 0')\n",
        "        return np.mean(self.get_class_f(beta))\n",
        "\n",
        "    def get_class_f(self, beta=1):\n",
        "        p = self.get_precision()\n",
        "        r = self.get_recall()\n",
        "\n",
        "        b = beta*beta\n",
        "        d = (b * p + r)\n",
        "        d = (d == 0) + d\n",
        "\n",
        "        return (b + 1) * p * r / d\n",
        "\n",
        "    def get_f(self, beta=1):\n",
        "        \"\"\"Get 2 class F_b, with adjustable beta (defaulting to F1)\n",
        "\n",
        "        :param beta: (``float``) defaults to 1 (F1)\n",
        "        :return: (``float``) 2-class F_b\n",
        "        \"\"\"\n",
        "        p = self.get_precision()[1]\n",
        "        r = self.get_recall()[1]\n",
        "        if beta < 0:\n",
        "            raise Exception('Beta must be greater than 0')\n",
        "        d = (beta*beta * p + r)\n",
        "        if d == 0:\n",
        "            return 0\n",
        "        return (beta*beta + 1) * p * r / d\n",
        "\n",
        "    def get_all_metrics(self):\n",
        "        \"\"\"Make a map of metrics suitable for reporting, keyed by metric name\n",
        "\n",
        "        :return: (``dict``) Map of metrics keyed by metric names\n",
        "        \"\"\"\n",
        "        metrics = {'acc': self.get_acc()}\n",
        "        # If 2 class, assume second class is positive AKA 1\n",
        "        if len(self.labels) == 2:\n",
        "            metrics['precision'] = self.get_precision()[1]\n",
        "            metrics['recall'] = self.get_recall()[1]\n",
        "            metrics['f1'] = self.get_f(1)\n",
        "        else:\n",
        "            metrics['mean_precision'] = self.get_mean_precision()\n",
        "            metrics['mean_recall'] = self.get_mean_recall()\n",
        "            metrics['macro_f1'] = self.get_macro_f(1)\n",
        "            metrics['weighted_precision'] = self.get_weighted_precision()\n",
        "            metrics['weighted_recall'] = self.get_weighted_recall()\n",
        "            metrics['weighted_f1'] = self.get_weighted_f(1)\n",
        "        return metrics\n",
        "\n",
        "    def add_batch(self, truth, guess):\n",
        "        \"\"\"Add a batch of data to the confusion matrix\n",
        "\n",
        "        :param truth: The truth tensor\n",
        "        :param guess: The guess tensor\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        for truth_i, guess_i in zip(truth, guess):\n",
        "            self.add(truth_i, guess_i)\n",
        "\n",
        "class Trainer:\n",
        "    def __init__(self, optimizer: torch.optim.Optimizer):\n",
        "        self.optimizer = optimizer\n",
        "\n",
        "    def run(self, model, labels, train, loss, batch_size): \n",
        "        model.train()       \n",
        "        train_loader = DataLoader(train, batch_size=batch_size, shuffle=True)\n",
        "\n",
        "        cm = ConfusionMatrix(labels)\n",
        "\n",
        "        for batch in train_loader:\n",
        "            loss_value, y_pred, y_actual = self.update(model, loss, batch)\n",
        "            _, best = y_pred.max(1)\n",
        "            yt = y_actual.cpu().int().numpy()\n",
        "            yp = best.cpu().int().numpy()\n",
        "            cm.add_batch(yt, yp)\n",
        "\n",
        "        print(cm.get_all_metrics())\n",
        "        return cm\n",
        "    \n",
        "    def update(self, model, loss, batch):\n",
        "        self.optimizer.zero_grad()\n",
        "        x, lengths, y = batch\n",
        "        lengths, perm_idx = lengths.sort(0, descending=True)\n",
        "        x_sorted = x[perm_idx]\n",
        "        y_sorted = y[perm_idx]\n",
        "        y_sorted = y_sorted.to('cuda:0')\n",
        "        inputs = (x_sorted.to('cuda:0'), lengths)\n",
        "        y_pred = model(inputs)\n",
        "        loss_value = loss(y_pred, y_sorted)\n",
        "        loss_value.backward()\n",
        "        self.optimizer.step()\n",
        "        return loss_value.item(), y_pred, y_sorted\n",
        "\n",
        "class Evaluator:\n",
        "    def __init__(self):\n",
        "        pass\n",
        "\n",
        "    def run(self, model, labels, dataset, batch_size=1):\n",
        "        model.eval()\n",
        "        valid_loader = DataLoader(dataset, batch_size=batch_size)\n",
        "        cm = ConfusionMatrix(labels)\n",
        "        for batch in valid_loader:\n",
        "            y_pred, y_actual = self.inference(model, batch)\n",
        "            _, best = y_pred.max(1)\n",
        "            yt = y_actual.cpu().int().numpy()\n",
        "            yp = best.cpu().int().numpy()\n",
        "            cm.add_batch(yt, yp)\n",
        "        return cm\n",
        "\n",
        "    def inference(self, model, batch):\n",
        "        with torch.no_grad():\n",
        "            x, lengths, y = batch\n",
        "            lengths, perm_idx = lengths.sort(0, descending=True)\n",
        "            x_sorted = x[perm_idx]\n",
        "            y_sorted = y[perm_idx]\n",
        "            y_sorted = y_sorted.to('cuda:0')\n",
        "            inputs = (x_sorted.to('cuda:0'), lengths)\n",
        "            y_pred = model(inputs)\n",
        "            return y_pred, y_sorted\n",
        "\n",
        "def fit(model, labels, optimizer, loss, epochs, batch_size, train, valid, test):\n",
        "\n",
        "    trainer = Trainer(optimizer)\n",
        "    evaluator = Evaluator()\n",
        "    best_acc = 0.0\n",
        "    \n",
        "    for epoch in range(epochs):\n",
        "        print('EPOCH {}'.format(epoch + 1))\n",
        "        print('=================================')\n",
        "        print('Training Results')\n",
        "        cm = trainer.run(model, labels, train, loss, batch_size)\n",
        "        print('Validation Results')\n",
        "        cm = evaluator.run(model, labels, valid)\n",
        "        print(cm.get_all_metrics())\n",
        "        if cm.get_acc() > best_acc:\n",
        "            print('New best model {:.2f}'.format(cm.get_acc()))\n",
        "            best_acc = cm.get_acc()\n",
        "            torch.save(model.state_dict(), './checkpoint.pth')\n",
        "    if test:\n",
        "        model.load_state_dict(torch.load('./checkpoint.pth'))\n",
        "        cm = evaluator.run(model, labels, test)\n",
        "        print('Final result')\n",
        "        print(cm.get_all_metrics())\n",
        "    return cm.get_acc()\n",
        "\n",
        "def whitespace_tokenizer(words: str) -> List[str]:\n",
        "    return words.split() \n",
        "\n",
        "def sst2_tokenizer(words: str) -> List[str]:\n",
        "    REPLACE = { \"'s\": \" 's \",\n",
        "                \"'ve\": \" 've \",\n",
        "                \"n't\": \" n't \",\n",
        "                \"'re\": \" 're \",\n",
        "                \"'d\": \" 'd \",\n",
        "                \"'ll\": \" 'll \",\n",
        "                \",\": \" , \",\n",
        "                \"!\": \" ! \",\n",
        "                }\n",
        "    words = words.lower()\n",
        "    words = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", words)\n",
        "    for k, v in REPLACE.items():\n",
        "            words = words.replace(k, v)\n",
        "    return [w.strip() for w in words.split()]\n",
        "\n",
        "\n",
        "class Reader:\n",
        "\n",
        "    def __init__(self, files, lowercase=True, min_freq=0,\n",
        "                 tokenizer=sst2_tokenizer, vectorizer=None):\n",
        "        self.lowercase = lowercase\n",
        "        self.tokenizer = tokenizer\n",
        "        build_vocab = vectorizer is None\n",
        "        self.vectorizer = vectorizer if vectorizer else self._vectorizer\n",
        "        x = Counter()\n",
        "        y = Counter()\n",
        "        for file_name in files:\n",
        "            if file_name is None:\n",
        "                continue\n",
        "            with codecs.open(file_name, encoding='utf-8', mode='r') as f:\n",
        "                for line in f:\n",
        "                    words = line.split()\n",
        "                    y.update(words[0])\n",
        "\n",
        "                    if build_vocab:\n",
        "                        words = self.tokenizer(' '.join(words[1:]))\n",
        "                        words = words if not self.lowercase else [w.lower() for w in words]\n",
        "                        x.update(words)\n",
        "        self.labels = list(y.keys())\n",
        "\n",
        "        if build_vocab:\n",
        "            x = dict(filter(lambda cnt: cnt[1] >= min_freq, x.items()))\n",
        "            alpha = list(x.keys())\n",
        "            alpha.sort()\n",
        "            self.vocab = {w: i+1 for i, w in enumerate(alpha)}\n",
        "            self.vocab['[PAD]'] = 0\n",
        "\n",
        "        self.labels.sort()\n",
        "\n",
        "    def _vectorizer(self, words: List[str]) -> List[int]:\n",
        "        return [self.vocab.get(w, 0) for w in words]\n",
        "\n",
        "    def load(self, filename: str) -> TensorDataset:\n",
        "        label2index = {l: i for i, l in enumerate(self.labels)}\n",
        "        xs = []\n",
        "        lengths = []\n",
        "        ys = []\n",
        "        with codecs.open(filename, encoding='utf-8', mode='r') as f:\n",
        "            for line in f:\n",
        "                words = line.split()\n",
        "                ys.append(label2index[words[0]])\n",
        "                words = self.tokenizer(' '.join(words[1:]))\n",
        "                words = words if not self.lowercase else [w.lower() for w in words]\n",
        "                vec = self.vectorizer(words)\n",
        "                lengths.append(len(vec))\n",
        "                xs.append(torch.tensor(vec, dtype=torch.long))\n",
        "        x_tensor = torch.nn.utils.rnn.pad_sequence(xs, batch_first=True)\n",
        "        lengths_tensor = torch.tensor(lengths, dtype=torch.long)\n",
        "        y_tensor = torch.tensor(ys, dtype=torch.long)\n",
        "        return TensorDataset(x_tensor, lengths_tensor, y_tensor)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fJpvG1NTxSo9",
        "colab_type": "text"
      },
      "source": [
        "### The new thing: set up to use ELMo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HH2fR03DxeNw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from allennlp.modules.elmo import Elmo, batch_to_ids\n",
        "\n",
        "\n",
        "def elmo_vectorizer(sentence):\n",
        "    character_ids = batch_to_ids([sentence])\n",
        "    return character_ids.squeeze(0)\n",
        "\n",
        "  \n",
        "class ElmoEmbedding(nn.Module):\n",
        "    def __init__(self, options_file, weight_file, dropout=0.5):\n",
        "        super().__init__()\n",
        "        self.elmo = Elmo(options_file, weight_file, 2, dropout=dropout)\n",
        "    def forward(self, xch):\n",
        "        elmo = self.elmo(xch)\n",
        "        e1, e2 = elmo['elmo_representations']\n",
        "        mask = elmo['mask']\n",
        "        embeddings = (e1 + e2) * mask.float().unsqueeze(-1)\n",
        "        return embeddings\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xuxm3Ugau6jW",
        "colab_type": "text"
      },
      "source": [
        "As before, we are going to load up our data with a reader.  This time, though, we will provide a vectorizer for ELMo.  In our simple example `Reader`, we only allow a single feature as our input vector to our classifier, so we can stop counting up our vocab.  In real life, you probably want to support both word vector features and context vector features so you might want to modify the code to support both.  This is a very common approach -- just using ELMo to augment an existing setup.  Here, we just look at using ELMo features by themselves.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-NzW34_RwGUw",
        "colab_type": "code",
        "outputId": "e667b3f4-a3b3-417f-8b1c-61180acd83f1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377
        }
      },
      "source": [
        "!wget https://www.dropbox.com/s/08km2ean8bkt7p3/trec.tar.gz?dl=1\n",
        "!tar -xzf 'trec.tar.gz?dl=1'"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-06-30 19:21:55--  https://www.dropbox.com/s/08km2ean8bkt7p3/trec.tar.gz?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.8.1, 2620:100:601b:1::a27d:801\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.8.1|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/dl/08km2ean8bkt7p3/trec.tar.gz [following]\n",
            "--2019-06-30 19:21:56--  https://www.dropbox.com/s/dl/08km2ean8bkt7p3/trec.tar.gz\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc7fa2ae1930db92d5916f06ba12.dl.dropboxusercontent.com/cd/0/get/Aj3XoF2sz7098a7ulJjBQP5DA6LkkkTQEAgFciDKPLgTZrHSUdejKQ7f8hkI3LiEt0BP_zf3LYg-ul8IZkevEcRCL4oxvYa8Uw-4SCn9GK2Lqw/file?dl=1# [following]\n",
            "--2019-06-30 19:21:56--  https://uc7fa2ae1930db92d5916f06ba12.dl.dropboxusercontent.com/cd/0/get/Aj3XoF2sz7098a7ulJjBQP5DA6LkkkTQEAgFciDKPLgTZrHSUdejKQ7f8hkI3LiEt0BP_zf3LYg-ul8IZkevEcRCL4oxvYa8Uw-4SCn9GK2Lqw/file?dl=1\n",
            "Resolving uc7fa2ae1930db92d5916f06ba12.dl.dropboxusercontent.com (uc7fa2ae1930db92d5916f06ba12.dl.dropboxusercontent.com)... 162.125.8.6, 2620:100:601b:6::a27d:806\n",
            "Connecting to uc7fa2ae1930db92d5916f06ba12.dl.dropboxusercontent.com (uc7fa2ae1930db92d5916f06ba12.dl.dropboxusercontent.com)|162.125.8.6|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 117253 (115K) [application/binary]\n",
            "Saving to: ‘trec.tar.gz?dl=1’\n",
            "\n",
            "trec.tar.gz?dl=1    100%[===================>] 114.50K  --.-KB/s    in 0.07s   \n",
            "\n",
            "2019-06-30 19:21:56 (1.71 MB/s) - ‘trec.tar.gz?dl=1’ saved [117253/117253]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPy-A82s048z",
        "colab_type": "text"
      },
      "source": [
        "We will set up our reader slightly differently than in the last experiment.  Here we will use an `elmo_vectorizer`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aI5r7tmlvgUu",
        "colab_type": "code",
        "outputId": "b4e390a9-7886-46d5-87a4-5c3b07c9602b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "BASE = 'trec'\n",
        "TRAIN = os.path.join(BASE, 'trec.nodev.utf8')\n",
        "VALID = os.path.join(BASE, 'trec.dev.utf8')\n",
        "TEST = os.path.join(BASE, 'trec.test.utf8')\n",
        "\n",
        "\n",
        "\n",
        "reader = Reader((TRAIN, VALID, TEST,), lowercase=False, vectorizer=elmo_vectorizer)\n",
        "train = reader.load(TRAIN)\n",
        "valid = reader.load(VALID)\n",
        "test = reader.load(TEST)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:392: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YmE1VJSpws30",
        "colab_type": "text"
      },
      "source": [
        "Building the network is basically the same as before, but we are using ELMo instead of word vectors.  The command below will take a few minutes -- this is a much larger (forward) network than before, even though the learnable parameters havent really changed"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_7y6t65zw_iV",
        "colab_type": "code",
        "outputId": "5e06aaca-85b9-4da8-ecd9-49fc81c553b8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "options_file = \"https://allennlp.s3.amazonaws.com/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json\"\n",
        "weight_file = \"https://allennlp.s3.amazonaws.com/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5\"\n",
        "embeddings = ElmoEmbedding(options_file, weight_file)\n",
        "model = LSTMClassifier(embeddings, len(reader.labels), embed_dims=1024, rnn_units=100, hidden_units=[100])\n",
        "\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "model.to('cuda:0')\n",
        "loss = torch.nn.NLLLoss()\n",
        "loss = loss.to('cuda:0')\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adadelta(learnable_params, lr=1.0)\n",
        "\n",
        "fit(model, reader.labels, optimizer, loss, 12, 50, train, valid, test)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 336/336 [00:00<00:00, 192499.13B/s]\n",
            "100%|██████████| 374434792/374434792 [00:07<00:00, 47927932.74B/s]\n",
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:54: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1\n",
            "  \"num_layers={}\".format(dropout, num_layers))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Model has 461114 parameters\n",
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.5608, 'mean_precision': 0.6483439079531595, 'mean_recall': 0.48504498768062404, 'macro_f1': 0.49849106627634976, 'weighted_precision': 0.5726962123308302, 'weighted_recall': 0.5608, 'weighted_f1': 0.5554788741148295}\n",
            "Validation Results\n",
            "{'acc': 0.7942477876106194, 'mean_precision': 0.8454765420711672, 'mean_recall': 0.7033276693176708, 'macro_f1': 0.7144610587048233, 'weighted_precision': 0.8102895316760798, 'weighted_recall': 0.7942477876106194, 'weighted_f1': 0.7887777126414355}\n",
            "New best model 0.79\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.806, 'mean_precision': 0.799350329535837, 'mean_recall': 0.7675872074813431, 'macro_f1': 0.780728542640896, 'weighted_precision': 0.8062829252605372, 'weighted_recall': 0.806, 'weighted_f1': 0.8058397968891035}\n",
            "Validation Results\n",
            "{'acc': 0.8628318584070797, 'mean_precision': 0.8566120843164245, 'mean_recall': 0.7974693543452065, 'macro_f1': 0.8182667932069821, 'weighted_precision': 0.8675313347987196, 'weighted_recall': 0.8628318584070797, 'weighted_f1': 0.8625189847178025}\n",
            "New best model 0.86\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8678, 'mean_precision': 0.8675015318855253, 'mean_recall': 0.8346532456259291, 'macro_f1': 0.8484927361816553, 'weighted_precision': 0.8682517001586247, 'weighted_recall': 0.8678, 'weighted_f1': 0.8677362764323896}\n",
            "Validation Results\n",
            "{'acc': 0.8451327433628318, 'mean_precision': 0.8284211573091326, 'mean_recall': 0.8093879960516328, 'macro_f1': 0.8110225138172149, 'weighted_precision': 0.8691115810447773, 'weighted_recall': 0.8451327433628318, 'weighted_f1': 0.8465397357783465}\n",
            "EPOCH 4\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8872, 'mean_precision': 0.8764661421280517, 'mean_recall': 0.8546009991636673, 'macro_f1': 0.8643704500888516, 'weighted_precision': 0.887561002584866, 'weighted_recall': 0.8872, 'weighted_f1': 0.8872276932804481}\n",
            "Validation Results\n",
            "{'acc': 0.911504424778761, 'mean_precision': 0.8515226408802844, 'mean_recall': 0.8617077224611437, 'macro_f1': 0.8561887828467433, 'weighted_precision': 0.9122749445064818, 'weighted_recall': 0.911504424778761, 'weighted_f1': 0.9118158975632823}\n",
            "New best model 0.91\n",
            "EPOCH 5\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9034, 'mean_precision': 0.9068352283292169, 'mean_recall': 0.8843802250756597, 'macro_f1': 0.8946296708241798, 'weighted_precision': 0.9040643245149811, 'weighted_recall': 0.9034, 'weighted_f1': 0.9035884797279896}\n",
            "Validation Results\n",
            "{'acc': 0.8871681415929203, 'mean_precision': 0.8310659320074388, 'mean_recall': 0.841863153832931, 'macro_f1': 0.8355145420604436, 'weighted_precision': 0.8885588116644558, 'weighted_recall': 0.8871681415929203, 'weighted_f1': 0.8871217953267708}\n",
            "EPOCH 6\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9136, 'mean_precision': 0.9192746333288291, 'mean_recall': 0.8914669258673943, 'macro_f1': 0.903828395837297, 'weighted_precision': 0.9139512391702285, 'weighted_recall': 0.9136, 'weighted_f1': 0.913614469191629}\n",
            "Validation Results\n",
            "{'acc': 0.9048672566371682, 'mean_precision': 0.8453363940567148, 'mean_recall': 0.8564313119872883, 'macro_f1': 0.8503405229048734, 'weighted_precision': 0.905338925288292, 'weighted_recall': 0.9048672566371682, 'weighted_f1': 0.9048873521303485}\n",
            "EPOCH 7\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9184, 'mean_precision': 0.9217954417236368, 'mean_recall': 0.9035341951741954, 'macro_f1': 0.9119837710405331, 'weighted_precision': 0.9188094085046438, 'weighted_recall': 0.9184, 'weighted_f1': 0.9185065760698944}\n",
            "Validation Results\n",
            "{'acc': 0.9004424778761062, 'mean_precision': 0.8343728710441182, 'mean_recall': 0.8542340405568197, 'macro_f1': 0.8426629413676556, 'weighted_precision': 0.9019737446759757, 'weighted_recall': 0.9004424778761062, 'weighted_f1': 0.9006848775343249}\n",
            "EPOCH 8\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9252, 'mean_precision': 0.9227662229391251, 'mean_recall': 0.9085845822017588, 'macro_f1': 0.9152560555320276, 'weighted_precision': 0.9254505663098069, 'weighted_recall': 0.9252, 'weighted_f1': 0.9252609572329403}\n",
            "Validation Results\n",
            "{'acc': 0.8960176991150443, 'mean_precision': 0.8848359324236518, 'mean_recall': 0.8515305594157283, 'macro_f1': 0.8641410893717477, 'weighted_precision': 0.897474904298379, 'weighted_recall': 0.8960176991150443, 'weighted_f1': 0.8954448264468791}\n",
            "EPOCH 9\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9366, 'mean_precision': 0.9421415595699045, 'mean_recall': 0.9253828413493465, 'macro_f1': 0.9332020129586184, 'weighted_precision': 0.9367741614764589, 'weighted_recall': 0.9366, 'weighted_f1': 0.9366203849323997}\n",
            "Validation Results\n",
            "{'acc': 0.9004424778761062, 'mean_precision': 0.8408851907016573, 'mean_recall': 0.8542708251432938, 'macro_f1': 0.8466559111080202, 'weighted_precision': 0.9022774132643538, 'weighted_recall': 0.9004424778761062, 'weighted_f1': 0.9006261595204735}\n",
            "EPOCH 10\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9422, 'mean_precision': 0.9415872377873563, 'mean_recall': 0.9301100255239593, 'macro_f1': 0.9356066415360083, 'weighted_precision': 0.9423787344008276, 'weighted_recall': 0.9422, 'weighted_f1': 0.9422531801175381}\n",
            "Validation Results\n",
            "{'acc': 0.9026548672566371, 'mean_precision': 0.8534388800712419, 'mean_recall': 0.855449985872144, 'macro_f1': 0.8538969412521858, 'weighted_precision': 0.9037659180936365, 'weighted_recall': 0.9026548672566371, 'weighted_f1': 0.90246529999771}\n",
            "EPOCH 11\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9432, 'mean_precision': 0.9422754608090832, 'mean_recall': 0.938376139581592, 'macro_f1': 0.9402970803722553, 'weighted_precision': 0.9432734858574917, 'weighted_recall': 0.9432, 'weighted_f1': 0.943229377825017}\n",
            "Validation Results\n",
            "{'acc': 0.9137168141592921, 'mean_precision': 0.8628400105220431, 'mean_recall': 0.8646482805732667, 'macro_f1': 0.8633776502808389, 'weighted_precision': 0.9132203845237589, 'weighted_recall': 0.9137168141592921, 'weighted_f1': 0.9130050592497775}\n",
            "New best model 0.91\n",
            "EPOCH 12\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9544, 'mean_precision': 0.9557163129978826, 'mean_recall': 0.9458500359607124, 'macro_f1': 0.9506063779628039, 'weighted_precision': 0.9545506681185594, 'weighted_recall': 0.9544, 'weighted_f1': 0.9544423559597639}\n",
            "Validation Results\n",
            "{'acc': 0.9092920353982301, 'mean_precision': 0.8510768742634296, 'mean_recall': 0.8608961905116529, 'macro_f1': 0.8550990513587272, 'weighted_precision': 0.9106944939486582, 'weighted_recall': 0.9092920353982301, 'weighted_f1': 0.9093039549088799}\n",
            "Final result\n",
            "{'acc': 0.944, 'mean_precision': 0.9333687372820768, 'mean_recall': 0.9161547629123813, 'macro_f1': 0.9230157805001022, 'weighted_precision': 0.9449538854974426, 'weighted_recall': 0.944, 'weighted_f1': 0.9429751846143404}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.944"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bgrkf2EP4htl",
        "colab_type": "text"
      },
      "source": [
        "Let's see how this number compares against a randomly initialized baseline model that is otherwise identical.  We dont really need to use such a huge embedding size in this case -- we are using word vectors instead of character compositional vectors and we dont really have enough information to train a huge word embedding from scratch.  Also, since we dont have much information, we will use lowercased features.  Note that using these word embeddings features, our model has **6x more parameters than before**.  Also, we might want to train it longer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b97zMOBr3-UA",
        "colab_type": "code",
        "outputId": "7079df5b-96c3-48d0-895e-e03be50b212d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "\n",
        "r = Reader((TRAIN, VALID, TEST,), lowercase=True)\n",
        "train = r.load(TRAIN)\n",
        "valid = r.load(VALID)\n",
        "test = r.load(TEST)\n",
        "\n",
        "embeddings = nn.Embedding(len(r.vocab), 300)\n",
        "model = LSTMClassifier(embeddings, len(r.labels), embeddings.weight.shape[1], rnn_units=100, hidden_units=[100])\n",
        "\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "model.to('cuda:0')\n",
        "loss = torch.nn.NLLLoss()\n",
        "loss = loss.to('cuda:0')\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adadelta(learnable_params, lr=1.0)\n",
        "\n",
        "fit(model, r.labels, optimizer, loss, 48, 50, train, valid, test)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:54: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1\n",
            "  \"num_layers={}\".format(dropout, num_layers))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Model has 2801306 parameters\n",
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.2542, 'mean_precision': 0.31261376969825, 'mean_recall': 0.2142077519956256, 'macro_f1': 0.21100023435767312, 'weighted_precision': 0.24503296037652494, 'weighted_recall': 0.2542, 'weighted_f1': 0.22759490197394544}\n",
            "Validation Results\n",
            "{'acc': 0.31194690265486724, 'mean_precision': 0.30219607432329726, 'mean_recall': 0.2697651533307747, 'macro_f1': 0.23033350512330675, 'weighted_precision': 0.3308620779538906, 'weighted_recall': 0.31194690265486724, 'weighted_f1': 0.25984647084205925}\n",
            "New best model 0.31\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.3776, 'mean_precision': 0.4211727454667504, 'mean_recall': 0.3614489295644783, 'macro_f1': 0.3731848509532312, 'weighted_precision': 0.37235350929169814, 'weighted_recall': 0.3776, 'weighted_f1': 0.3631077775707718}\n",
            "Validation Results\n",
            "{'acc': 0.4557522123893805, 'mean_precision': 0.486368442230124, 'mean_recall': 0.44059437145396335, 'macro_f1': 0.4116212305386946, 'weighted_precision': 0.5028626993515615, 'weighted_recall': 0.4557522123893805, 'weighted_f1': 0.42325171491881114}\n",
            "New best model 0.46\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.5432, 'mean_precision': 0.5874298893317987, 'mean_recall': 0.5213749406751867, 'macro_f1': 0.5432030532887705, 'weighted_precision': 0.5414128746603742, 'weighted_recall': 0.5432, 'weighted_f1': 0.539731134218227}\n",
            "Validation Results\n",
            "{'acc': 0.6592920353982301, 'mean_precision': 0.6419746068058818, 'mean_recall': 0.6268845865056035, 'macro_f1': 0.6307977552497951, 'weighted_precision': 0.662392825769239, 'weighted_recall': 0.6592920353982301, 'weighted_f1': 0.6571270878109032}\n",
            "New best model 0.66\n",
            "EPOCH 4\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.6652, 'mean_precision': 0.7019592671173288, 'mean_recall': 0.6360073323821094, 'macro_f1': 0.6596012872763003, 'weighted_precision': 0.6699451752521243, 'weighted_recall': 0.6652, 'weighted_f1': 0.6663461696747198}\n",
            "Validation Results\n",
            "{'acc': 0.7256637168141593, 'mean_precision': 0.6947344026048728, 'mean_recall': 0.684312146723145, 'macro_f1': 0.6882240153205471, 'weighted_precision': 0.7329104731535425, 'weighted_recall': 0.7256637168141593, 'weighted_f1': 0.7276818270316268}\n",
            "New best model 0.73\n",
            "EPOCH 5\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7262, 'mean_precision': 0.7559759521433286, 'mean_recall': 0.6964030168119725, 'macro_f1': 0.718961419314074, 'weighted_precision': 0.7314068173369025, 'weighted_recall': 0.7262, 'weighted_f1': 0.7276772804705455}\n",
            "Validation Results\n",
            "{'acc': 0.7699115044247787, 'mean_precision': 0.767162729738685, 'mean_recall': 0.7171683804327849, 'macro_f1': 0.7358018640759697, 'weighted_precision': 0.7869035814233268, 'weighted_recall': 0.7699115044247787, 'weighted_f1': 0.7730466432148493}\n",
            "New best model 0.77\n",
            "EPOCH 6\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7774, 'mean_precision': 0.7991829240246574, 'mean_recall': 0.7460179937770648, 'macro_f1': 0.7670813561197591, 'weighted_precision': 0.7829082373273452, 'weighted_recall': 0.7774, 'weighted_f1': 0.7790066442175066}\n",
            "Validation Results\n",
            "{'acc': 0.7942477876106194, 'mean_precision': 0.7722010233604815, 'mean_recall': 0.7413137565797516, 'macro_f1': 0.7527105440234411, 'weighted_precision': 0.7965885015875331, 'weighted_recall': 0.7942477876106194, 'weighted_f1': 0.7932362973950707}\n",
            "New best model 0.79\n",
            "EPOCH 7\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8094, 'mean_precision': 0.8190739759778141, 'mean_recall': 0.7893403911783444, 'macro_f1': 0.802535501291079, 'weighted_precision': 0.8128192118492741, 'weighted_recall': 0.8094, 'weighted_f1': 0.8105827362652248}\n",
            "Validation Results\n",
            "{'acc': 0.7876106194690266, 'mean_precision': 0.7485831299937852, 'mean_recall': 0.737125918471548, 'macro_f1': 0.739828985707053, 'weighted_precision': 0.7889226923611903, 'weighted_recall': 0.7876106194690266, 'weighted_f1': 0.7851833379237152}\n",
            "EPOCH 8\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8334, 'mean_precision': 0.8425566386344959, 'mean_recall': 0.8020487930554469, 'macro_f1': 0.8188927928986053, 'weighted_precision': 0.8361427439552401, 'weighted_recall': 0.8334, 'weighted_f1': 0.8341487335405394}\n",
            "Validation Results\n",
            "{'acc': 0.8053097345132744, 'mean_precision': 0.7836886850840924, 'mean_recall': 0.7717037075656982, 'macro_f1': 0.7766933663665941, 'weighted_precision': 0.8124743795936054, 'weighted_recall': 0.8053097345132744, 'weighted_f1': 0.8076258267664438}\n",
            "New best model 0.81\n",
            "EPOCH 9\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8468, 'mean_precision': 0.8504311729666333, 'mean_recall': 0.8200134573572487, 'macro_f1': 0.8331837029506359, 'weighted_precision': 0.8494250195065668, 'weighted_recall': 0.8468, 'weighted_f1': 0.8475914482379496}\n",
            "Validation Results\n",
            "{'acc': 0.8163716814159292, 'mean_precision': 0.8122787276154813, 'mean_recall': 0.7728318623385028, 'macro_f1': 0.7845697172233551, 'weighted_precision': 0.841008158316879, 'weighted_recall': 0.8163716814159292, 'weighted_f1': 0.8194346806526578}\n",
            "New best model 0.82\n",
            "EPOCH 10\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.856, 'mean_precision': 0.8583456863001512, 'mean_recall': 0.8344367321617767, 'macro_f1': 0.8452794073478773, 'weighted_precision': 0.858502410730061, 'weighted_recall': 0.856, 'weighted_f1': 0.8568463379552975}\n",
            "Validation Results\n",
            "{'acc': 0.827433628318584, 'mean_precision': 0.8065578820468894, 'mean_recall': 0.7875269000666668, 'macro_f1': 0.7948885464239536, 'weighted_precision': 0.8376851538502355, 'weighted_recall': 0.827433628318584, 'weighted_f1': 0.8300346063518604}\n",
            "New best model 0.83\n",
            "EPOCH 11\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8752, 'mean_precision': 0.8867543443057883, 'mean_recall': 0.8682828032176982, 'macro_f1': 0.876877938560822, 'weighted_precision': 0.8765970442190595, 'weighted_recall': 0.8752, 'weighted_f1': 0.8756987373269298}\n",
            "Validation Results\n",
            "{'acc': 0.8429203539823009, 'mean_precision': 0.810890350645565, 'mean_recall': 0.8200588537595558, 'macro_f1': 0.8091225485091558, 'weighted_precision': 0.856571882711008, 'weighted_recall': 0.8429203539823009, 'weighted_f1': 0.8457428127083414}\n",
            "New best model 0.84\n",
            "EPOCH 12\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8792, 'mean_precision': 0.8792557729505756, 'mean_recall': 0.8559171118516193, 'macro_f1': 0.8664273096431591, 'weighted_precision': 0.8802943928708451, 'weighted_recall': 0.8792, 'weighted_f1': 0.8794929463166824}\n",
            "Validation Results\n",
            "{'acc': 0.831858407079646, 'mean_precision': 0.7962677456184699, 'mean_recall': 0.8169955255150682, 'macro_f1': 0.8012212821990726, 'weighted_precision': 0.8439342848824284, 'weighted_recall': 0.831858407079646, 'weighted_f1': 0.8345906017116217}\n",
            "EPOCH 13\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8982, 'mean_precision': 0.8926979300525084, 'mean_recall': 0.8720075500179675, 'macro_f1': 0.8813865977662623, 'weighted_precision': 0.8990274368829654, 'weighted_recall': 0.8982, 'weighted_f1': 0.898400074877493}\n",
            "Validation Results\n",
            "{'acc': 0.8407079646017699, 'mean_precision': 0.8044426437483548, 'mean_recall': 0.8230926174040855, 'macro_f1': 0.8115058510891844, 'weighted_precision': 0.8446897114706726, 'weighted_recall': 0.8407079646017699, 'weighted_f1': 0.8418088284238727}\n",
            "EPOCH 14\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8994, 'mean_precision': 0.90397473118525, 'mean_recall': 0.8869886283764901, 'macro_f1': 0.8949155550364044, 'weighted_precision': 0.9000862304408612, 'weighted_recall': 0.8994, 'weighted_f1': 0.8996270341701729}\n",
            "Validation Results\n",
            "{'acc': 0.834070796460177, 'mean_precision': 0.7941534082526428, 'mean_recall': 0.8181802560779231, 'macro_f1': 0.802448595927142, 'weighted_precision': 0.839962405943381, 'weighted_recall': 0.834070796460177, 'weighted_f1': 0.8360121837053236}\n",
            "EPOCH 15\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9078, 'mean_precision': 0.9025906984450397, 'mean_recall': 0.8840592072689094, 'macro_f1': 0.8925740985975588, 'weighted_precision': 0.9079516195367331, 'weighted_recall': 0.9078, 'weighted_f1': 0.9077917625402113}\n",
            "Validation Results\n",
            "{'acc': 0.8429203539823009, 'mean_precision': 0.8178604107482675, 'mean_recall': 0.8027279926613361, 'macro_f1': 0.807847400976306, 'weighted_precision': 0.8541454685154969, 'weighted_recall': 0.8429203539823009, 'weighted_f1': 0.8452297642709833}\n",
            "EPOCH 16\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9114, 'mean_precision': 0.9091156125381755, 'mean_recall': 0.8954153246439646, 'macro_f1': 0.901884018226399, 'weighted_precision': 0.9120534355149411, 'weighted_recall': 0.9114, 'weighted_f1': 0.9115938328769627}\n",
            "Validation Results\n",
            "{'acc': 0.8584070796460177, 'mean_precision': 0.8452925879669096, 'mean_recall': 0.8140923455585375, 'macro_f1': 0.8271059195675882, 'weighted_precision': 0.866784367505974, 'weighted_recall': 0.8584070796460177, 'weighted_f1': 0.8601283537102449}\n",
            "New best model 0.86\n",
            "EPOCH 17\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.922, 'mean_precision': 0.92318827916946, 'mean_recall': 0.9167872228173993, 'macro_f1': 0.919907344495105, 'weighted_precision': 0.9222631334663767, 'weighted_recall': 0.922, 'weighted_f1': 0.9220983595773123}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8226381745596281, 'mean_recall': 0.8064587339697171, 'macro_f1': 0.8118331344786779, 'weighted_precision': 0.8588090104120005, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.8509990153606489}\n",
            "EPOCH 18\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.925, 'mean_precision': 0.9272713764228396, 'mean_recall': 0.9097196644145997, 'macro_f1': 0.9178903031944213, 'weighted_precision': 0.9256665310570704, 'weighted_recall': 0.925, 'weighted_f1': 0.9251760923127654}\n",
            "Validation Results\n",
            "{'acc': 0.8362831858407079, 'mean_precision': 0.8130991700945112, 'mean_recall': 0.8186032795814143, 'macro_f1': 0.8123228744939271, 'weighted_precision': 0.8468684662820305, 'weighted_recall': 0.8362831858407079, 'weighted_f1': 0.8392648607279487}\n",
            "EPOCH 19\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9226, 'mean_precision': 0.9265697322783559, 'mean_recall': 0.907365970956147, 'macro_f1': 0.9162751616718158, 'weighted_precision': 0.9229974594112873, 'weighted_recall': 0.9226, 'weighted_f1': 0.922695479769021}\n",
            "Validation Results\n",
            "{'acc': 0.8407079646017699, 'mean_precision': 0.7961368682179472, 'mean_recall': 0.8225977371214205, 'macro_f1': 0.8033500248964449, 'weighted_precision': 0.8499048273885906, 'weighted_recall': 0.8407079646017699, 'weighted_f1': 0.8433057108007154}\n",
            "EPOCH 20\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9306, 'mean_precision': 0.9325672078372992, 'mean_recall': 0.9292912885334105, 'macro_f1': 0.9308909230234336, 'weighted_precision': 0.9309373491584835, 'weighted_recall': 0.9306, 'weighted_f1': 0.9307301028725288}\n",
            "Validation Results\n",
            "{'acc': 0.8451327433628318, 'mean_precision': 0.8169164169164169, 'mean_recall': 0.8240567649306224, 'macro_f1': 0.81237323931053, 'weighted_precision': 0.8649006598121644, 'weighted_recall': 0.8451327433628318, 'weighted_f1': 0.8481262461147536}\n",
            "EPOCH 21\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9372, 'mean_precision': 0.94778588624423, 'mean_recall': 0.931739382135247, 'macro_f1': 0.9392845360347503, 'weighted_precision': 0.9376509979046739, 'weighted_recall': 0.9372, 'weighted_f1': 0.9373365710922599}\n",
            "Validation Results\n",
            "{'acc': 0.8429203539823009, 'mean_precision': 0.8122060187568131, 'mean_recall': 0.8039074327168304, 'macro_f1': 0.8074842893609094, 'weighted_precision': 0.846085726902366, 'weighted_recall': 0.8429203539823009, 'weighted_f1': 0.843907236330442}\n",
            "EPOCH 22\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9396, 'mean_precision': 0.9374128845763084, 'mean_recall': 0.919864102313256, 'macro_f1': 0.9279841844104714, 'weighted_precision': 0.9395460389520726, 'weighted_recall': 0.9396, 'weighted_f1': 0.9395077256432057}\n",
            "Validation Results\n",
            "{'acc': 0.838495575221239, 'mean_precision': 0.8166287688346512, 'mean_recall': 0.7965319356459588, 'macro_f1': 0.8036210071046136, 'weighted_precision': 0.8514493271781142, 'weighted_recall': 0.838495575221239, 'weighted_f1': 0.8411121771904451}\n",
            "EPOCH 23\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9394, 'mean_precision': 0.9343864652716637, 'mean_recall': 0.9292029606780483, 'macro_f1': 0.9317217171943124, 'weighted_precision': 0.9395732746438947, 'weighted_recall': 0.9394, 'weighted_f1': 0.9394587072607552}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8152645128671007, 'mean_recall': 0.8101445689658334, 'macro_f1': 0.8116836837706645, 'weighted_precision': 0.853045900832462, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.8499444374703017}\n",
            "EPOCH 24\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9366, 'mean_precision': 0.9385826391144599, 'mean_recall': 0.9310259128633419, 'macro_f1': 0.9346948428938683, 'weighted_precision': 0.9367440103332517, 'weighted_recall': 0.9366, 'weighted_f1': 0.9366526615949582}\n",
            "Validation Results\n",
            "{'acc': 0.8473451327433629, 'mean_precision': 0.8025432410455261, 'mean_recall': 0.8283322432914666, 'macro_f1': 0.8061102573975748, 'weighted_precision': 0.861458116214064, 'weighted_recall': 0.8473451327433629, 'weighted_f1': 0.8502841373610759}\n",
            "EPOCH 25\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9474, 'mean_precision': 0.945203027270548, 'mean_recall': 0.9397940805142109, 'macro_f1': 0.9424193517790628, 'weighted_precision': 0.9476039831561757, 'weighted_recall': 0.9474, 'weighted_f1': 0.9474674977170525}\n",
            "Validation Results\n",
            "{'acc': 0.8517699115044248, 'mean_precision': 0.8122309943824346, 'mean_recall': 0.8316675962171595, 'macro_f1': 0.817457525740406, 'weighted_precision': 0.8589009917207507, 'weighted_recall': 0.8517699115044248, 'weighted_f1': 0.8536305273856584}\n",
            "EPOCH 26\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9412, 'mean_precision': 0.9333724069841898, 'mean_recall': 0.9327119878566131, 'macro_f1': 0.9330083105008247, 'weighted_precision': 0.9415099002878724, 'weighted_recall': 0.9412, 'weighted_f1': 0.9413101950625798}\n",
            "Validation Results\n",
            "{'acc': 0.8672566371681416, 'mean_precision': 0.8605083530628996, 'mean_recall': 0.8246033707390573, 'macro_f1': 0.8391902596303691, 'weighted_precision': 0.8669141290171042, 'weighted_recall': 0.8672566371681416, 'weighted_f1': 0.8665873995194234}\n",
            "New best model 0.87\n",
            "EPOCH 27\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9452, 'mean_precision': 0.9393570838696085, 'mean_recall': 0.9381922191949007, 'macro_f1': 0.938758641015847, 'weighted_precision': 0.9451643947972791, 'weighted_recall': 0.9452, 'weighted_f1': 0.9451693027949712}\n",
            "Validation Results\n",
            "{'acc': 0.8539823008849557, 'mean_precision': 0.8230172208866048, 'mean_recall': 0.8123173810979951, 'macro_f1': 0.8156989547159269, 'weighted_precision': 0.86038962724662, 'weighted_recall': 0.8539823008849557, 'weighted_f1': 0.8546422455744229}\n",
            "EPOCH 28\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9496, 'mean_precision': 0.9460303491056411, 'mean_recall': 0.9315008817259457, 'macro_f1': 0.938347207012888, 'weighted_precision': 0.9496092730822427, 'weighted_recall': 0.9496, 'weighted_f1': 0.9495572903768849}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8264021292481143, 'mean_recall': 0.8075150926865313, 'macro_f1': 0.8142132100384322, 'weighted_precision': 0.8614722274611524, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.8519763491364317}\n",
            "EPOCH 29\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9492, 'mean_precision': 0.9523110976436452, 'mean_recall': 0.9431909498306, 'macro_f1': 0.9475782265270856, 'weighted_precision': 0.9492766331969764, 'weighted_recall': 0.9492, 'weighted_f1': 0.9492146339395731}\n",
            "Validation Results\n",
            "{'acc': 0.8517699115044248, 'mean_precision': 0.8258343506751618, 'mean_recall': 0.8103395389633586, 'macro_f1': 0.8158066635203088, 'weighted_precision': 0.860982906249134, 'weighted_recall': 0.8517699115044248, 'weighted_f1': 0.8534222923294074}\n",
            "EPOCH 30\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9486, 'mean_precision': 0.9560017443521739, 'mean_recall': 0.9400692468871957, 'macro_f1': 0.9475746328501561, 'weighted_precision': 0.9488545438754485, 'weighted_recall': 0.9486, 'weighted_f1': 0.9486620015461529}\n",
            "Validation Results\n",
            "{'acc': 0.8517699115044248, 'mean_precision': 0.8177916897275509, 'mean_recall': 0.8317336038356844, 'macro_f1': 0.8189379757266612, 'weighted_precision': 0.8641615919629556, 'weighted_recall': 0.8517699115044248, 'weighted_f1': 0.8543175095648456}\n",
            "EPOCH 31\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.958, 'mean_precision': 0.9538284266182177, 'mean_recall': 0.9446486493014933, 'macro_f1': 0.9490660805682173, 'weighted_precision': 0.9581057645690427, 'weighted_recall': 0.958, 'weighted_f1': 0.958018653218701}\n",
            "Validation Results\n",
            "{'acc': 0.8517699115044248, 'mean_precision': 0.8013986378685444, 'mean_recall': 0.8119935701678367, 'macro_f1': 0.804411328159357, 'weighted_precision': 0.8579675339835053, 'weighted_recall': 0.8517699115044248, 'weighted_f1': 0.8534679803361881}\n",
            "EPOCH 32\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.954, 'mean_precision': 0.9512729471583152, 'mean_recall': 0.9522326430845925, 'macro_f1': 0.9517306869942752, 'weighted_precision': 0.9541486554643743, 'weighted_recall': 0.954, 'weighted_f1': 0.9540543038088639}\n",
            "Validation Results\n",
            "{'acc': 0.8584070796460177, 'mean_precision': 0.8247875724959964, 'mean_recall': 0.813382140867244, 'macro_f1': 0.8148679951657951, 'weighted_precision': 0.8728418185739959, 'weighted_recall': 0.8584070796460177, 'weighted_f1': 0.8607256592741758}\n",
            "EPOCH 33\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.957, 'mean_precision': 0.9565681480304719, 'mean_recall': 0.9454327046857133, 'macro_f1': 0.9507466808370677, 'weighted_precision': 0.9571322707135252, 'weighted_recall': 0.957, 'weighted_f1': 0.9570200735546852}\n",
            "Validation Results\n",
            "{'acc': 0.8407079646017699, 'mean_precision': 0.8012295177369267, 'mean_recall': 0.7988123932835594, 'macro_f1': 0.7956541104094256, 'weighted_precision': 0.854940819905837, 'weighted_recall': 0.8407079646017699, 'weighted_f1': 0.843834784432942}\n",
            "EPOCH 34\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9562, 'mean_precision': 0.9492352170638396, 'mean_recall': 0.9501709445363726, 'macro_f1': 0.9496843838684074, 'weighted_precision': 0.9563024102937812, 'weighted_recall': 0.9562, 'weighted_f1': 0.9562374926155631}\n",
            "Validation Results\n",
            "{'acc': 0.8539823008849557, 'mean_precision': 0.8074247463353991, 'mean_recall': 0.8132105098252905, 'macro_f1': 0.8095076634215594, 'weighted_precision': 0.8571530973672192, 'weighted_recall': 0.8539823008849557, 'weighted_f1': 0.8549969022822371}\n",
            "EPOCH 35\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.958, 'mean_precision': 0.9581863087122168, 'mean_recall': 0.9556184908351323, 'macro_f1': 0.9568856509156166, 'weighted_precision': 0.9581083670874939, 'weighted_recall': 0.958, 'weighted_f1': 0.9580411865238977}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8178061479593234, 'mean_recall': 0.805632013633958, 'macro_f1': 0.807789691127139, 'weighted_precision': 0.8649796275543147, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.8526672381640437}\n",
            "EPOCH 36\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9574, 'mean_precision': 0.9531272685674982, 'mean_recall': 0.9515769491207995, 'macro_f1': 0.9523405767311283, 'weighted_precision': 0.9573866537776834, 'weighted_recall': 0.9574, 'weighted_f1': 0.9573873211612607}\n",
            "Validation Results\n",
            "{'acc': 0.8561946902654868, 'mean_precision': 0.8131151427942895, 'mean_recall': 0.8110658945033787, 'macro_f1': 0.8077795032214256, 'weighted_precision': 0.8693120626269972, 'weighted_recall': 0.8561946902654868, 'weighted_f1': 0.8589532280938094}\n",
            "EPOCH 37\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9566, 'mean_precision': 0.9589041860848261, 'mean_recall': 0.9547448481946009, 'macro_f1': 0.9567909158741493, 'weighted_precision': 0.9566010555107037, 'weighted_recall': 0.9566, 'weighted_f1': 0.9565890848568762}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8030750892649611, 'mean_recall': 0.8075017730562345, 'macro_f1': 0.8022990697875542, 'weighted_precision': 0.858293669895924, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.851768488915397}\n",
            "EPOCH 38\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.957, 'mean_precision': 0.9544796723873579, 'mean_recall': 0.9522358611983669, 'macro_f1': 0.9533352482886838, 'weighted_precision': 0.9571066919837921, 'weighted_recall': 0.957, 'weighted_f1': 0.9570317579959188}\n",
            "Validation Results\n",
            "{'acc': 0.8539823008849557, 'mean_precision': 0.8106297488632482, 'mean_recall': 0.8123418403963019, 'macro_f1': 0.8098548414705528, 'weighted_precision': 0.8585169985936104, 'weighted_recall': 0.8539823008849557, 'weighted_f1': 0.8548092957757126}\n",
            "EPOCH 39\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9564, 'mean_precision': 0.9532129259082603, 'mean_recall': 0.9506704508751782, 'macro_f1': 0.951926709753589, 'weighted_precision': 0.9564913201918928, 'weighted_recall': 0.9564, 'weighted_f1': 0.9564356136646421}\n",
            "Validation Results\n",
            "{'acc': 0.8539823008849557, 'mean_precision': 0.8017848860212725, 'mean_recall': 0.8119553221526408, 'macro_f1': 0.804904791937732, 'weighted_precision': 0.8588867681742558, 'weighted_recall': 0.8539823008849557, 'weighted_f1': 0.855560943534421}\n",
            "EPOCH 40\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9644, 'mean_precision': 0.9670062774145092, 'mean_recall': 0.9573277463956331, 'macro_f1': 0.9619863456379548, 'weighted_precision': 0.9645118227152805, 'weighted_recall': 0.9644, 'weighted_f1': 0.9644225052829172}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8119564587793305, 'mean_recall': 0.8080099729691964, 'macro_f1': 0.8081568266024983, 'weighted_precision': 0.8580128676213757, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.8519747328540073}\n",
            "EPOCH 41\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9628, 'mean_precision': 0.9582207266041486, 'mean_recall': 0.9557915435105366, 'macro_f1': 0.956989832500887, 'weighted_precision': 0.9628821971783511, 'weighted_recall': 0.9628, 'weighted_f1': 0.9628296736582989}\n",
            "Validation Results\n",
            "{'acc': 0.8451327433628318, 'mean_precision': 0.7965093061348186, 'mean_recall': 0.8045728236987139, 'macro_f1': 0.7982598158685201, 'weighted_precision': 0.8513942451058316, 'weighted_recall': 0.8451327433628318, 'weighted_f1': 0.8470011392375405}\n",
            "EPOCH 42\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.964, 'mean_precision': 0.9645740109204436, 'mean_recall': 0.9606825654964162, 'macro_f1': 0.9625915166364768, 'weighted_precision': 0.9640606955680181, 'weighted_recall': 0.964, 'weighted_f1': 0.9640155053392376}\n",
            "Validation Results\n",
            "{'acc': 0.8495575221238938, 'mean_precision': 0.8037359987261851, 'mean_recall': 0.8058441622200901, 'macro_f1': 0.8012095203517711, 'weighted_precision': 0.8602910880270568, 'weighted_recall': 0.8495575221238938, 'weighted_f1': 0.8518937207419582}\n",
            "EPOCH 43\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.967, 'mean_precision': 0.9673205256002168, 'mean_recall': 0.9608809490156882, 'macro_f1': 0.9640313629040639, 'weighted_precision': 0.967090742201184, 'weighted_recall': 0.967, 'weighted_f1': 0.9670273824588701}\n",
            "Validation Results\n",
            "{'acc': 0.8584070796460177, 'mean_precision': 0.8071223611395778, 'mean_recall': 0.815180322836991, 'macro_f1': 0.8086659140417566, 'weighted_precision': 0.8654518199826157, 'weighted_recall': 0.8584070796460177, 'weighted_f1': 0.8603401318409323}\n",
            "EPOCH 44\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9658, 'mean_precision': 0.9609493483829002, 'mean_recall': 0.9637935616364777, 'macro_f1': 0.9623305485433838, 'weighted_precision': 0.9659282069261604, 'weighted_recall': 0.9658, 'weighted_f1': 0.9658413169266654}\n",
            "Validation Results\n",
            "{'acc': 0.8451327433628318, 'mean_precision': 0.811964839602692, 'mean_recall': 0.804398454041185, 'macro_f1': 0.8075118081867685, 'weighted_precision': 0.8495195126057683, 'weighted_recall': 0.8451327433628318, 'weighted_f1': 0.846453823025803}\n",
            "EPOCH 45\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9658, 'mean_precision': 0.9598439053443757, 'mean_recall': 0.9526976401858064, 'macro_f1': 0.9561667547996912, 'weighted_precision': 0.9657984947117778, 'weighted_recall': 0.9658, 'weighted_f1': 0.9657855836823659}\n",
            "Validation Results\n",
            "{'acc': 0.8539823008849557, 'mean_precision': 0.8027867894992161, 'mean_recall': 0.8119553221526408, 'macro_f1': 0.8053321278501304, 'weighted_precision': 0.8592071325485433, 'weighted_recall': 0.8539823008849557, 'weighted_f1': 0.8556693507865624}\n",
            "EPOCH 46\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9662, 'mean_precision': 0.9604700540065974, 'mean_recall': 0.9625075307979163, 'macro_f1': 0.9614812705685866, 'weighted_precision': 0.9662056816206401, 'weighted_recall': 0.9662, 'weighted_f1': 0.966200656893312}\n",
            "Validation Results\n",
            "{'acc': 0.8606194690265486, 'mean_precision': 0.8296025254745819, 'mean_recall': 0.8174572008429252, 'macro_f1': 0.8224370751844012, 'weighted_precision': 0.8663240618203951, 'weighted_recall': 0.8606194690265486, 'weighted_f1': 0.8621524978508673}\n",
            "EPOCH 47\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9658, 'mean_precision': 0.9623643280997959, 'mean_recall': 0.9601608706065794, 'macro_f1': 0.9612521277633985, 'weighted_precision': 0.9657974850075743, 'weighted_recall': 0.9658, 'weighted_f1': 0.9657933462320052}\n",
            "Validation Results\n",
            "{'acc': 0.8606194690265486, 'mean_precision': 0.8289130643629324, 'mean_recall': 0.8162020131702755, 'macro_f1': 0.8212697994296662, 'weighted_precision': 0.8663358703067995, 'weighted_recall': 0.8606194690265486, 'weighted_f1': 0.8619390343788721}\n",
            "EPOCH 48\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9694, 'mean_precision': 0.965498989334344, 'mean_recall': 0.9691871765140303, 'macro_f1': 0.9673126581580327, 'weighted_precision': 0.969441121137196, 'weighted_recall': 0.9694, 'weighted_f1': 0.969411138333605}\n",
            "Validation Results\n",
            "{'acc': 0.8473451327433629, 'mean_precision': 0.8108888178689653, 'mean_recall': 0.8042876327132501, 'macro_f1': 0.8053138141785771, 'weighted_precision': 0.8569467358619901, 'weighted_recall': 0.8473451327433629, 'weighted_f1': 0.8498193213999172}\n",
            "Final result\n",
            "{'acc': 0.882, 'mean_precision': 0.8966656639557661, 'mean_recall': 0.8599837937999005, 'macro_f1': 0.8737226125229776, 'weighted_precision': 0.8813809951547577, 'weighted_recall': 0.882, 'weighted_f1': 0.878768700658464}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.882"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iPWVd-iW1mMG",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "Without even concatenating word features, our ELMo model, with far fewer parameters, surpasses the performance of the randomly initialized baseline, which we would expect.  It also significantly out-performs our CNN pre-trained, fine-tuned word embeddings baseline from the last section -- that model's max performance is around 93.  Note that this dataset is tiny, and the variance is large between datasets, but this model consistently outperforms both CNN and LSTM baselines.\n",
        "\n",
        "Contextual embeddings consistently outperform non-contextual embeddings on almost every task in NLP, not just in text classification.  This method is becoming so commonly used that some papers have even started reporting this approach as a baseline.\n",
        "\n",
        "### Some more references\n",
        "\n",
        "- The PyTorch examples actually contain a [nice word-language model](https://github.com/pytorch/examples/tree/master/word_language_model)\n",
        "\n",
        "- There is a [Tensorflow tutorial](https://www.tensorflow.org/tutorials/sequences/recurrent) as well\n",
        "\n",
        "- The original source code for training [ELMo's bilm is here](https://github.com/allenai/bilm-tf/tree/master/bilm)\n",
        "\n",
        "- [A succinct implementation](https://github.com/dpressel/baseline/blob/master/python/baseline/pytorch/embeddings.py#L63) of character-compositional embeddings in Baseline for PyTorch\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ]
}