NLP_NMT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLP_NMT.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XYpOsk3hmO9w",
        "colab_type": "text"
      },
      "source": [
        "# **Neural Machine Translation**\n",
        "This notebook allows to train and interactively use a Neural Network model for the sequence to sequence translation of German sentences to English. The IWSLT14 German-English dataset is utilised to train and evaluate the performance of the model.\n",
        "\n",
        "The BLEU Score of the trained model will also evaluate its translation quality. The method presented here is a state-of-the-art model when it comes to German-to-English translation.\n",
        "\n",
        "The same provided code can be used to train any other language pair. However, some language pairs might be more challenging to train, as they require a more complex network architecture and a larger training dataset. In such cases, one GPU is not enough and usually around 8 GPUs are for example required to train an English-to-German model.\n",
        "\n",
        "> ***Note***: There will be no mathematical expressions included in this notebook to make it easy to comprehend and follow.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ewlz35VAFqlT",
        "colab_type": "text"
      },
      "source": [
        "##**1. Evolution of Machine Translation**\n",
        "The [timeline below](https://towardsdatascience.com/evolution-of-machine-translation-5524f1c88b25) shows a concise history of machine translation. However, since only Neural Machine Translation NMT is within the scope of this notebook, we will not go into detail about the different machine translation phases before the adoption of neural networks. \n",
        "\n",
        "A significant achievement in Statistical Machine Translation SMT was the introduction of neural networks. NMT has shown better results than SMT and is considered as the future of machine translation.\n",
        "SMT utilizes statistical methods to represent the translation patterns, whereas Neural MT, as the name suggests, requires training a Deep Neural Network to learn a statistical model for the translation.\n",
        "\n",
        "<center><img src=https://miro.medium.com/max/1400/1*XuR_iuPOuY-8i5A3cGmcBw.png width=\"700\"/></center>\n",
        "\n",
        "\n",
        "There exist [various types of neural architectures](http://www.cse.iitd.ac.in/~mausam/courses/col772/spring2018/lectures/21-seq2seq.pdf) that can handle different combinations of inputs and outputs involving sequences, e.g., CNNs or BiLSTM acceptors to map a sequence to a single decision, Siamese networks for mapping between two sequences and a single decision and BiLSTM transducers that are networks for same length sequence to sequence mapping. However, for the task of machine translation, the model is required to handle an output sequence with a different length than the input sequence.\n",
        "\n",
        "The following [timeline](https://medium.com/@bgg/seq2seq-pay-attention-to-self-attention-part-1-d332e85e9aad) highlights the most important milestones in NMT, which we will briefly review and explain.\n",
        "\n",
        "<center><img src=https://miro.medium.com/max/1400/1*C_ERqLXFW0MZ8IUuPbR19A.png width=\"700\"/></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SxdEgH5SWcAR",
        "colab_type": "text"
      },
      "source": [
        "###**[1.1. Recurrent Neural Networks](https://leonardoaraujosantos.gitbook.io/artificial-inteligence/machine_learning/supervised_learning/recurrent_neural_networks)**\n",
        "Some of the first neural methods that have been widely used but are now considered to be less capable are for example multi-layered Recurrent Neural Networks (RNNs). This structure can process sequential input and output data, as it has a **notion of time or sequence**. RNNs are unrolled as a feedforward network with shared weights. By having the same parameters for each time-step, they can learn a more general model for all steps. The output of step $i$ is input to step $i+1$, which makes each output dependent on all previous elements in the sequence. \n",
        "\n",
        "*   **Recurrent Neural Networks**  \n",
        "Because the weight are multiplied over and over again, vanilla RNNs suffer from the vanishing and exploding gradients problem when learning long-term dependencies. The later is easier to solve using gradient clipping. In addition, RNNs are slow learner because they cannot be parallelized since they are processed sequentially over time.\n",
        "\n",
        "*   **Long Short-Term Memory**  \n",
        "The vanishing gradient problem of vanilla RNN was alleviated thanks to Long Short-Term Memory (LSTM) modules which are similar to a RestNet. LSTM can remember the information from older past cells and thus helps maintain long-term dependencies. The additional cell state is responsible for the transport of the information through the unit, as it provides a highway for the gradient to flow.\n",
        "LSTMs are therefore more powerful learners of sequential data compared to simple RNNs.  \n",
        "A tutorial of a simple LSTM that is very similar to this one, as it makes use of the fairseq toolkit can be found [here](https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html#).\n",
        "\n",
        "*   **Gated Recurrent Unit**  \n",
        "Gated Recurrent Unit (GRU) cells are a more computationally efficient variant of the LSTM module.\n",
        "\n",
        "The cells of the different types of Recurrent Neural Networks can be seen [here](http://dprogrammer.org/rnn-lstm-gru):\n",
        "<center><img src=http://dprogrammer.org/wp-content/uploads/2019/04/RNN-vs-LSTM-vs-GRU.png width=\"1000\"/></center>\n",
        "\n",
        "The [illustration below](https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/) shows an example of an unrolled sequence to sequence RNN used for translation. This simple encoder-decoder model obviously will not perform well with sentence longer than 3-4 words. A more in depth explanation of the model as well as various ways to improve it can be found [here](https://leonardoaraujosantos.gitbook.io/artificial-inteligence/machine_learning/supervised_learning/recurrent_neural_networks/machine-translation-using-rnn).\n",
        "\n",
        "<center><img src=https://cdn.analyticsvidhya.com/wp-content/uploads/2019/06/seq2seq.gif width=\"600\"/></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xB0jFKHcb-u",
        "colab_type": "text"
      },
      "source": [
        "###**[1.2. Attention Models](https://medium.com/@gautam.karmakar/attention-for-neural-connectionist-machine-translation-b833d1e085a3)**\n",
        "One big disadvantage of the RNN is that all the information perceived by the encoder is stored in a single intermediate (encoder) vector which then serves as a first hidden state of the decoder. This results in RNNs' incapibility to return good translations of long sentences.  \n",
        "That being said, attention enables neural machine translation to memorize long source sentences. To achieve this, we no longer only build one encoder vector from the encoder’s last hidden state. Instead, we create as much context vectors as there are words in the source sentence. As can be seen [below](https://medium.com/@umerfarooq_26378/neural-machine-translation-with-code-68c425044bbd), at each decoding stage, attention passes an attention score (the context vector) to the decoder, which equals a learned weighted sum of all hidden states.\n",
        "By enabling the model to weight elements of the source sentence differently and incorporate information from older inputs, we can capture much more information from the encoder side.    \n",
        "Today, all state-of-the-art architectures use some sort of attention, which has been found to be a very effective method for sequence-to-sequence tasks. Another advantage of attention is the better interpretability of models.\n",
        "\n",
        "<center><img src=https://miro.medium.com/max/2100/1*75Jb0q3sX1GDYmJSfl-gOw.gif width=\"600\"/></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B31R-d8kNPJ1",
        "colab_type": "text"
      },
      "source": [
        "###**[1.3. Convolutional Neural Networks](https://engineering.fb.com/ml-applications/a-novel-approach-to-neural-machine-translation/)**\n",
        "\n",
        "Convolutions are the application of a filter to a function, where the smaller one is called the filter kernel. Convolutional layers were mostly implemented with datasets containing images, e.g., image classification, object detection and semantic segmentation.\n",
        "However, Convolutional Neural Networks (CNNs) were later adopted instead of RNNs, by interpreting a sentence similar to a 1D image. CNN-based models not only improve the translation performance  thanks to the hierarchical processing of information, but they are also parallelizable, which makes them computationally more efficient. \n",
        "\n",
        "The [following illustration](https://engineering.fb.com/ml-applications/a-novel-approach-to-neural-machine-translation/) shows the architecture designed by the Facebook AI Research team that has the multi-hop attention mechanism to thank for its improved results. Here the encoder CNN processes all the words from the source sentence simultaneously. Afterwards, the decoder sequentially outputs the translated words using CNNs and attention.\n",
        "<center><img src=https://engineering.fb.com/wp-content/uploads/2017/05/translation_illustration.gif?w=640 width=\"700\"/></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-UofSC-DB5F",
        "colab_type": "text"
      },
      "source": [
        "##**2. Self-Attention and Transformers** \n",
        "To understand how the model architecture we are using works, we must first understand what Transformers are and how they work. But let's start with self-attention, since Transformers rely entirely on this mechanism."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kwPfv21SLDez",
        "colab_type": "text"
      },
      "source": [
        "###**2.1. From Attention to Self-Attention**\n",
        "\n",
        "We already encountered the encoder-decoder attention. The disadvantage of such attention mechanism is that it cannot be parallelized, and it ignores the attention information inside the source sentence as well as the target sentence. So, how is self-attention different?\n",
        "\n",
        "The first important thing to know is that the encoding and decoding blocks are usually a stack of the same number of encoders and decoders respectively ([see illustration](http://jalammar.github.io/illustrated-transformer/)).\n",
        "<center><img src=http://jalammar.github.io/images/t/The_transformer_encoder_decoder_stack.png width=\"500\"/></center> \n",
        "\n",
        "Let's take a look at [this example](http://jalammar.github.io/illustrated-transformer/) for a second. We, as humans, can easily see that the word \"it\" in this sentence stands for \"animal\" and not \"street\". However, an algorithm would need a little help, and that's where self-attention comes. Using the self-attention mechanism helps better encode each input words by enriching it with context words from the other positions in the input sequence.\n",
        "\n",
        "<center><img src=http://jalammar.github.io/images/t/transformer_self-attention_visualization.png width=\"400\"/></center>\n",
        "\n",
        "The idea behind self-attention is to use a special case of attention between the different encoder layers and decoder layers respectively. On the encoder side, self-attention is computed between an input token and other input tokens, whereas for the decoder, the attention score is calculated using an output word and the previously produced output words."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UK9761m8LcbZ",
        "colab_type": "text"
      },
      "source": [
        "###**2.2. From Self-Attention to Transformers**\n",
        "\n",
        "The encoder–decoder architecture that is currently dominating in many NLP tasks is the [Transformer](https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf) model [displayed below](https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04).\n",
        "\n",
        "<center><img src=https://miro.medium.com/max/1400/1*BHzGVskWGS_3jEcYYi6miQ.png width=\"400\"/></center>\n",
        "\n",
        "As we mentionned before, the encoder block consists of serveral encoders of identical structure. Each encoder is composed of a self-attention layer followed by a feed-forward neural network. There are also residual connections and normalization layers in the encoder. An important note is that the individual encoders do not share weights.  \n",
        "The decoder is also built similarly. We can also see the self-attention layer, the feed-forward neural network, as well as the residual connection and the normalization layer. The only difference is that the self-attention layer is only allowed to attend to previously generated target tokens, by masking future positions.  \n",
        "The encoder-decoder attention mechanism is also utilized between the last layer of the encoder and each layer in the decoder.  \n",
        "In the architecture figure we saw before, the Self-Attention block was calles Multi-head Attention: this simply means that Self-Attention is computed multiple times in parallel and independently between one encoder and the next.  \n",
        "After the last decoder layer, we must convert the float vectors into output words. We first use a linear layer to get a larger logits vector. This vector contains scores for each unique word in our output vocabulary. Afterwards, these scores are tranformed into probabilities using the Softmax layer and the output word for that time step is the word corresponding to the highest probability.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BtQ5nxFHtMOv",
        "colab_type": "text"
      },
      "source": [
        "##**3. Joint Source-Target Self Attention with Locality Constraints** \n",
        "This Notebook is based on the work done by José A. R. Fonollosa, Noe Casas, and Marta R. Costa-jussá in [\"Joint Source-Target Self Attention with Locality Constraints\"](http://arxiv.org/abs/1905.06596) and uses their code provided in https://github.com/jarfo/joint.\n",
        "\n",
        "> ***Note***: This takes around 5 hours to train on Google Colab using the provided GPU.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_nC-CjOwjkO",
        "colab_type": "text"
      },
      "source": [
        "###**3.1. Model Architecture**###\n",
        "The following [model architecture](https://arxiv.org/abs/1905.06596) and the name of the paper kind of seems a little complicated at first sight. This model architecture combines the ideas of two papers: [\"*Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation*\"](https://pdfs.semanticscholar.org/005c/d149aa86b1be8a22706c8d29095bbf46d192.pdf?_ga=2.75612565.1449333594.1596232318-210385577.1596232318) and [\"*Pay Less Attention with Lightweight and Dynamic Convolutions*\"](https://arxiv.org/pdf/1901.10430.pdf). Let’s now go through this architecture step by step. \n",
        "\n",
        "<center><img src=https://d3i71xaburhd42.cloudfront.net/b0f0a5a21619d70748a4dc007983cc111f1b301e/2-Figure1-1.png width=\"700\"/></center>\n",
        "\n",
        "As input to the network, we provide both the source and the target sequences concatenated together. The idea behind such a concept is to train the model as a language model. You might ask: but what is a language model?  \n",
        "We all encounter the most famous language model everyday but might not know that it actually is one. I'm talking about smartphone keyboards, which suggest different next words based on what has been typed so far. What language models do is to calculate the probabilities of the next words depending on previous ones, and the word with the highest probability is chosen, as can be [seen below](http://jalammar.github.io/illustrated-word2vec/).  \n",
        "The advantage of such models is that it allows to learn a joint source-target representations from the early layers.\n",
        "\n",
        "<center><img src=http://jalammar.github.io/images/word2vec/language_model_blackbox_output_vector.png width=\"600\"/></center>\n",
        "\n",
        "The next question you might ask yourself is: How do we ensure this language model-like training?  \n",
        "The answer to that is by combining the encoder and decoder into a single block and that way we no longer use an independent encoder. \n",
        "A layer-wise coordination of the Transformer results in two modification. The individual decoder layers now attend to their corresponding layer in the encoder. This is different from the concept of the Transformer where each decoder layer attends to the last layer of the encoder. The second alteration aims at ensuring that the outputs of the encoder and the decoder from the same level are also in the same semantic level. This is realised by sharing the parameters of the attention and the feed-forward layer are shared between the encoder and decoder.\n",
        "\n",
        "As a consequence of the merging of the encoder and decoder, we no longer need an encoder-decoder attention mechanism to extract information from the source sequence. Instead, we use a so-called mix-attention on the decoder side, which is a combination of both the encoder-decoder attention and self-attention. This mechanism allows target tokens from the decoder to attend to all source tokens as well as the preceding target tokens ([see figure](https://pdfs.semanticscholar.org/005c/d149aa86b1be8a22706c8d29095bbf46d192.pdf?_ga=2.75612565.1449333594.1596232318-210385577.1596232318)).\n",
        "\n",
        "<center><img src=https://d3i71xaburhd42.cloudfront.net/b12ccd118974839db290f15c989649b2b5188636/4-Figure1-1.png width=\"600\"/></center>\n",
        "\n",
        "Because Tranformers are not based on recurrent operations like RNNs and because of the joint representation and the layer-wise coordinated Transformer that results in weight sharing, new problems arise. \n",
        "How will the network be able to remember the word ordering in the sentence and differentiate the source language (German tokens) from the target ones (English tokens)?  \n",
        "The first problem is solved by giving explicit information about the position of the different words in the sequence. This is known as position embedding and is achieved with the help of a sinusoidal function. This way, a resettable position embedding ensures that the source tokens start with zero and is then resetted to zero when it encounters the first token of the target sequence.  \n",
        "However, the position embedding alone is not capable to identify to which language the tokens belong to. For this reason, we make use of a second embedding:  Differnet from the positional embedding, this source/target embedding is learned end-to-end during training.\n",
        "\n",
        "If you look closer at the actual figure of the model architecture (first figure in this section), you can see that the self-attention and the mixed-attention are different from the figure above. This is because we also impose locality constraints to the receptive field of the self-attention layers. Such a locality constraints helps enhance the ability of capturing useful local context by attending only to a reduced number of tokens in the vicinity of the actual token we're currently looking at. On the source side, the reduction of the receptive field is centered on the encoder at each token, whereas on the target side, we only attend to previous tokens. This approach is known as masking, and it is similar to the one used for the decoder of the Transformer, where we also do not consider future target tokens. The band with the specified receptive field size as width grows after each attention layer.\n",
        "\n",
        "During inference, the model generates the translated sequence by outputting one token at a time, from left to right. This works similar to a Model Language that interprets the source sequence as starting point.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AMji9dFidKzs",
        "colab_type": "text"
      },
      "source": [
        "###**3.2.   Requirements**###\n",
        "\n",
        "For model building, training and translation, the fairseq toolkit is used. Fairseq is a sequence-to-sequence modeling toolkit implemented in PyTorch by Facebook AI Research. It enables to build and train custom models for translation purposes, among others. \n",
        "\n",
        "  1.   Check the Python version. Required **Python >= 3.6.**\n",
        "  2.   Check the Pytorch version. Required **Pytorch >= 1.4.0.**\n",
        "  3.   Install **fairseq >= 0.6.2**.\n",
        "  4.   Fairseq requires a **GPU** for faster training. Turn on the GPU on Google Colab: `Edit > Notebook settings > Hardware accelerator: GPU`. For better and more efficient results, you can also use **NCCL**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mEQeIMhBdCLH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 581
        },
        "outputId": "0ff1188e-b2ce-4f94-cbd1-a43b0a31b3b8"
      },
      "source": [
        "!python --version\n",
        "\n",
        "import torch\n",
        "print(torch.__version__)\n",
        "\n",
        "# Install fairseq from source in the same location as the notebook\n",
        "%cd \"/content/\"\n",
        "!git clone https://github.com/pytorch/fairseq\n",
        "%cd fairseq\n",
        "!pip install --editable ."
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Python 3.6.9\n",
            "1.6.0+cu101\n",
            "/content\n",
            "Cloning into 'fairseq'...\n",
            "remote: Enumerating objects: 31, done.\u001b[K\n",
            "remote: Counting objects: 100% (31/31), done.\u001b[K\n",
            "remote: Compressing objects: 100% (21/21), done.\u001b[K\n",
            "remote: Total 16937 (delta 14), reused 22 (delta 10), pack-reused 16906\u001b[K\n",
            "Receiving objects: 100% (16937/16937), 7.82 MiB | 26.97 MiB/s, done.\n",
            "Resolving deltas: 100% (12482/12482), done.\n",
            "/content/fairseq\n",
            "Obtaining file:///content/fairseq\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from fairseq==0.9.0) (1.6.0+cu101)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from fairseq==0.9.0) (1.18.5)\n",
            "Collecting sacrebleu\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/23/d3/be980ad7cda7c4bbfa97ee3de062fb3014fc1a34d6dd5b82d7b92f8d6522/sacrebleu-1.4.13-py3-none-any.whl (43kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 2.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from fairseq==0.9.0) (4.41.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from fairseq==0.9.0) (2019.12.20)\n",
            "Requirement already satisfied: cython in /usr/local/lib/python3.6/dist-packages (from fairseq==0.9.0) (0.29.21)\n",
            "Requirement already satisfied: cffi in /usr/local/lib/python3.6/dist-packages (from fairseq==0.9.0) (1.14.1)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch->fairseq==0.9.0) (0.16.0)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/89/a6/3814b7107e0788040870e8825eebf214d72166adf656ba7d4bf14759a06a/portalocker-2.0.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi->fairseq==0.9.0) (2.20)\n",
            "Installing collected packages: portalocker, sacrebleu, fairseq\n",
            "  Running setup.py develop for fairseq\n",
            "Successfully installed fairseq portalocker-2.0.0 sacrebleu-1.4.13\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BAY23F4iKWwb",
        "colab_type": "text"
      },
      "source": [
        "###**3.3. Dataset Download and Preparation**\n",
        "\n",
        "As mentioned above, we will train our model on the standard benchmark dataset IWSLT14 German-English, which contains $160$K training sentence pairs. This choice of the data set was restricted by the available processing power. \n",
        "\n",
        "An important task to perform before training with any possibly unstructured data, e.g. text data, is the pre-proceesing step. A pre-processing script `prepare-iwslt14.sh` for the IWSLT14 German-English corpus is already provided by fairseq. \n",
        "That file was modified by the authors to extend the vocabulary to $31$K joint source target BPE tokens.\n",
        "\n",
        "This pre-processing file includes:\n",
        "\n",
        "*   **Text Cleaning**  \n",
        "This step includes converting the text to lower case, removing empty lines and redundant space characters and only accepting sentences that are within a limited sentence length. It is realized with the [`lowercase.perl`](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/lowercase.perl) and [`clean-corpus-n.perl`](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/training/clean-corpus-n.perl) scripts from the [Moses](https://github.com/moses-smt/mosesdecoder) toolkit.\n",
        "\n",
        "*   **Tokenization**  \n",
        "The Moses [`tokenizer.perl`](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl) separates punctuation from words, with the exception of special instances such as URL or dates. A text string is thus segmented into a list of tokens (mostly words), for example, tokenizing the sentence `'Hello World!'` returns `['Hello', 'World', '!']`.\n",
        "\n",
        "*   **Byte Pair Encoding (BPE)**  \n",
        "The second type of tokenization that is applied afterwards is BPE. It is a special type of tokenization and is one of various subword tokenization methods.  \n",
        "It is self-evident that in any language some words occur more often than others. For this reason, we want the words that are less frequent in the dataset not to have their own identifier. This is mainly because a larger vocabulary will slow down the model. Therefore, the idea of all subword tokenizers is to decompose less frequent words into subword units while preserving their meaning. An example of a subword tokenization is to partition the word \"loving\" into \"lov\" and \"ing\" and the word \"loving\" into \"lov\" and \"ed\". As a result, the model has a reasonably sized vocabulary and can nevertheless be generalized to new words.  \n",
        "For more information about the main tokenization algorithms, you can check out this [post](https://mlexplained.com/2019/11/06/a-deep-dive-into-the-wonderful-world-of-preprocessing-in-nlp/). The repository utilized for this step can also be found [here](https://github.com/rsennrich/subword-nmt.git).\n",
        "\n",
        "We first download the IWSLT14 German-English pre-processing script from the Github repository of the authors of the paper, and then run it. To better understand the individual steps of the pre-processing task, you may want to take a closer look at the `prepare-iwslt14-31K.sh` file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cq56dSD4hfyH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "da8a3391-202a-4076-fec6-739da40e5967"
      },
      "source": [
        "%cd /content/\n",
        "\n",
        "# Download pre-processing script\n",
        "!wget https://raw.githubusercontent.com/jarfo/joint/master/examples/prepare-iwslt14-31K.sh\n",
        "\n",
        "# Dataset download and preparation \n",
        "!bash /content/prepare-iwslt14-31K.sh"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "--2020-08-03 20:52:24--  https://raw.githubusercontent.com/jarfo/joint/master/examples/prepare-iwslt14-31K.sh\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2970 (2.9K) [text/plain]\n",
            "Saving to: ‘prepare-iwslt14-31K.sh’\n",
            "\n",
            "prepare-iwslt14-31K 100%[===================>]   2.90K  --.-KB/s    in 0s      \n",
            "\n",
            "2020-08-03 20:52:24 (52.1 MB/s) - ‘prepare-iwslt14-31K.sh’ saved [2970/2970]\n",
            "\n",
            "Cloning Moses github repository (for tokenization scripts)...\n",
            "Cloning into 'mosesdecoder'...\n",
            "remote: Enumerating objects: 58, done.\u001b[K\n",
            "remote: Counting objects: 100% (58/58), done.\u001b[K\n",
            "remote: Compressing objects: 100% (48/48), done.\u001b[K\n",
            "remote: Total 147572 (delta 29), reused 21 (delta 9), pack-reused 147514\u001b[K\n",
            "Receiving objects: 100% (147572/147572), 129.76 MiB | 22.13 MiB/s, done.\n",
            "Resolving deltas: 100% (114014/114014), done.\n",
            "Cloning Subword NMT repository (for BPE pre-processing)...\n",
            "Cloning into 'subword-nmt'...\n",
            "remote: Enumerating objects: 576, done.\u001b[K\n",
            "remote: Total 576 (delta 0), reused 0 (delta 0), pack-reused 576\u001b[K\n",
            "Receiving objects: 100% (576/576), 233.12 KiB | 11.10 MiB/s, done.\n",
            "Resolving deltas: 100% (349/349), done.\n",
            "Downloading data from https://wit3.fbk.eu/archive/2014-01/texts/de/en/de-en.tgz...\n",
            "--2020-08-03 20:52:37--  https://wit3.fbk.eu/archive/2014-01/texts/de/en/de-en.tgz\n",
            "Resolving wit3.fbk.eu (wit3.fbk.eu)... 217.77.80.8\n",
            "Connecting to wit3.fbk.eu (wit3.fbk.eu)|217.77.80.8|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 19982877 (19M) [application/x-gzip]\n",
            "Saving to: ‘de-en.tgz’\n",
            "\n",
            "de-en.tgz           100%[===================>]  19.06M  5.57MB/s    in 6.1s    \n",
            "\n",
            "2020-08-03 20:52:44 (3.11 MB/s) - ‘de-en.tgz’ saved [19982877/19982877]\n",
            "\n",
            "Data successfully downloaded.\n",
            "de-en/\n",
            "de-en/IWSLT14.TED.dev2010.de-en.de.xml\n",
            "de-en/IWSLT14.TED.dev2010.de-en.en.xml\n",
            "de-en/IWSLT14.TED.tst2010.de-en.de.xml\n",
            "de-en/IWSLT14.TED.tst2010.de-en.en.xml\n",
            "de-en/IWSLT14.TED.tst2011.de-en.de.xml\n",
            "de-en/IWSLT14.TED.tst2011.de-en.en.xml\n",
            "de-en/IWSLT14.TED.tst2012.de-en.de.xml\n",
            "de-en/IWSLT14.TED.tst2012.de-en.en.xml\n",
            "de-en/IWSLT14.TEDX.dev2012.de-en.de.xml\n",
            "de-en/IWSLT14.TEDX.dev2012.de-en.en.xml\n",
            "de-en/README\n",
            "de-en/train.en\n",
            "de-en/train.tags.de-en.de\n",
            "de-en/train.tags.de-en.en\n",
            "pre-processing train data...\n",
            "Tokenizer Version 1.1\n",
            "Language: de\n",
            "Number of threads: 8\n",
            "\n",
            "Tokenizer Version 1.1\n",
            "Language: en\n",
            "Number of threads: 8\n",
            "\n",
            "clean-corpus.perl: processing iwslt14.tokenized.31K.de-en/tmp/train.tags.de-en.tok.de & .en to iwslt14.tokenized.31K.de-en/tmp/train.tags.de-en.clean, cutoff 1-175, ratio 1.5\n",
            "..........(100000).......\n",
            "Input sentences: 174443  Output sentences:  167522\n",
            "pre-processing valid/test data...\n",
            "orig/de-en/IWSLT14.TED.dev2010.de-en.de.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.dev2010.de-en.de\n",
            "Tokenizer Version 1.1\n",
            "Language: de\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.tst2010.de-en.de.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.tst2010.de-en.de\n",
            "Tokenizer Version 1.1\n",
            "Language: de\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.tst2011.de-en.de.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.tst2011.de-en.de\n",
            "Tokenizer Version 1.1\n",
            "Language: de\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.tst2012.de-en.de.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.tst2012.de-en.de\n",
            "Tokenizer Version 1.1\n",
            "Language: de\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TEDX.dev2012.de-en.de.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TEDX.dev2012.de-en.de\n",
            "Tokenizer Version 1.1\n",
            "Language: de\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.dev2010.de-en.en.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.dev2010.de-en.en\n",
            "Tokenizer Version 1.1\n",
            "Language: en\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.tst2010.de-en.en.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.tst2010.de-en.en\n",
            "Tokenizer Version 1.1\n",
            "Language: en\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.tst2011.de-en.en.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.tst2011.de-en.en\n",
            "Tokenizer Version 1.1\n",
            "Language: en\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TED.tst2012.de-en.en.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TED.tst2012.de-en.en\n",
            "Tokenizer Version 1.1\n",
            "Language: en\n",
            "Number of threads: 8\n",
            "\n",
            "orig/de-en/IWSLT14.TEDX.dev2012.de-en.en.xml iwslt14.tokenized.31K.de-en/tmp/IWSLT14.TEDX.dev2012.de-en.en\n",
            "Tokenizer Version 1.1\n",
            "Language: en\n",
            "Number of threads: 8\n",
            "\n",
            "creating train, valid, test...\n",
            "learn_bpe.py on iwslt14.tokenized.31K.de-en/tmp/train.en-de...\n",
            "subword-nmt/learn_bpe.py:332: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "apply_bpe.py to train.de...\n",
            "subword-nmt/apply_bpe.py:396: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "subword-nmt/apply_bpe.py:416: ResourceWarning: unclosed file <_io.TextIOWrapper name='iwslt14.tokenized.31K.de-en/code' mode='r' encoding='UTF-8'>\n",
            "  args.codes = codecs.open(args.codes.name, encoding='utf-8')\n",
            "apply_bpe.py to valid.de...\n",
            "subword-nmt/apply_bpe.py:396: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "subword-nmt/apply_bpe.py:416: ResourceWarning: unclosed file <_io.TextIOWrapper name='iwslt14.tokenized.31K.de-en/code' mode='r' encoding='UTF-8'>\n",
            "  args.codes = codecs.open(args.codes.name, encoding='utf-8')\n",
            "apply_bpe.py to test.de...\n",
            "subword-nmt/apply_bpe.py:396: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "subword-nmt/apply_bpe.py:416: ResourceWarning: unclosed file <_io.TextIOWrapper name='iwslt14.tokenized.31K.de-en/code' mode='r' encoding='UTF-8'>\n",
            "  args.codes = codecs.open(args.codes.name, encoding='utf-8')\n",
            "apply_bpe.py to train.en...\n",
            "subword-nmt/apply_bpe.py:396: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "subword-nmt/apply_bpe.py:416: ResourceWarning: unclosed file <_io.TextIOWrapper name='iwslt14.tokenized.31K.de-en/code' mode='r' encoding='UTF-8'>\n",
            "  args.codes = codecs.open(args.codes.name, encoding='utf-8')\n",
            "apply_bpe.py to valid.en...\n",
            "subword-nmt/apply_bpe.py:396: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "subword-nmt/apply_bpe.py:416: ResourceWarning: unclosed file <_io.TextIOWrapper name='iwslt14.tokenized.31K.de-en/code' mode='r' encoding='UTF-8'>\n",
            "  args.codes = codecs.open(args.codes.name, encoding='utf-8')\n",
            "apply_bpe.py to test.en...\n",
            "subword-nmt/apply_bpe.py:396: DeprecationWarning: this script's location has moved to /content/subword-nmt/subword_nmt. This symbolic link will be removed in a future version. Please point to the new location, or install the package and use the command 'subword-nmt'\n",
            "  DeprecationWarning\n",
            "subword-nmt/apply_bpe.py:416: ResourceWarning: unclosed file <_io.TextIOWrapper name='iwslt14.tokenized.31K.de-en/code' mode='r' encoding='UTF-8'>\n",
            "  args.codes = codecs.open(args.codes.name, encoding='utf-8')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uYGPMQfqK2ss",
        "colab_type": "text"
      },
      "source": [
        "###**3.4.   Dataset Binarization**\n",
        "\n",
        "In this next step, the `fairseq-preprocess` command-line tool is executed to build the vocabularies and convert the previously pre-processed training data to a binary format. During this step, a word level processing will generate a vocabulary of all words and subwords that appear in the dataset. The sentences are then converted into a sequence of integers by giving each word in the vocabulary a unique number. Naturally, the sentences are of different lengths. However, since most neural networks require inputs of identical size, we ensure that all sequences are of same length by padding the rest of the sequence with zeros.\n",
        "\n",
        "The [following figure](https://towardsdatascience.com/nlp-preparing-text-for-deep-learning-model-using-tensorflow2-461428138657) shows the different steps a text sequence goes through during pre-processing.\n",
        "<center><img src=https://miro.medium.com/max/1218/1*zsIXWoN0_CE9PXzmY3tIjQ.png width=\"500\"/></center>\n",
        "\n",
        "\n",
        "The binarized data that is later used for model training will be written to `data-bin/iwslt14.joined-dictionary.31K.de-en`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i3QDXVd1Job0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        },
        "outputId": "7cdd5d6e-35ba-448d-8344-3d7592a8c51d"
      },
      "source": [
        "# Dataset binarization\n",
        "TEXT='iwslt14.tokenized.31K.de-en'\n",
        "!fairseq-preprocess --joined-dictionary --source-lang de --target-lang en \\\n",
        "  --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \\\n",
        "  --destdir data-bin/iwslt14.joined-dictionary.31K.de-en"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-08-03 20:54:40 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, bf16=False, bpe=None, checkpoint_suffix='', cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='data-bin/iwslt14.joined-dictionary.31K.de-en', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=True, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=False, optimizer='nag', padding_factor=8, profile=False, quantization_config_path=None, seed=None, source_lang='de', srcdict=None, target_lang='en', task='translation', tensorboard_logdir='', testpref='iwslt14.tokenized.31K.de-en/test', tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='iwslt14.tokenized.31K.de-en/train', user_dir=None, validpref='iwslt14.tokenized.31K.de-en/valid', workers=1)\n",
            "2020-08-03 20:54:59 | INFO | fairseq_cli.preprocess | [de] Dictionary: 30760 types\n",
            "2020-08-03 20:55:21 | INFO | fairseq_cli.preprocess | [de] iwslt14.tokenized.31K.de-en/train.de: 160239 sents, 3580755 tokens, 0.0% replaced by <unk>\n",
            "2020-08-03 20:55:21 | INFO | fairseq_cli.preprocess | [de] Dictionary: 30760 types\n",
            "2020-08-03 20:55:22 | INFO | fairseq_cli.preprocess | [de] iwslt14.tokenized.31K.de-en/valid.de: 7283 sents, 162499 tokens, 0.00492% replaced by <unk>\n",
            "2020-08-03 20:55:22 | INFO | fairseq_cli.preprocess | [de] Dictionary: 30760 types\n",
            "2020-08-03 20:55:23 | INFO | fairseq_cli.preprocess | [de] iwslt14.tokenized.31K.de-en/test.de: 6750 sents, 145406 tokens, 0.0344% replaced by <unk>\n",
            "2020-08-03 20:55:23 | INFO | fairseq_cli.preprocess | [en] Dictionary: 30760 types\n",
            "2020-08-03 20:55:46 | INFO | fairseq_cli.preprocess | [en] iwslt14.tokenized.31K.de-en/train.en: 160239 sents, 3603388 tokens, 0.0% replaced by <unk>\n",
            "2020-08-03 20:55:46 | INFO | fairseq_cli.preprocess | [en] Dictionary: 30760 types\n",
            "2020-08-03 20:55:47 | INFO | fairseq_cli.preprocess | [en] iwslt14.tokenized.31K.de-en/valid.en: 7283 sents, 163398 tokens, 0.00245% replaced by <unk>\n",
            "2020-08-03 20:55:47 | INFO | fairseq_cli.preprocess | [en] Dictionary: 30760 types\n",
            "2020-08-03 20:55:48 | INFO | fairseq_cli.preprocess | [en] iwslt14.tokenized.31K.de-en/test.en: 6750 sents, 144709 tokens, 0.0076% replaced by <unk>\n",
            "2020-08-03 20:55:48 | INFO | fairseq_cli.preprocess | Wrote preprocessed data to data-bin/iwslt14.joined-dictionary.31K.de-en\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "78emERNpxMoQ",
        "colab_type": "text"
      },
      "source": [
        "###**3.5.   Model Building**\n",
        "\n",
        "As mentioned at the beginning, this notebook exploits the functions and models already implemented by the fairseq toolkit to build our neural machine translation model. For this reason, most of the code provided here either modifies some already implemented moduless to achieve the desired results or wraps some of the existing classes to adjust them to the new model architecture."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L2sSBju836v_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "438b40c1-f81b-4266-8f32-9b87e2476b66"
      },
      "source": [
        "%cd \"/content/fairseq\""
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/fairseq\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jq733bYi-IGg",
        "colab_type": "text"
      },
      "source": [
        "####**3.5.1. Build the Multihead Attention Model**####\n",
        "The Fairseq toolkit already has an implementation of the multi-head attention mechanism under `fairseq.modules.multihead_attention`. However, since we want to add locality constrains to the receptive field of the self-attention layers, we must adapt the implementation of multi-head attention appropriately. Therefore, the self-attention layer has to be able to mask the tokens that are not contained within a specific band and to mask the padding symbols we used to make the sequences of same length as well. This is achieved by setting the corresponding positions to $-\\infty$ in the self-attention calculation.  \n",
        "However, when combinated with padding masking, such a local attention masking can lead to all $-\\infty$ attention rows. This adapted version detects and corrects this situation.\n",
        "\n",
        "For a better understanding of the math behind multi-head attention, I recommend reading through the [\"*Attention Is All You Need*\"](https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf) paper."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mO3lSWWB93Oq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn.functional as F\n",
        "from torch import nn\n",
        "from torch.nn import Parameter\n",
        "from fairseq.incremental_decoding_utils import with_incremental_state\n",
        "\n",
        "from fairseq import utils\n",
        "\n",
        "\n",
        "@with_incremental_state\n",
        "class ProtectedMultiheadAttention(nn.Module):\n",
        "\n",
        "    def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bias_kv=False, add_zero_attn=False):\n",
        "        super().__init__()\n",
        "        self.embed_dim = embed_dim\n",
        "        self.num_heads = num_heads\n",
        "        self.dropout = dropout\n",
        "        self.head_dim = embed_dim // num_heads\n",
        "        assert self.head_dim * num_heads == self.embed_dim, \"embed_dim must be divisible by num_heads\"\n",
        "        self.scaling = self.head_dim ** -0.5\n",
        "\n",
        "        self.in_proj_weight = Parameter(torch.Tensor(3 * embed_dim, embed_dim))\n",
        "        if bias:\n",
        "            self.in_proj_bias = Parameter(torch.Tensor(3 * embed_dim))\n",
        "        else:\n",
        "            self.register_parameter('in_proj_bias', None)\n",
        "        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)\n",
        "\n",
        "        if add_bias_kv:\n",
        "            self.bias_k = Parameter(torch.Tensor(1, 1, embed_dim))\n",
        "            self.bias_v = Parameter(torch.Tensor(1, 1, embed_dim))\n",
        "        else:\n",
        "            self.bias_k = self.bias_v = None\n",
        "\n",
        "        self.add_zero_attn = add_zero_attn\n",
        "\n",
        "        self.reset_parameters()\n",
        "\n",
        "        self.onnx_trace = False\n",
        "\n",
        "    def prepare_for_onnx_export_(self):\n",
        "        self.onnx_trace = True\n",
        "\n",
        "    def reset_parameters(self):\n",
        "        nn.init.xavier_uniform_(self.in_proj_weight)\n",
        "        nn.init.xavier_uniform_(self.out_proj.weight)\n",
        "        if self.in_proj_bias is not None:\n",
        "            nn.init.constant_(self.in_proj_bias, 0.)\n",
        "            nn.init.constant_(self.out_proj.bias, 0.)\n",
        "        if self.bias_k is not None:\n",
        "            nn.init.xavier_normal_(self.bias_k)\n",
        "        if self.bias_v is not None:\n",
        "            nn.init.xavier_normal_(self.bias_v)\n",
        "\n",
        "    def forward(self, query, key, value, key_padding_mask=None, incremental_state=None,\n",
        "                need_weights=True, static_kv=False, attn_mask=None):\n",
        "        \"\"\"Input shape: Time x Batch x Channel\n",
        "\n",
        "        Self-attention can be implemented by passing in the same arguments for\n",
        "        query, key and value. Timesteps can be masked by supplying a T x T mask in the\n",
        "        `attn_mask` argument. Padding elements can be excluded from\n",
        "        the key by passing a binary ByteTensor (`key_padding_mask`) with shape:\n",
        "        batch x src_len, where padding elements are indicated by 1s.\n",
        "        \"\"\"\n",
        "\n",
        "        qkv_same = query.data_ptr() == key.data_ptr() == value.data_ptr()\n",
        "        kv_same = key.data_ptr() == value.data_ptr()\n",
        "\n",
        "        tgt_len, bsz, embed_dim = query.size()\n",
        "        assert embed_dim == self.embed_dim\n",
        "        assert list(query.size()) == [tgt_len, bsz, embed_dim]\n",
        "        assert key.size() == value.size()\n",
        "\n",
        "        if incremental_state is not None:\n",
        "            saved_state = self._get_input_buffer(incremental_state)\n",
        "            if 'prev_key' in saved_state:\n",
        "                # previous time steps are cached - no need to recompute\n",
        "                # key and value if they are static\n",
        "                if static_kv:\n",
        "                    assert kv_same and not qkv_same\n",
        "                    key = value = None\n",
        "        else:\n",
        "            saved_state = None\n",
        "\n",
        "        if qkv_same:\n",
        "            # self-attention\n",
        "            q, k, v = self.in_proj_qkv(query)\n",
        "        elif kv_same:\n",
        "            # encoder-decoder attention\n",
        "            q = self.in_proj_q(query)\n",
        "            if key is None:\n",
        "                assert value is None\n",
        "                k = v = None\n",
        "            else:\n",
        "                k, v = self.in_proj_kv(key)\n",
        "        else:\n",
        "            q = self.in_proj_q(query)\n",
        "            k = self.in_proj_k(key)\n",
        "            v = self.in_proj_v(value)\n",
        "        q *= self.scaling\n",
        "\n",
        "        if self.bias_k is not None:\n",
        "            assert self.bias_v is not None\n",
        "            k = torch.cat([k, self.bias_k.repeat(1, bsz, 1)])\n",
        "            v = torch.cat([v, self.bias_v.repeat(1, bsz, 1)])\n",
        "            if attn_mask is not None:\n",
        "                attn_mask = torch.cat([attn_mask, attn_mask.new_zeros(attn_mask.size(0), 1)], dim=1)\n",
        "            if key_padding_mask is not None:\n",
        "                key_padding_mask = torch.cat(\n",
        "                    [key_padding_mask, key_padding_mask.new_zeros(key_padding_mask.size(0), 1)], dim=1)\n",
        "\n",
        "        q = q.contiguous().view(tgt_len, bsz * self.num_heads, self.head_dim).transpose(0, 1)\n",
        "        if k is not None:\n",
        "            k = k.contiguous().view(-1, bsz * self.num_heads, self.head_dim).transpose(0, 1)\n",
        "        if v is not None:\n",
        "            v = v.contiguous().view(-1, bsz * self.num_heads, self.head_dim).transpose(0, 1)\n",
        "\n",
        "        if saved_state is not None:\n",
        "            # saved states are stored with shape (bsz, num_heads, seq_len, head_dim)\n",
        "            if 'prev_key' in saved_state:\n",
        "                prev_key = saved_state['prev_key'].view(bsz * self.num_heads, -1, self.head_dim)\n",
        "                if static_kv:\n",
        "                    k = prev_key\n",
        "                else:\n",
        "                    k = torch.cat((prev_key, k), dim=1)\n",
        "            if 'prev_value' in saved_state:\n",
        "                prev_value = saved_state['prev_value'].view(bsz * self.num_heads, -1, self.head_dim)\n",
        "                if static_kv:\n",
        "                    v = prev_value\n",
        "                else:\n",
        "                    v = torch.cat((prev_value, v), dim=1)\n",
        "            saved_state['prev_key'] = k.view(bsz, self.num_heads, -1, self.head_dim)\n",
        "            saved_state['prev_value'] = v.view(bsz, self.num_heads, -1, self.head_dim)\n",
        "\n",
        "            self._set_input_buffer(incremental_state, saved_state)\n",
        "\n",
        "        src_len = k.size(1)\n",
        "\n",
        "        if key_padding_mask is not None:\n",
        "            assert key_padding_mask.size(0) == bsz\n",
        "            assert key_padding_mask.size(1) == src_len\n",
        "\n",
        "        if self.add_zero_attn:\n",
        "            src_len += 1\n",
        "            k = torch.cat([k, k.new_zeros((k.size(0), 1) + k.size()[2:])], dim=1)\n",
        "            v = torch.cat([v, v.new_zeros((v.size(0), 1) + v.size()[2:])], dim=1)\n",
        "            if attn_mask is not None:\n",
        "                attn_mask = torch.cat([attn_mask, attn_mask.new_zeros(attn_mask.size(0), 1)], dim=1)\n",
        "            if key_padding_mask is not None:\n",
        "                key_padding_mask = torch.cat(\n",
        "                    [key_padding_mask, torch.zeros(key_padding_mask.size(0), 1).type_as(key_padding_mask)], dim=1)\n",
        "\n",
        "        attn_weights = torch.bmm(q, k.transpose(1, 2))\n",
        "        assert list(attn_weights.size()) == [bsz * self.num_heads, tgt_len, src_len]\n",
        "\n",
        "        if attn_mask is not None:\n",
        "            attn_mask = attn_mask.unsqueeze(0)\n",
        "            if self.onnx_trace:\n",
        "                attn_mask = attn_mask.repeat(attn_weights.size(0), 1, 1)\n",
        "            attn_weights += attn_mask\n",
        "\n",
        "        if key_padding_mask is not None:\n",
        "            # don't attend to padding symbols\n",
        "            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)\n",
        "            if self.onnx_trace:\n",
        "                attn_weights = torch.where(\n",
        "                    key_padding_mask.unsqueeze(1).unsqueeze(2),\n",
        "                    torch.Tensor([float(\"-Inf\")]),\n",
        "                    attn_weights.float()\n",
        "                ).type_as(attn_weights)\n",
        "            else:\n",
        "                attn_weights = attn_weights.float().masked_fill(\n",
        "                    key_padding_mask.unsqueeze(1).unsqueeze(2),\n",
        "                    float('-inf'),\n",
        "                ).type_as(attn_weights)  # FP16 support: cast to float and back\n",
        "            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\n",
        "            all_inf = torch.isinf(attn_weights).all(dim=-1)\n",
        "            if all_inf.any():\n",
        "                attn_weights = attn_weights.float().masked_fill(\n",
        "                    all_inf.unsqueeze(-1),\n",
        "                    0,\n",
        "                ).type_as(attn_weights)  # FP16 support: cast to float and back\n",
        "\n",
        "\n",
        "        attn_weights = F.softmax(attn_weights.float(), dim=-1).type_as(attn_weights)\n",
        "        attn_weights = F.dropout(attn_weights, p=self.dropout, training=self.training)\n",
        "\n",
        "        attn = torch.bmm(attn_weights, v)\n",
        "        assert list(attn.size()) == [bsz * self.num_heads, tgt_len, self.head_dim]\n",
        "        if (self.onnx_trace and attn.size(1) == 1):\n",
        "            # when ONNX tracing a single decoder step (sequence length == 1)\n",
        "            # the transpose is a no-op copy before view, thus unnecessary\n",
        "            attn = attn.contiguous().view(tgt_len, bsz, embed_dim)\n",
        "        else:\n",
        "            attn = attn.transpose(0, 1).contiguous().view(tgt_len, bsz, embed_dim)\n",
        "        attn = self.out_proj(attn)\n",
        "\n",
        "        if need_weights:\n",
        "            # average attention weights over heads\n",
        "            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)\n",
        "            attn_weights = attn_weights.sum(dim=1) / self.num_heads\n",
        "        else:\n",
        "            attn_weights = None\n",
        "\n",
        "        return attn, attn_weights\n",
        "\n",
        "    def in_proj_qkv(self, query):\n",
        "        return self._in_proj(query).chunk(3, dim=-1)\n",
        "\n",
        "    def in_proj_kv(self, key):\n",
        "        return self._in_proj(key, start=self.embed_dim).chunk(2, dim=-1)\n",
        "\n",
        "    def in_proj_q(self, query):\n",
        "        return self._in_proj(query, end=self.embed_dim)\n",
        "\n",
        "    def in_proj_k(self, key):\n",
        "        return self._in_proj(key, start=self.embed_dim, end=2 * self.embed_dim)\n",
        "\n",
        "    def in_proj_v(self, value):\n",
        "        return self._in_proj(value, start=2 * self.embed_dim)\n",
        "\n",
        "    def _in_proj(self, input, start=0, end=None):\n",
        "        weight = self.in_proj_weight\n",
        "        bias = self.in_proj_bias\n",
        "        weight = weight[start:end, :]\n",
        "        if bias is not None:\n",
        "            bias = bias[start:end]\n",
        "        return F.linear(input, weight, bias)\n",
        "\n",
        "    def reorder_incremental_state(self, incremental_state, new_order):\n",
        "        \"\"\"Reorder buffered internal state (for incremental generation).\"\"\"\n",
        "        input_buffer = self._get_input_buffer(incremental_state)\n",
        "        if input_buffer is not None:\n",
        "            for k in input_buffer.keys():\n",
        "                input_buffer[k] = input_buffer[k].index_select(0, new_order)\n",
        "            self._set_input_buffer(incremental_state, input_buffer)\n",
        "\n",
        "    def _get_input_buffer(self, incremental_state):\n",
        "        return self.get_incremental_state(\n",
        "            incremental_state,\n",
        "            'attn_state',\n",
        "        ) or {}\n",
        "\n",
        "    def _set_input_buffer(self, incremental_state, buffer):\n",
        "        return self.set_incremental_state(\n",
        "            incremental_state,\n",
        "            'attn_state',\n",
        "            buffer,\n",
        "        )\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q9XLcgkCvLN3",
        "colab_type": "text"
      },
      "source": [
        "#### **3.5.2. Adapt some Functions** ####\n",
        "\n",
        "We use some of the precomputed Pytorch modules to get the functions for word embedding (mapping the different words from the vocabulary to a vector of real numbers), language embedding, normalization layer and the the linear layer.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nrnQ4ALBv8yV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import math\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "from fairseq import options\n",
        "from fairseq import utils\n",
        "\n",
        "\n",
        "def Embedding(num_embeddings, embedding_dim, padding_idx):\n",
        "    m = nn.Embedding(num_embeddings, embedding_dim, padding_idx=padding_idx)\n",
        "    nn.init.normal_(m.weight, mean=0, std=embedding_dim ** -0.5)\n",
        "    nn.init.constant_(m.weight[padding_idx], 0)\n",
        "    return m\n",
        "\n",
        "\n",
        "def LanguageEmbedding(embedding_dim):\n",
        "    m = nn.Parameter(torch.Tensor(embedding_dim))\n",
        "    nn.init.normal_(m, mean=0, std=embedding_dim ** -0.5)\n",
        "    return m\n",
        "\n",
        "\n",
        "def LayerNorm(embedding_dim):\n",
        "    m = nn.LayerNorm(embedding_dim)\n",
        "    return m\n",
        "\n",
        "\n",
        "def Linear(in_features, out_features, bias=True):\n",
        "    m = nn.Linear(in_features, out_features, bias)\n",
        "    nn.init.xavier_uniform_(m.weight)\n",
        "    if bias:\n",
        "        nn.init.constant_(m.bias, 0.)\n",
        "    return m"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJs7HC5G9H_H",
        "colab_type": "text"
      },
      "source": [
        "#### **3.5.3. Build the Encoder Block**\n",
        "\n",
        "As we discussed in the model architecture section, the \n",
        "joint attention model does not make use of an independant encoder. The encoder is only needed for the source embeddings computation, including pasdding the sequences to get network inputs of same-length . \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5tdO37a5YHnM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fairseq.models import FairseqEncoder\n",
        "from fairseq.modules import PositionalEmbedding\n",
        "\n",
        "\n",
        "class JointAttentionEncoder(FairseqEncoder):\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        args (argparse.Namespace): parsed command-line arguments\n",
        "        dictionary (~fairseq.data.Dictionary): encoding dictionary\n",
        "        embed_tokens (torch.nn.Embedding): input embedding\n",
        "        left_pad (bool): whether the input is left-padded\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, args, dictionary, embed_tokens, left_pad):\n",
        "        super().__init__(dictionary)\n",
        "        self.dropout = args.dropout\n",
        "\n",
        "        embed_dim = embed_tokens.embedding_dim\n",
        "        self.padding_idx = embed_tokens.padding_idx\n",
        "        self.max_source_positions = args.max_source_positions\n",
        "\n",
        "        self.embed_tokens = embed_tokens\n",
        "        self.embed_scale = math.sqrt(embed_dim)\n",
        "        self.embed_positions = PositionalEmbedding(\n",
        "            args.max_source_positions, embed_dim, self.padding_idx,\n",
        "            learned=args.encoder_learned_pos,\n",
        "        ) if not args.no_token_positional_embeddings else None\n",
        "        self.embed_language = LanguageEmbedding(embed_dim) if args.language_embeddings else None\n",
        "\n",
        "        self.register_buffer('version', torch.Tensor([2]))\n",
        "\n",
        "    def forward(self, src_tokens, src_lengths):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            src_tokens (LongTensor): tokens in the source language of shape\n",
        "                `(batch, src_len)`\n",
        "            src_lengths (torch.LongTensor): lengths of each source sentence of\n",
        "                shape `(batch)`\n",
        "\n",
        "        Returns:\n",
        "            dict:\n",
        "                - **encoder_out** (Tensor): embedding output of shape\n",
        "                  `(src_len, batch, embed_dim)`\n",
        "                - **encoder_padding_mask** (ByteTensor): the positions of\n",
        "                  padding elements of shape `(batch, src_len)`\n",
        "        \"\"\"\n",
        "        # embed tokens and positions\n",
        "        x = self.embed_scale * self.embed_tokens(src_tokens)\n",
        "        if self.embed_positions is not None:\n",
        "            x += self.embed_positions(src_tokens)\n",
        "        # language embedding\n",
        "        if self.embed_language is not None:\n",
        "            lang_emb = self.embed_scale * self.embed_language.view(1, 1, -1)\n",
        "            x += lang_emb\n",
        "        x = F.dropout(x, p=self.dropout, training=self.training)\n",
        "\n",
        "        # B x T x C -> T x B x C\n",
        "        x = x.transpose(0, 1)\n",
        "\n",
        "        # compute padding mask\n",
        "        encoder_padding_mask = src_tokens.eq(self.padding_idx)\n",
        "        if not encoder_padding_mask.any():\n",
        "            encoder_padding_mask = None\n",
        "\n",
        "        return {\n",
        "            'encoder_out': x,  # T x B x C\n",
        "            'encoder_padding_mask': encoder_padding_mask,  # B x T\n",
        "        }\n",
        "\n",
        "    def reorder_encoder_out(self, encoder_out, new_order):\n",
        "        \"\"\"\n",
        "        Reorder encoder output according to *new_order*.\n",
        "\n",
        "        Args:\n",
        "            encoder_out: output from the ``forward()`` method\n",
        "            new_order (LongTensor): desired order\n",
        "\n",
        "        Returns:\n",
        "            *encoder_out* rearranged according to *new_order*\n",
        "        \"\"\"\n",
        "        if encoder_out['encoder_out'] is not None:\n",
        "            encoder_out['encoder_out'] = \\\n",
        "                encoder_out['encoder_out'].index_select(1, new_order)\n",
        "        if encoder_out['encoder_padding_mask'] is not None:\n",
        "            encoder_out['encoder_padding_mask'] = \\\n",
        "                encoder_out['encoder_padding_mask'].index_select(0, new_order)\n",
        "        return encoder_out\n",
        "\n",
        "    def max_positions(self):\n",
        "        \"\"\"Maximum input length supported by the encoder.\"\"\"\n",
        "        if self.embed_positions is None:\n",
        "            return self.max_source_positions\n",
        "        return min(self.max_source_positions, self.embed_positions.max_positions)\n"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fg1jg_IGiOvG",
        "colab_type": "text"
      },
      "source": [
        "#### **3.5.4. Adapt the Decoder** ####\n",
        "\n",
        "We use the preimplemented decoder from `fairseq.models.transformer.py` and adapt it to use `ProtectedMultiheadAttention` we implemented before. The decoder is postprocessed with dropout, a residual layer and a normalization layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tK1K7l-B94qX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ProtectedTransformerDecoderLayer(nn.Module):\n",
        "    \"\"\"Decoder layer block.\n",
        "    In the original paper each operation (multi-head attention, encoder\n",
        "    attention or FFN) is postprocessed with: `dropout -> add residual ->\n",
        "    layernorm`. In the tensor2tensor code they suggest that learning is more\n",
        "    robust when preprocessing each layer with layernorm and postprocessing with:\n",
        "    `dropout -> add residual`. We default to the approach in the paper, but the\n",
        "    tensor2tensor approach can be enabled by setting\n",
        "    *args.decoder_normalize_before* to ``True``.\n",
        "    Args:\n",
        "        args (argparse.Namespace): parsed command-line arguments\n",
        "        no_encoder_attn (bool, optional): whether to attend to encoder outputs\n",
        "            (default: False).\n",
        "    \"\"\"\n",
        "    \n",
        "    def __init__(self, args, no_encoder_attn=False):\n",
        "        super().__init__()\n",
        "        self.embed_dim = args.decoder_embed_dim\n",
        "        self.self_attn = ProtectedMultiheadAttention(\n",
        "            self.embed_dim, args.decoder_attention_heads,\n",
        "            dropout=args.attention_dropout,\n",
        "        )\n",
        "        self.dropout = args.dropout\n",
        "        self.relu_dropout = args.relu_dropout\n",
        "        self.normalize_before = args.decoder_normalize_before\n",
        "\n",
        "        self.self_attn_layer_norm = LayerNorm(self.embed_dim)\n",
        "\n",
        "        if no_encoder_attn:\n",
        "            self.encoder_attn = None\n",
        "            self.encoder_attn_layer_norm = None\n",
        "        else:\n",
        "            self.encoder_attn = ProtectedMultiheadAttention(\n",
        "                self.embed_dim, args.decoder_attention_heads,\n",
        "                dropout=args.attention_dropout,\n",
        "            )\n",
        "            self.encoder_attn_layer_norm = LayerNorm(self.embed_dim)\n",
        "\n",
        "        self.fc1 = Linear(self.embed_dim, args.decoder_ffn_embed_dim)\n",
        "        self.fc2 = Linear(args.decoder_ffn_embed_dim, self.embed_dim)\n",
        "\n",
        "        self.final_layer_norm = LayerNorm(self.embed_dim)\n",
        "        self.need_attn = True\n",
        "\n",
        "        self.onnx_trace = False\n",
        "\n",
        "    def prepare_for_onnx_export_(self):\n",
        "        self.onnx_trace = True\n",
        "\n",
        "    def forward(self, x, encoder_out, encoder_padding_mask, incremental_state,\n",
        "                prev_self_attn_state=None, prev_attn_state=None, self_attn_mask=None,\n",
        "                self_attn_padding_mask=None):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x (Tensor): input to the layer of shape `(seq_len, batch, embed_dim)`\n",
        "            encoder_padding_mask (ByteTensor): binary ByteTensor of shape\n",
        "                `(batch, src_len)` where padding elements are indicated by ``1``.\n",
        "\n",
        "        Returns:\n",
        "            encoded output of shape `(batch, src_len, embed_dim)`\n",
        "        \"\"\"\n",
        "        residual = x\n",
        "        x = self.maybe_layer_norm(self.self_attn_layer_norm, x, before=True)\n",
        "        if prev_self_attn_state is not None:\n",
        "            if incremental_state is None:\n",
        "                incremental_state = {}\n",
        "            prev_key, prev_value = prev_self_attn_state\n",
        "            saved_state = {\"prev_key\": prev_key, \"prev_value\": prev_value}\n",
        "            self.self_attn._set_input_buffer(incremental_state, saved_state)\n",
        "        x, _ = self.self_attn(\n",
        "            query=x,\n",
        "            key=x,\n",
        "            value=x,\n",
        "            key_padding_mask=self_attn_padding_mask,\n",
        "            incremental_state=incremental_state,\n",
        "            need_weights=False,\n",
        "            attn_mask=self_attn_mask,\n",
        "        )\n",
        "        x = F.dropout(x, p=self.dropout, training=self.training)\n",
        "        x = residual + x\n",
        "        x = self.maybe_layer_norm(self.self_attn_layer_norm, x, after=True)\n",
        "\n",
        "        attn = None\n",
        "        if self.encoder_attn is not None:\n",
        "            residual = x\n",
        "            x = self.maybe_layer_norm(self.encoder_attn_layer_norm, x, before=True)\n",
        "            if prev_attn_state is not None:\n",
        "                if incremental_state is None:\n",
        "                    incremental_state = {}\n",
        "                prev_key, prev_value = prev_attn_state\n",
        "                saved_state = {\"prev_key\": prev_key, \"prev_value\": prev_value}\n",
        "                self.encoder_attn._set_input_buffer(incremental_state, saved_state)\n",
        "            x, attn = self.encoder_attn(\n",
        "                query=x,\n",
        "                key=encoder_out,\n",
        "                value=encoder_out,\n",
        "                key_padding_mask=encoder_padding_mask,\n",
        "                incremental_state=incremental_state,\n",
        "                static_kv=True,\n",
        "                need_weights=(not self.training and self.need_attn),\n",
        "            )\n",
        "            x = F.dropout(x, p=self.dropout, training=self.training)\n",
        "            x = residual + x\n",
        "            x = self.maybe_layer_norm(self.encoder_attn_layer_norm, x, after=True)\n",
        "\n",
        "        residual = x\n",
        "        x = self.maybe_layer_norm(self.final_layer_norm, x, before=True)\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.dropout(x, p=self.relu_dropout, training=self.training)\n",
        "        x = self.fc2(x)\n",
        "        x = F.dropout(x, p=self.dropout, training=self.training)\n",
        "        x = residual + x\n",
        "        x = self.maybe_layer_norm(self.final_layer_norm, x, after=True)\n",
        "        if self.onnx_trace:\n",
        "            saved_state = self.self_attn._get_input_buffer(incremental_state)\n",
        "            self_attn_state = saved_state[\"prev_key\"], saved_state[\"prev_value\"]\n",
        "            return x, attn, self_attn_state\n",
        "        return x, attn\n",
        "\n",
        "    def maybe_layer_norm(self, layer_norm, x, before=False, after=False):\n",
        "        assert before ^ after\n",
        "        if after ^ self.normalize_before:\n",
        "            return layer_norm(x)\n",
        "        else:\n",
        "            return x\n",
        "\n",
        "    def make_generation_fast_(self, need_attn=False, **kwargs):\n",
        "        self.need_attn = need_attn\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtRJsI9ptFD7",
        "colab_type": "text"
      },
      "source": [
        "#### **3.5.5. Build the Decoder Block**\n",
        "We wil now build the decoder block consisting of `args.decoder_layers` decoders from the class `ProtectedTransformerDecoderLayer`.\n",
        "Here, we implement an incremental decoding with the `FairseqIncrementalDecoder` interface. The incremental decoder interface is preferred over the `FairseqDecoder` interface because it results in faster generations during inference time. This is achieved by inputting only the immediate previous state and caching all previous states in an `incremental_state` argument.\n",
        "\n",
        "For a detailed explanation of this of this choice of decoder, refer to this [tutorial](https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html#making-generation-faster)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8010Huz8YnEL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fairseq.models import FairseqIncrementalDecoder\n",
        "\n",
        "class JointAttentionDecoder(FairseqIncrementalDecoder):\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        args (argparse.Namespace): parsed command-line arguments\n",
        "        dictionary (~fairseq.data.Dictionary): decoding dictionary\n",
        "        embed_tokens (torch.nn.Embedding): output embedding\n",
        "        left_pad (bool, optional): whether the input is left-padded. Default:\n",
        "            ``False``\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, args, dictionary, embed_tokens, left_pad=False, final_norm=True):\n",
        "        super().__init__(dictionary)\n",
        "        self.dropout = args.dropout\n",
        "        self.share_input_output_embed = args.share_decoder_input_output_embed\n",
        "        self.kernel_size_list = args.kernel_size_list\n",
        "\n",
        "        input_embed_dim = embed_tokens.embedding_dim\n",
        "        embed_dim = args.decoder_embed_dim\n",
        "        output_embed_dim = args.decoder_output_dim\n",
        "\n",
        "        padding_idx = embed_tokens.padding_idx\n",
        "        self.max_target_positions = args.max_target_positions\n",
        "\n",
        "        self.embed_tokens = embed_tokens\n",
        "        self.embed_scale = math.sqrt(embed_dim)\n",
        "\n",
        "        self.project_in_dim = Linear(input_embed_dim, embed_dim, bias=False) if embed_dim != input_embed_dim else None\n",
        "\n",
        "        self.embed_positions = PositionalEmbedding(\n",
        "            args.max_target_positions, embed_dim, padding_idx,\n",
        "            learned=args.decoder_learned_pos,\n",
        "        ) if not args.no_token_positional_embeddings else None\n",
        "\n",
        "        self.embed_language = LanguageEmbedding(embed_dim) if args.language_embeddings else None\n",
        "\n",
        "        self.layers = nn.ModuleList([])\n",
        "        self.layers.extend([\n",
        "            ProtectedTransformerDecoderLayer(args, no_encoder_attn=True)\n",
        "            for _ in range(args.decoder_layers)\n",
        "        ])\n",
        "\n",
        "        self.project_out_dim = Linear(embed_dim, output_embed_dim, bias=False) \\\n",
        "            if embed_dim != output_embed_dim and not args.tie_adaptive_weights else None\n",
        "\n",
        "        if not self.share_input_output_embed:\n",
        "            self.embed_out = nn.Parameter(torch.Tensor(len(dictionary), output_embed_dim))\n",
        "            nn.init.normal_(self.embed_out, mean=0, std=output_embed_dim ** -0.5)\n",
        "        self.register_buffer('version', torch.Tensor([2]))\n",
        "        self.normalize = args.decoder_normalize_before and final_norm\n",
        "        if self.normalize:\n",
        "            self.layer_norm = LayerNorm(embed_dim)\n",
        "\n",
        "    def forward(self, prev_output_tokens, encoder_out, incremental_state=None):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            input (dict): with\n",
        "                prev_output_tokens (LongTensor): previous decoder outputs of shape\n",
        "                    `(batch, tgt_len)`, for input feeding/teacher forcing\n",
        "            encoder_out (Tensor, optional): output from the encoder, used for\n",
        "                encoder-side attention\n",
        "            incremental_state (dict): dictionary used for storing state during\n",
        "                :ref:`Incremental decoding`\n",
        "\n",
        "        Returns:\n",
        "            tuple:\n",
        "                - the last decoder layer's output of shape `(batch, tgt_len,\n",
        "                  vocab)`\n",
        "                - the last decoder layer's attention weights of shape `(batch,\n",
        "                  tgt_len, src_len)`\n",
        "        \"\"\"\n",
        "        tgt_len = prev_output_tokens.size(1)\n",
        "\n",
        "        # embed positions\n",
        "        positions = self.embed_positions(\n",
        "            prev_output_tokens,\n",
        "            incremental_state=incremental_state,\n",
        "        ) if self.embed_positions is not None else None\n",
        "\n",
        "        if incremental_state is not None:\n",
        "            prev_output_tokens = prev_output_tokens[:, -1:]\n",
        "            if positions is not None:\n",
        "                positions = positions[:, -1:]\n",
        "\n",
        "        # embed tokens and positions\n",
        "        x = self.embed_scale * self.embed_tokens(prev_output_tokens)\n",
        "\n",
        "        if self.project_in_dim is not None:\n",
        "            x = self.project_in_dim(x)\n",
        "\n",
        "        if positions is not None:\n",
        "            x += positions\n",
        "\n",
        "        # language embedding\n",
        "        if self.embed_language is not None:\n",
        "            lang_emb = self.embed_scale * self.embed_language.view(1, 1, -1)\n",
        "            x += lang_emb\n",
        "\n",
        "        x = F.dropout(x, p=self.dropout, training=self.training)\n",
        "\n",
        "        # B x T x C -> T x B x C\n",
        "        x = x.transpose(0, 1)\n",
        "        attn = None\n",
        "        inner_states = [x]\n",
        "        source = encoder_out['encoder_out']\n",
        "        process_source = incremental_state is None or len(incremental_state) == 0\n",
        "\n",
        "        # extended padding mask\n",
        "        source_padding_mask = encoder_out['encoder_padding_mask']\n",
        "        if source_padding_mask is not None:\n",
        "            target_padding_mask = source_padding_mask.new_zeros((source_padding_mask.size(0), tgt_len))\n",
        "            self_attn_padding_mask = torch.cat((source_padding_mask, target_padding_mask), dim=1)\n",
        "        else:\n",
        "            self_attn_padding_mask = None\n",
        "\n",
        "        # transformer layers\n",
        "        for i, layer in enumerate(self.layers):\n",
        "\n",
        "            if self.kernel_size_list is not None:\n",
        "                target_mask = self.local_mask(x, self.kernel_size_list[i], causal=True, tgt_len=tgt_len)\n",
        "            elif incremental_state is None:\n",
        "                target_mask = self.buffered_future_mask(x)\n",
        "            else:\n",
        "                target_mask = None\n",
        "\n",
        "            if target_mask is not None:\n",
        "                zero_mask = target_mask.new_zeros((target_mask.size(0), source.size(0)))\n",
        "                self_attn_mask = torch.cat((zero_mask, target_mask), dim=1)\n",
        "            else:\n",
        "                self_attn_mask = None\n",
        "\n",
        "            state = incremental_state\n",
        "            if process_source:\n",
        "                if state is None:\n",
        "                    state = {}\n",
        "                if self.kernel_size_list is not None:\n",
        "                    source_mask = self.local_mask(source, self.kernel_size_list[i], causal=False)\n",
        "                else:\n",
        "                    source_mask = None\n",
        "                source, attn = layer(\n",
        "                    source,\n",
        "                    None,\n",
        "                    None,\n",
        "                    state,\n",
        "                    self_attn_mask=source_mask,\n",
        "                    self_attn_padding_mask=source_padding_mask\n",
        "                )\n",
        "                inner_states.append(source)\n",
        "\n",
        "            x, attn = layer(\n",
        "                x,\n",
        "                None,\n",
        "                None,\n",
        "                state,\n",
        "                self_attn_mask=self_attn_mask,\n",
        "                self_attn_padding_mask=self_attn_padding_mask\n",
        "            )\n",
        "            inner_states.append(x)\n",
        "\n",
        "        if self.normalize:\n",
        "            x = self.layer_norm(x)\n",
        "\n",
        "        # T x B x C -> B x T x C\n",
        "        x = x.transpose(0, 1)\n",
        "\n",
        "        if self.project_out_dim is not None:\n",
        "            x = self.project_out_dim(x)\n",
        "\n",
        "        # project back to size of vocabulary\n",
        "        if self.share_input_output_embed:\n",
        "            x = F.linear(x, self.embed_tokens.weight)\n",
        "        else:\n",
        "            x = F.linear(x, self.embed_out)\n",
        "\n",
        "        pred = x\n",
        "        info = {'attn': attn, 'inner_states': inner_states}\n",
        "\n",
        "        return pred, info\n",
        "\n",
        "    def max_positions(self):\n",
        "        \"\"\"Maximum output length supported by the decoder.\"\"\"\n",
        "        if self.embed_positions is None:\n",
        "            return self.max_target_positions\n",
        "        return min(self.max_target_positions, self.embed_positions.max_positions)\n",
        "\n",
        "    def buffered_future_mask(self, tensor):\n",
        "        \"\"\"Cached future mask.\"\"\"\n",
        "        dim = tensor.size(0)\n",
        "        #pylint: disable=access-member-before-definition, attribute-defined-outside-init\n",
        "        if not hasattr(self, '_future_mask') or self._future_mask is None or self._future_mask.device != tensor.device:\n",
        "            self._future_mask = torch.triu(utils.fill_with_neg_inf(tensor.new(dim, dim)), 1)\n",
        "        if self._future_mask.size(0) < dim:\n",
        "            self._future_mask = torch.triu(utils.fill_with_neg_inf(self._future_mask.resize_(dim, dim)), 1)\n",
        "        return self._future_mask[:dim, :dim]\n",
        "\n",
        "    def local_mask(self, tensor, kernel_size, causal, tgt_len=None):\n",
        "        \"\"\"Locality constraint mask.\"\"\"\n",
        "        rows = tensor.size(0)\n",
        "        cols = tensor.size(0) if tgt_len is None else tgt_len\n",
        "        if causal:\n",
        "            if rows == 1:\n",
        "                mask = utils.fill_with_neg_inf(tensor.new(1, cols))\n",
        "                mask[0, -kernel_size:] = 0\n",
        "                return mask\n",
        "            else:\n",
        "                diag_u, diag_l = 1, kernel_size\n",
        "        else:\n",
        "            diag_u, diag_l = ((kernel_size + 1) // 2, (kernel_size + 1) // 2) if kernel_size % 2 == 1 \\\n",
        "                else (kernel_size // 2, kernel_size // 2 + 1)\n",
        "        mask1 = torch.triu(utils.fill_with_neg_inf(tensor.new(rows, cols)), diag_u)\n",
        "        mask2 = torch.tril(utils.fill_with_neg_inf(tensor.new(rows, cols)), -diag_l)\n",
        "\n",
        "        return mask1 + mask2\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fN3EtVetqwd3",
        "colab_type": "text"
      },
      "source": [
        "#### **3.5.6. Register the Encoder-Decoder Model**\n",
        "\n",
        "After we have finished defining the encoder and decoder, we can extend the fairseq library with the neural network model [plug-in](https://fairseq.readthedocs.io/en/latest/overview.html) by registering it with the help of the function decorater `register_model()` provided by fairseq. This is mandatory in order to use the fairseq command-line tools to train the model and later evaluate its performance by calculating its BLEU score.\n",
        "\n",
        "The class [`BaseFairseqModel`](https://fairseq.readthedocs.io/en/latest/models.html#fairseq.models.BaseFairseqModel) serves as a base class for all fairseq models. For this step we must therefore implement a wrapper around the [`FairseqEncoderDecoderModel`](https://fairseq.readthedocs.io/en/latest/models.html#fairseq.models.FairseqEncoderDecoderModel) interface and extend it with the two functions `add_args()` and `build_model()`, as we are dealing with sequence-to-sequence models.  \n",
        "On the one hand, the first function allows to expand the comman-line with new model-specific arguments. In this case, arguments like dropout, kernel size and the dimensionality of the embeddings are added.\n",
        "On the other hand, `build_model()` initializes the encoder and decoder models and returns a `JointAttentionModel` model instance. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wK8KWjTPXkBH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fairseq.models import (FairseqEncoderDecoderModel, register_model)\n",
        "\n",
        "@register_model('joint_attention')\n",
        "class JointAttentionModel(FairseqEncoderDecoderModel):\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        encoder (JointAttentionEncoder): the encoder\n",
        "        decoder (JointAttentionDecoder): the decoder\n",
        "\n",
        "    The joint source-target model provides the following named architectures and\n",
        "    command-line arguments:\n",
        "\n",
        "    .. argparse::\n",
        "        :ref: fairseq.models.joint_attention_parser\n",
        "        :prog:\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, encoder, decoder):\n",
        "        super().__init__(encoder, decoder)\n",
        "\n",
        "    @staticmethod\n",
        "    def add_args(parser):\n",
        "        \"\"\"Add model-specific arguments to the parser.\"\"\"\n",
        "        parser.add_argument('--encoder-embed-path', type=str, metavar='STR',\n",
        "                            help='path to pre-trained encoder embedding')\n",
        "        parser.add_argument('--encoder-embed-dim', type=int, metavar='N',\n",
        "                            help='encoder embedding dimension')\n",
        "        parser.add_argument('--encoder-learned-pos', action='store_true',\n",
        "                            help='use learned positional embeddings in the encoder')\n",
        "        parser.add_argument('--decoder-embed-path', type=str, metavar='STR',\n",
        "                            help='path to pre-trained decoder embedding')\n",
        "        parser.add_argument('--decoder-embed-dim', type=int, metavar='N',\n",
        "                            help='decoder embedding dimension')\n",
        "        parser.add_argument('--decoder-learned-pos', action='store_true',\n",
        "                            help='use learned positional embeddings in the decoder')\n",
        "        parser.add_argument('--decoder-normalize-before', action='store_true',\n",
        "                            help='apply layernorm before each decoder block')\n",
        "        parser.add_argument('--share-decoder-input-output-embed', action='store_true',\n",
        "                            help='share decoder input and output embeddings')\n",
        "        parser.add_argument('--share-all-embeddings', action='store_true',\n",
        "                            help='share encoder, decoder and output embeddings'\n",
        "                                 ' (requires shared dictionary and embed dim)')\n",
        "        parser.add_argument('--dropout', type=float, metavar='D',\n",
        "                            help='dropout probability')\n",
        "        parser.add_argument('--attention-dropout', type=float, metavar='D',\n",
        "                            help='dropout probability for attention weights')\n",
        "        parser.add_argument('--relu-dropout', type=float, metavar='D',\n",
        "                            help='dropout probability after ReLU in FFN')\n",
        "        parser.add_argument('--decoder-layers', type=int, metavar='N',\n",
        "                            help='num layers')\n",
        "        parser.add_argument('--decoder-ffn-embed-dim', type=int, metavar='N',\n",
        "                            help='embedding dimension for FFN')\n",
        "        parser.add_argument('--decoder-attention-heads', type=int, metavar='N',\n",
        "                            help='num attention heads')\n",
        "        parser.add_argument('--kernel-size-list', type=lambda x: options.eval_str_list(x, int),\n",
        "                            help='list of kernel size (default: None)')\n",
        "        parser.add_argument('--language-embeddings', action='store_true',\n",
        "                            help='use language embeddings')\n",
        "\n",
        "    @classmethod\n",
        "    def build_model(cls, args, task):\n",
        "        \"\"\"Build a new model instance.\"\"\"\n",
        "\n",
        "        # make sure all arguments are present in older models\n",
        "        base_architecture(args)\n",
        "\n",
        "        if not hasattr(args, 'max_source_positions'):\n",
        "            args.max_source_positions = 1024\n",
        "        if not hasattr(args, 'max_target_positions'):\n",
        "            args.max_target_positions = 1024\n",
        "\n",
        "        src_dict, tgt_dict = task.source_dictionary, task.target_dictionary\n",
        "\n",
        "        def build_embedding(dictionary, embed_dim, path=None):\n",
        "            num_embeddings = len(dictionary)\n",
        "            padding_idx = dictionary.pad()\n",
        "            emb = Embedding(num_embeddings, embed_dim, padding_idx)\n",
        "            # if provided, load from preloaded dictionaries\n",
        "            if path:\n",
        "                embed_dict = utils.parse_embedding(path)\n",
        "                utils.load_embedding(embed_dict, dictionary, emb)\n",
        "            return emb\n",
        "\n",
        "        if args.share_all_embeddings:\n",
        "            if src_dict != tgt_dict:\n",
        "                raise ValueError('--share-all-embeddings requires a joined dictionary')\n",
        "            if args.encoder_embed_dim != args.decoder_embed_dim:\n",
        "                raise ValueError(\n",
        "                    '--share-all-embeddings requires --encoder-embed-dim to match --decoder-embed-dim')\n",
        "            if args.decoder_embed_path and (\n",
        "                    args.decoder_embed_path != args.encoder_embed_path):\n",
        "                raise ValueError('--share-all-embeddings not compatible with --decoder-embed-path')\n",
        "            encoder_embed_tokens = build_embedding(\n",
        "                src_dict, args.encoder_embed_dim, args.encoder_embed_path\n",
        "            )\n",
        "            decoder_embed_tokens = encoder_embed_tokens\n",
        "            args.share_decoder_input_output_embed = True\n",
        "        else:\n",
        "            if args.encoder_embed_dim != args.decoder_embed_dim:\n",
        "                raise ValueError(\n",
        "                    'The joint_attention model requires --encoder-embed-dim to match --decoder-embed-dim')\n",
        "            encoder_embed_tokens = build_embedding(\n",
        "                src_dict, args.encoder_embed_dim, args.encoder_embed_path\n",
        "            )\n",
        "            decoder_embed_tokens = build_embedding(\n",
        "                tgt_dict, args.decoder_embed_dim, args.decoder_embed_path\n",
        "            )\n",
        "\n",
        "        encoder = JointAttentionEncoder(args, src_dict, encoder_embed_tokens, left_pad=args.left_pad_source)\n",
        "        decoder = JointAttentionDecoder(args, tgt_dict, decoder_embed_tokens, left_pad=args.left_pad_target)\n",
        "        return JointAttentionModel(encoder, decoder)\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oBfYRr-nxsXZ",
        "colab_type": "text"
      },
      "source": [
        "#### **3.5.7. Register the Model Architecture**\n",
        "\n",
        "After having registered the new plug-in in the previous step, we can now define the desired architecture and register it as well. In the `register_model_architecture()` function decorator, the first argument should be the name of the above registered model, i.e., `'joint_attention'`, whereas the second argument corresponds to the name of the model architecture. The function we register has one argument *args*: it modifies the model in-place with the user-defined architecture parameters.\n",
        "\n",
        "As we explained before, the local joint attention model we are building here is an extension of the [joint attention model](http://papers.nips.cc/paper/8019-layer-wise-coordination-between-encoder-and-decoder-for-neural-machine-translation.pdf). For this reason we register three seperate architectures: the `'local_joint_attention_iwslt_de_en'` which is upgrades `'joint_attention_iwslt_de_en'` which is on its turn based on the base architecture `'joint_attention'` we registered before. By defining these two different model architectures in seperate functions, we can also train the other architecture (`'joint_attention_iwslt_de_en'`) and compare its performance. \n",
        "\n",
        "> **Model Parameter**  \n",
        "To later compare the translation quality of this architecture with the one of the well-known Big Transformer, we ensure that both have a similar number of trainable parameters by chosing the number of layers of our joint self-attention model accordingly.\n",
        " * Number of layers: 14\n",
        " * Embedding size: $256$\n",
        " * Feedforward expansion size: $1024$\n",
        " * Attention heads: $4$\n",
        " * Attention window sizes from input layers to output layers: $3, 5, 7, 9, 11, 13, 15, 17, 21, 25, 29, 33, 37, 41$\n",
        "\n",
        "Having registered the model and the new architecture allows us to make use of the command-line tools from fairseq to train and evaluate the new model by directly selecting the stored model architecture by specifying `--arch local_joint_attention_iwslt_de_en`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BmhpL15Axtzo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fairseq.models import register_model_architecture\n",
        "\n",
        "@register_model_architecture('joint_attention', 'joint_attention')\n",
        "def base_architecture(args):\n",
        "    args.encoder_embed_path = getattr(args, 'encoder_embed_path', None)\n",
        "    args.encoder_embed_dim = getattr(args, 'encoder_embed_dim', 512)\n",
        "    args.encoder_learned_pos = getattr(args, 'encoder_learned_pos', False)\n",
        "\n",
        "    args.decoder_embed_path = getattr(args, 'decoder_embed_path', None)\n",
        "    args.decoder_embed_dim = getattr(args, 'decoder_embed_dim', args.encoder_embed_dim)\n",
        "    args.decoder_learned_pos = getattr(args, 'decoder_learned_pos', False)\n",
        "    args.decoder_output_dim = getattr(args, 'decoder_output_dim', args.decoder_embed_dim)\n",
        "    args.decoder_input_dim = getattr(args, 'decoder_input_dim', args.decoder_embed_dim)\n",
        "\n",
        "    args.decoder_ffn_embed_dim = getattr(args, 'decoder_ffn_embed_dim', 2048)\n",
        "    args.decoder_attention_heads = getattr(args, 'decoder_attention_heads', 8)\n",
        "    args.decoder_layers = getattr(args, 'decoder_layers', 14)\n",
        "\n",
        "    args.decoder_normalize_before = getattr(args, 'decoder_normalize_before', False)\n",
        "    args.attention_dropout = getattr(args, 'attention_dropout', 0.)\n",
        "    args.relu_dropout = getattr(args, 'relu_dropout', 0.)\n",
        "    args.dropout = getattr(args, 'dropout', 0.1)\n",
        "    args.share_decoder_input_output_embed = getattr(args, 'share_decoder_input_output_embed', True)\n",
        "    args.share_all_embeddings = getattr(args, 'share_all_embeddings', False)\n",
        "    args.no_token_positional_embeddings = getattr(args, 'no_token_positional_embeddings', False)\n",
        "    args.kernel_size_list = getattr(args, 'kernel_size_list', None)\n",
        "    assert args.kernel_size_list is None or len(args.kernel_size_list) == args.decoder_layers, \"kernel_size_list doesn't match decoder_layers\"\n",
        "    args.language_embeddings = getattr(args, 'language_embeddings', True)\n",
        "\n",
        "\n",
        "@register_model_architecture('joint_attention', 'joint_attention_iwslt_de_en')\n",
        "def joint_attention_iwslt_de_en(args):\n",
        "    args.encoder_embed_dim = getattr(args, 'encoder_embed_dim', 256)\n",
        "    args.decoder_embed_dim = getattr(args, 'decoder_embed_dim', 256)\n",
        "    args.decoder_ffn_embed_dim = getattr(args, 'decoder_ffn_embed_dim', 1024)\n",
        "    args.decoder_attention_heads = getattr(args, 'decoder_attention_heads', 4)\n",
        "    args.attention_dropout = getattr(args, 'attention_dropout', 0.1)\n",
        "    args.dropout = getattr(args, 'dropout', 0.3)\n",
        "    base_architecture(args)\n",
        "\n",
        "\n",
        "@register_model_architecture('joint_attention', 'local_joint_attention_iwslt_de_en')\n",
        "def local_joint_attention_iwslt_de_en(args):\n",
        "    args.kernel_size_list = getattr(args, 'kernel_size_list', [3, 5, 7, 9, 11, 13, 15, 17, 21, 25, 29, 33, 37, 41])\n",
        "    joint_attention_iwslt_de_en(args)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwfd0KURPksN",
        "colab_type": "text"
      },
      "source": [
        "###**3.6.   Train the Built Model**####\n",
        "\n",
        "Run the `fairseq-train` command-line tool from the fairseq library to train a new model from scratch. We must specify the registered model architecture in the command-line argument `--arch local_joint_attention_iwslt_de_en`. `fairseq-train` will use all GPUs if available.\n",
        "\n",
        "> **Hyperparameters**, defined based on [Wu et al., 2019](https://arxiv.org/abs/1901.10430):\n",
        "The following hyperparameter will be specified to the `fairseq-train` comman-line arguments:\n",
        " * Optimization policy: adam optimizer with $\\epsilon = 10^{-9}$, $\\beta_1 = 0.9$ and $\\beta_2 = 0.98$\n",
        " * Batch size: $4$K source tokens. The batch size is specified in terms of the maximum number of tokens per batch `--max-tokens`\n",
        " * Training steps: $85$K\n",
        " * Learning rate: linearly warmed up for the first $4$K steps from $10^{−7}$ up to a maximum learning rate of $10^{−3}$, followed by an inverse square root scheduler with a weight decay of $10^{−4}$ until it reaches a minimum learning rate of $10^{−9}$.\n",
        " * Loss function: [cross entropy with label smoothing](https://medium.com/@nainaakash012/when-does-label-smoothing-help-89654ec75326) with a label smoothing of $0.1$.\n",
        " * Gradient Clipping: clip threshold of gradients to $0$.\n",
        " * Logging interval: every $100$ batches.\n",
        " * Checkpointing parameter: keep $10$ checkpoint files."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kEdMA6TK63Qb",
        "colab_type": "text"
      },
      "source": [
        "> ***Note***: Unfortunately, I did not find a way to train the model directly using the code we just executed in this Notebook. For this reason, we have to put all the code we just saw in a python file to be able to run the `fairseq-train` command-line. Since the code is already provided by the authors of the paper in Github, we just download the folder that has the entire code already written in python files, and then we train using those."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S40MGnFrDDLI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 884
        },
        "outputId": "a48fa090-07db-4e7f-9488-2b2e72e5d450"
      },
      "source": [
        "%cd /content/\n",
        "!apt install subversion\n",
        "!svn checkout https://github.com/jarfo/joint/trunk/models"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-440\n",
            "Use 'apt autoremove' to remove it.\n",
            "The following additional packages will be installed:\n",
            "  libapr1 libaprutil1 libserf-1-1 libsvn1\n",
            "Suggested packages:\n",
            "  db5.3-util libapache2-mod-svn subversion-tools\n",
            "The following NEW packages will be installed:\n",
            "  libapr1 libaprutil1 libserf-1-1 libsvn1 subversion\n",
            "0 upgraded, 5 newly installed, 0 to remove and 35 not upgraded.\n",
            "Need to get 2,237 kB of archives.\n",
            "After this operation, 9,910 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libapr1 amd64 1.6.3-2 [90.9 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libaprutil1 amd64 1.6.1-2 [84.4 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libserf-1-1 amd64 1.3.9-6 [44.4 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libsvn1 amd64 1.9.7-4ubuntu1 [1,183 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 subversion amd64 1.9.7-4ubuntu1 [834 kB]\n",
            "Fetched 2,237 kB in 2s (1,402 kB/s)\n",
            "Selecting previously unselected package libapr1:amd64.\n",
            "(Reading database ... 144487 files and directories currently installed.)\n",
            "Preparing to unpack .../libapr1_1.6.3-2_amd64.deb ...\n",
            "Unpacking libapr1:amd64 (1.6.3-2) ...\n",
            "Selecting previously unselected package libaprutil1:amd64.\n",
            "Preparing to unpack .../libaprutil1_1.6.1-2_amd64.deb ...\n",
            "Unpacking libaprutil1:amd64 (1.6.1-2) ...\n",
            "Selecting previously unselected package libserf-1-1:amd64.\n",
            "Preparing to unpack .../libserf-1-1_1.3.9-6_amd64.deb ...\n",
            "Unpacking libserf-1-1:amd64 (1.3.9-6) ...\n",
            "Selecting previously unselected package libsvn1:amd64.\n",
            "Preparing to unpack .../libsvn1_1.9.7-4ubuntu1_amd64.deb ...\n",
            "Unpacking libsvn1:amd64 (1.9.7-4ubuntu1) ...\n",
            "Selecting previously unselected package subversion.\n",
            "Preparing to unpack .../subversion_1.9.7-4ubuntu1_amd64.deb ...\n",
            "Unpacking subversion (1.9.7-4ubuntu1) ...\n",
            "Setting up libapr1:amd64 (1.6.3-2) ...\n",
            "Setting up libaprutil1:amd64 (1.6.1-2) ...\n",
            "Setting up libserf-1-1:amd64 (1.3.9-6) ...\n",
            "Setting up libsvn1:amd64 (1.9.7-4ubuntu1) ...\n",
            "Setting up subversion (1.9.7-4ubuntu1) ...\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
            "A    models/__init__.py\n",
            "A    models/joint.py\n",
            "A    models/protected_multihead_attention.py\n",
            "Checked out revision 38.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9U3eEUwRPZ7H",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "1c54e782-72d1-41ca-ee7c-cee0a0b7ed67"
      },
      "source": [
        "# Create a Folder to save the Checkpoints\n",
        "SAVE=\"/content/checkpoints/local_joint_attention_iwslt_de_en\"\n",
        "!mkdir -p $SAVE\n",
        "\n",
        "# Use the train function from the fairseq library to train the new model on the IWSLT 2014 dataset\n",
        "!fairseq-train data-bin/iwslt14.joined-dictionary.31K.de-en \\\n",
        "    --user-dir /content/models/ \\\n",
        "    --arch local_joint_attention_iwslt_de_en \\\n",
        "    --clip-norm 0 --optimizer adam --lr 0.001 --dropout 0.3\\\n",
        "    --source-lang de --target-lang en --max-tokens 4000 --no-progress-bar \\\n",
        "    --log-interval 100 --min-lr '1e-09' --weight-decay 0.0001 \\\n",
        "    --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \\\n",
        "    --lr-scheduler inverse_sqrt \\\n",
        "    --ddp-backend=no_c10d \\\n",
        "    --max-update 85000 --warmup-updates 4000 --warmup-init-lr '1e-07' \\\n",
        "    --adam-betas '(0.9, 0.98)' --adam-eps '1e-09' --keep-last-epochs 10 \\\n",
        "    --arch local_joint_attention_iwslt_de_en --share-all-embeddings \\\n",
        "    --save-dir $SAVE"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-08-03 20:56:08 | INFO | fairseq_cli.train | Namespace(adam_betas='(0.9, 0.98)', adam_eps=1e-09, all_gather_list_size=16384, arch='local_joint_attention_iwslt_de_en', attention_dropout=0.1, best_checkpoint_metric='loss', bf16=False, bpe=None, broadcast_buffers=False, bucket_cap_mb=25, checkpoint_suffix='', clip_norm=0.0, cpu=False, criterion='label_smoothed_cross_entropy', curriculum=0, data='data-bin/iwslt14.joined-dictionary.31K.de-en', data_buffer_size=10, dataset_impl=None, ddp_backend='no_c10d', decoder_attention_heads=4, decoder_embed_dim=256, decoder_embed_path=None, decoder_ffn_embed_dim=1024, decoder_input_dim=256, decoder_layers=14, decoder_learned_pos=False, decoder_normalize_before=False, decoder_output_dim=256, device_id=0, disable_validation=False, distributed_backend='nccl', distributed_init_method=None, distributed_no_spawn=False, distributed_port=-1, distributed_rank=0, distributed_world_size=1, distributed_wrapper='DDP', dropout=0.3, empty_cache_freq=0, encoder_embed_dim=256, encoder_embed_path=None, encoder_learned_pos=False, eval_bleu=False, eval_bleu_args=None, eval_bleu_detok='space', eval_bleu_detok_args=None, eval_bleu_print_samples=False, eval_bleu_remove_bpe=None, eval_tokenized_bleu=False, fast_stat_sync=False, find_unused_parameters=False, fix_batches_to_gpus=False, fixed_validation_seed=None, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, keep_best_checkpoints=-1, keep_interval_updates=-1, keep_last_epochs=10, kernel_size_list=[3, 5, 7, 9, 11, 13, 15, 17, 21, 25, 29, 33, 37, 41], label_smoothing=0.1, language_embeddings=True, left_pad_source='True', left_pad_target='False', load_alignments=False, localsgd_frequency=3, log_format=None, log_interval=100, lr=[0.001], lr_scheduler='inverse_sqrt', max_epoch=0, max_sentences=None, max_sentences_valid=None, max_source_positions=1024, max_target_positions=1024, max_tokens=4000, max_tokens_valid=4000, max_update=85000, maximize_best_checkpoint_metric=False, memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, min_lr=1e-09, model_parallel_size=1, no_epoch_checkpoints=False, no_last_checkpoints=False, no_progress_bar=True, no_save=False, no_save_optimizer_state=False, no_seed_provided=True, no_token_positional_embeddings=False, nprocs_per_node=1, num_batch_buckets=0, num_workers=1, optimizer='adam', optimizer_overrides='{}', patience=-1, profile=False, quantization_config_path=None, relu_dropout=0.0, required_batch_size_multiple=8, reset_dataloader=False, reset_lr_scheduler=False, reset_meters=False, reset_optimizer=False, restore_file='checkpoint_last.pt', save_dir='/content/checkpoints/local_joint_attention_iwslt_de_en', save_interval=1, save_interval_updates=0, seed=1, sentence_avg=False, share_all_embeddings=True, share_decoder_input_output_embed=True, skip_invalid_size_inputs_valid_test=False, slowmo_algorithm='LocalSGD', slowmo_momentum=None, source_lang='de', stop_time_hours=0, target_lang='en', task='translation', tensorboard_logdir='', threshold_loss_scale=None, tokenizer=None, tpu=False, train_subset='train', truncate_source=False, update_freq=[1], upsample_primary=1, use_bmuf=False, use_old_adam=False, user_dir='/content/models/', valid_subset='valid', validate_interval=1, warmup_init_lr=1e-07, warmup_updates=4000, weight_decay=0.0001)\n",
            "2020-08-03 20:56:08 | INFO | fairseq.tasks.translation | [de] dictionary: 30760 types\n",
            "2020-08-03 20:56:08 | INFO | fairseq.tasks.translation | [en] dictionary: 30760 types\n",
            "2020-08-03 20:56:08 | INFO | fairseq.data.data_utils | loaded 7283 examples from: data-bin/iwslt14.joined-dictionary.31K.de-en/valid.de-en.de\n",
            "2020-08-03 20:56:08 | INFO | fairseq.data.data_utils | loaded 7283 examples from: data-bin/iwslt14.joined-dictionary.31K.de-en/valid.de-en.en\n",
            "2020-08-03 20:56:08 | INFO | fairseq.tasks.translation | data-bin/iwslt14.joined-dictionary.31K.de-en valid de-en 7283 examples\n",
            "2020-08-03 20:56:08 | INFO | fairseq_cli.train | JointAttentionModel(\n",
            "  (encoder): JointAttentionEncoder(\n",
            "    (embed_tokens): Embedding(30760, 256, padding_idx=1)\n",
            "    (embed_positions): SinusoidalPositionalEmbedding()\n",
            "  )\n",
            "  (decoder): JointAttentionDecoder(\n",
            "    (embed_tokens): Embedding(30760, 256, padding_idx=1)\n",
            "    (embed_positions): SinusoidalPositionalEmbedding()\n",
            "    (layers): ModuleList(\n",
            "      (0): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (1): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (2): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (3): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (4): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (5): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (6): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (7): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (8): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (9): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (10): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (11): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (12): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "      (13): ProtectedTransformerDecoderLayer(\n",
            "        (self_attn): ProtectedMultiheadAttention(\n",
            "          (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
            "        )\n",
            "        (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "        (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
            "        (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
            "        (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n",
            "2020-08-03 20:56:08 | INFO | fairseq_cli.train | model local_joint_attention_iwslt_de_en, criterion LabelSmoothedCrossEntropyCriterion\n",
            "2020-08-03 20:56:08 | INFO | fairseq_cli.train | num. model params: 18931712 (num. trained: 18931712)\n",
            "2020-08-03 20:56:18 | INFO | fairseq.trainer | detected shared parameter: encoder.embed_tokens.weight <- decoder.embed_tokens.weight\n",
            "2020-08-03 20:56:18 | INFO | fairseq.utils | ***********************CUDA enviroments for all 1 workers***********************\n",
            "2020-08-03 20:56:18 | INFO | fairseq.utils | rank   0: capabilities =  7.5  ; total memory = 14.726 GB ; name = Tesla T4                                \n",
            "2020-08-03 20:56:18 | INFO | fairseq.utils | ***********************CUDA enviroments for all 1 workers***********************\n",
            "2020-08-03 20:56:18 | INFO | fairseq_cli.train | training on 1 devices (GPUs/TPUs)\n",
            "2020-08-03 20:56:18 | INFO | fairseq_cli.train | max tokens per GPU = 4000 and max sentences per GPU = None\n",
            "2020-08-03 20:56:18 | INFO | fairseq.trainer | no existing checkpoint found /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_last.pt\n",
            "2020-08-03 20:56:18 | INFO | fairseq.trainer | loading train data for epoch 1\n",
            "2020-08-03 20:56:18 | INFO | fairseq.data.data_utils | loaded 160239 examples from: data-bin/iwslt14.joined-dictionary.31K.de-en/train.de-en.de\n",
            "2020-08-03 20:56:18 | INFO | fairseq.data.data_utils | loaded 160239 examples from: data-bin/iwslt14.joined-dictionary.31K.de-en/train.de-en.en\n",
            "2020-08-03 20:56:18 | INFO | fairseq.tasks.translation | data-bin/iwslt14.joined-dictionary.31K.de-en train de-en 160239 examples\n",
            "2020-08-03 20:56:19 | INFO | fairseq.trainer | NOTE: your device may support faster training with --fp16\n",
            "2020-08-03 20:56:19 | INFO | fairseq_cli.train | begin training epoch 1\n",
            "/content/fairseq/fairseq/utils.py:305: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library\n",
            "  \"amp_C fused kernels unavailable, disabling multi_tensor_l2norm; \"\n",
            "2020-08-03 20:56:49 | INFO | train_inner | epoch 001:    100 / 1015 loss=14.433, nll_loss=14.3, ppl=20169.1, wps=12192.3, ups=3.41, wpb=3570.6, bsz=149.6, num_updates=100, lr=2.50975e-05, gnorm=2.738, train_wall=30, wall=31\n",
            "2020-08-03 20:57:19 | INFO | train_inner | epoch 001:    200 / 1015 loss=12.793, nll_loss=12.478, ppl=5706.67, wps=11794.5, ups=3.33, wpb=3537.7, bsz=151.9, num_updates=200, lr=5.0095e-05, gnorm=1.574, train_wall=30, wall=61\n",
            "2020-08-03 20:57:50 | INFO | train_inner | epoch 001:    300 / 1015 loss=11.211, nll_loss=10.694, ppl=1656.71, wps=11347.2, ups=3.21, wpb=3532.8, bsz=166.9, num_updates=300, lr=7.50925e-05, gnorm=1.435, train_wall=31, wall=92\n",
            "2020-08-03 20:58:21 | INFO | train_inner | epoch 001:    400 / 1015 loss=10.212, nll_loss=9.504, ppl=726.05, wps=11729.6, ups=3.26, wpb=3594.5, bsz=149.5, num_updates=400, lr=0.00010009, gnorm=1.168, train_wall=31, wall=123\n",
            "2020-08-03 20:58:51 | INFO | train_inner | epoch 001:    500 / 1015 loss=9.913, nll_loss=9.11, ppl=552.56, wps=11929.4, ups=3.27, wpb=3642.8, bsz=158.2, num_updates=500, lr=0.000125087, gnorm=1.227, train_wall=30, wall=153\n",
            "2020-08-03 20:59:22 | INFO | train_inner | epoch 001:    600 / 1015 loss=9.732, nll_loss=8.893, ppl=475.24, wps=11548.2, ups=3.26, wpb=3541.8, bsz=180.9, num_updates=600, lr=0.000150085, gnorm=1.342, train_wall=31, wall=184\n",
            "2020-08-03 20:59:53 | INFO | train_inner | epoch 001:    700 / 1015 loss=9.733, nll_loss=8.892, ppl=475.02, wps=11508.6, ups=3.26, wpb=3527.6, bsz=150.2, num_updates=700, lr=0.000175082, gnorm=1.251, train_wall=31, wall=215\n",
            "2020-08-03 21:00:23 | INFO | train_inner | epoch 001:    800 / 1015 loss=9.567, nll_loss=8.704, ppl=417.01, wps=11572, ups=3.25, wpb=3556.8, bsz=170.3, num_updates=800, lr=0.00020008, gnorm=1.195, train_wall=31, wall=245\n",
            "2020-08-03 21:00:54 | INFO | train_inner | epoch 001:    900 / 1015 loss=9.579, nll_loss=8.714, ppl=420.04, wps=11336, ups=3.26, wpb=3482.1, bsz=148.8, num_updates=900, lr=0.000225077, gnorm=1.285, train_wall=31, wall=276\n",
            "2020-08-03 21:01:25 | INFO | train_inner | epoch 001:   1000 / 1015 loss=9.418, nll_loss=8.535, ppl=370.92, wps=11380, ups=3.24, wpb=3511.3, bsz=152, num_updates=1000, lr=0.000250075, gnorm=1.041, train_wall=31, wall=307\n",
            "2020-08-03 21:01:29 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:01:36 | INFO | valid | epoch 001 | valid on 'valid' subset | loss 9.173 | nll_loss 8.237 | ppl 301.79 | wps 26972.8 | wpb 2866.6 | bsz 127.8 | num_updates 1015\n",
            "2020-08-03 21:01:36 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:01:38 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint1.pt (epoch 1 @ 1015 updates, score 9.173) (writing took 2.605313768999963 seconds)\n",
            "2020-08-03 21:01:38 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:01:38 | INFO | fairseq_cli.train | end of epoch 1 (average epoch stats below)\n",
            "2020-08-03 21:01:38 | INFO | train | epoch 001 | loss 10.642 | nll_loss 9.963 | ppl 997.79 | wps 11308.7 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 1015 | lr 0.000253825 | gnorm 1.42 | train_wall 309 | wall 320\n",
            "2020-08-03 21:01:38 | INFO | fairseq_cli.train | begin training epoch 1\n",
            "2020-08-03 21:02:05 | INFO | train_inner | epoch 002:     85 / 1015 loss=9.277, nll_loss=8.377, ppl=332.37, wps=8989.3, ups=2.52, wpb=3567.2, bsz=147, num_updates=1100, lr=0.000275072, gnorm=1.044, train_wall=31, wall=347\n",
            "2020-08-03 21:02:35 | INFO | train_inner | epoch 002:    185 / 1015 loss=8.98, nll_loss=8.039, ppl=262.96, wps=11639.2, ups=3.26, wpb=3572, bsz=173.3, num_updates=1200, lr=0.00030007, gnorm=1.153, train_wall=31, wall=377\n",
            "2020-08-03 21:03:06 | INFO | train_inner | epoch 002:    285 / 1015 loss=8.9, nll_loss=7.949, ppl=247.04, wps=11717.8, ups=3.25, wpb=3602.9, bsz=164.9, num_updates=1300, lr=0.000325067, gnorm=1.05, train_wall=31, wall=408\n",
            "2020-08-03 21:03:37 | INFO | train_inner | epoch 002:    385 / 1015 loss=8.828, nll_loss=7.866, ppl=233.23, wps=11557.5, ups=3.24, wpb=3566.5, bsz=164.5, num_updates=1400, lr=0.000350065, gnorm=1.034, train_wall=31, wall=439\n",
            "2020-08-03 21:04:08 | INFO | train_inner | epoch 002:    485 / 1015 loss=8.82, nll_loss=7.856, ppl=231.71, wps=11604.5, ups=3.25, wpb=3574.6, bsz=144.5, num_updates=1500, lr=0.000375062, gnorm=1.028, train_wall=31, wall=470\n",
            "2020-08-03 21:04:38 | INFO | train_inner | epoch 002:    585 / 1015 loss=8.649, nll_loss=7.661, ppl=202.38, wps=11614.8, ups=3.26, wpb=3561.2, bsz=166.6, num_updates=1600, lr=0.00040006, gnorm=1.057, train_wall=31, wall=500\n",
            "2020-08-03 21:05:09 | INFO | train_inner | epoch 002:    685 / 1015 loss=8.612, nll_loss=7.619, ppl=196.6, wps=11518.6, ups=3.24, wpb=3549.9, bsz=157.5, num_updates=1700, lr=0.000425057, gnorm=1.05, train_wall=31, wall=531\n",
            "2020-08-03 21:05:40 | INFO | train_inner | epoch 002:    785 / 1015 loss=8.609, nll_loss=7.615, ppl=196.07, wps=11354, ups=3.28, wpb=3466.6, bsz=151.2, num_updates=1800, lr=0.000450055, gnorm=1.087, train_wall=30, wall=562\n",
            "2020-08-03 21:06:11 | INFO | train_inner | epoch 002:    885 / 1015 loss=8.427, nll_loss=7.409, ppl=170.01, wps=11584.4, ups=3.24, wpb=3578, bsz=164.5, num_updates=1900, lr=0.000475052, gnorm=1.01, train_wall=31, wall=593\n",
            "2020-08-03 21:06:41 | INFO | train_inner | epoch 002:    985 / 1015 loss=8.425, nll_loss=7.405, ppl=169.43, wps=11503.5, ups=3.3, wpb=3484.6, bsz=157.4, num_updates=2000, lr=0.00050005, gnorm=1.127, train_wall=30, wall=623\n",
            "2020-08-03 21:06:50 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:06:56 | INFO | valid | epoch 002 | valid on 'valid' subset | loss 8.153 | nll_loss 7.079 | ppl 135.18 | wps 27072.3 | wpb 2866.6 | bsz 127.8 | num_updates 2030 | best_loss 8.153\n",
            "2020-08-03 21:06:56 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:06:59 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint2.pt (epoch 2 @ 2030 updates, score 8.153) (writing took 2.6128047629999855 seconds)\n",
            "2020-08-03 21:06:59 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:06:59 | INFO | fairseq_cli.train | end of epoch 2 (average epoch stats below)\n",
            "2020-08-03 21:06:59 | INFO | train | epoch 002 | loss 8.74 | nll_loss 7.765 | ppl 217.57 | wps 11245.1 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 2030 | lr 0.000507549 | gnorm 1.061 | train_wall 311 | wall 641\n",
            "2020-08-03 21:06:59 | INFO | fairseq_cli.train | begin training epoch 2\n",
            "2020-08-03 21:07:20 | INFO | train_inner | epoch 003:     70 / 1015 loss=8.409, nll_loss=7.389, ppl=167.62, wps=9017.2, ups=2.54, wpb=3550.7, bsz=138.9, num_updates=2100, lr=0.000525047, gnorm=1.01, train_wall=30, wall=662\n",
            "2020-08-03 21:07:51 | INFO | train_inner | epoch 003:    170 / 1015 loss=8.188, nll_loss=7.137, ppl=140.76, wps=11648.8, ups=3.25, wpb=3582, bsz=162.4, num_updates=2200, lr=0.000550045, gnorm=1.052, train_wall=31, wall=693\n",
            "2020-08-03 21:08:22 | INFO | train_inner | epoch 003:    270 / 1015 loss=8.17, nll_loss=7.118, ppl=138.89, wps=11640.7, ups=3.27, wpb=3565.2, bsz=154.8, num_updates=2300, lr=0.000575042, gnorm=1.001, train_wall=31, wall=724\n",
            "2020-08-03 21:08:52 | INFO | train_inner | epoch 003:    370 / 1015 loss=8.194, nll_loss=7.145, ppl=141.54, wps=11556.4, ups=3.26, wpb=3539.7, bsz=144, num_updates=2400, lr=0.00060004, gnorm=1.009, train_wall=31, wall=754\n",
            "2020-08-03 21:09:23 | INFO | train_inner | epoch 003:    470 / 1015 loss=8.009, nll_loss=6.935, ppl=122.39, wps=11619.5, ups=3.29, wpb=3535.3, bsz=158.4, num_updates=2500, lr=0.000625037, gnorm=1.007, train_wall=30, wall=785\n",
            "2020-08-03 21:09:53 | INFO | train_inner | epoch 003:    570 / 1015 loss=7.868, nll_loss=6.775, ppl=109.52, wps=11561.2, ups=3.29, wpb=3511.4, bsz=169.3, num_updates=2600, lr=0.000650035, gnorm=1.052, train_wall=30, wall=815\n",
            "2020-08-03 21:10:24 | INFO | train_inner | epoch 003:    670 / 1015 loss=7.777, nll_loss=6.672, ppl=102, wps=11583.9, ups=3.25, wpb=3559.1, bsz=168.8, num_updates=2700, lr=0.000675032, gnorm=0.981, train_wall=31, wall=846\n",
            "2020-08-03 21:10:54 | INFO | train_inner | epoch 003:    770 / 1015 loss=7.878, nll_loss=6.786, ppl=110.39, wps=11530.7, ups=3.29, wpb=3509.3, bsz=143.6, num_updates=2800, lr=0.00070003, gnorm=0.993, train_wall=30, wall=876\n",
            "2020-08-03 21:11:25 | INFO | train_inner | epoch 003:    870 / 1015 loss=7.679, nll_loss=6.561, ppl=94.45, wps=11536.7, ups=3.27, wpb=3533.4, bsz=165.2, num_updates=2900, lr=0.000725027, gnorm=1.043, train_wall=31, wall=907\n",
            "2020-08-03 21:11:55 | INFO | train_inner | epoch 003:    970 / 1015 loss=7.651, nll_loss=6.529, ppl=92.35, wps=11752.7, ups=3.29, wpb=3568, bsz=153.4, num_updates=3000, lr=0.000750025, gnorm=1.023, train_wall=30, wall=937\n",
            "2020-08-03 21:12:09 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:12:15 | INFO | valid | epoch 003 | valid on 'valid' subset | loss 7.417 | nll_loss 6.209 | ppl 73.99 | wps 27066.7 | wpb 2866.6 | bsz 127.8 | num_updates 3045 | best_loss 7.417\n",
            "2020-08-03 21:12:15 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:12:18 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint3.pt (epoch 3 @ 3045 updates, score 7.417) (writing took 2.6153591289998985 seconds)\n",
            "2020-08-03 21:12:18 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:12:18 | INFO | fairseq_cli.train | end of epoch 3 (average epoch stats below)\n",
            "2020-08-03 21:12:18 | INFO | train | epoch 003 | loss 7.943 | nll_loss 6.861 | ppl 116.24 | wps 11292 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 3045 | lr 0.000761274 | gnorm 1.02 | train_wall 309 | wall 960\n",
            "2020-08-03 21:12:18 | INFO | fairseq_cli.train | begin training epoch 3\n",
            "2020-08-03 21:12:35 | INFO | train_inner | epoch 004:     55 / 1015 loss=7.559, nll_loss=6.426, ppl=85.98, wps=8960.2, ups=2.53, wpb=3538.8, bsz=162, num_updates=3100, lr=0.000775022, gnorm=1.063, train_wall=31, wall=977\n",
            "2020-08-03 21:13:05 | INFO | train_inner | epoch 004:    155 / 1015 loss=7.59, nll_loss=6.459, ppl=87.96, wps=11479.9, ups=3.27, wpb=3510.8, bsz=137.8, num_updates=3200, lr=0.00080002, gnorm=1.044, train_wall=30, wall=1007\n",
            "2020-08-03 21:13:36 | INFO | train_inner | epoch 004:    255 / 1015 loss=7.458, nll_loss=6.309, ppl=79.28, wps=11777.2, ups=3.26, wpb=3614.1, bsz=151.2, num_updates=3300, lr=0.000825017, gnorm=1.031, train_wall=31, wall=1038\n",
            "2020-08-03 21:14:07 | INFO | train_inner | epoch 004:    355 / 1015 loss=7.363, nll_loss=6.2, ppl=73.52, wps=11785.5, ups=3.26, wpb=3612.6, bsz=151.2, num_updates=3400, lr=0.000850015, gnorm=1.03, train_wall=31, wall=1069\n",
            "2020-08-03 21:14:37 | INFO | train_inner | epoch 004:    455 / 1015 loss=7.386, nll_loss=6.227, ppl=74.89, wps=11619.3, ups=3.28, wpb=3546.2, bsz=143.4, num_updates=3500, lr=0.000875012, gnorm=1.03, train_wall=30, wall=1099\n",
            "2020-08-03 21:15:07 | INFO | train_inner | epoch 004:    555 / 1015 loss=7.198, nll_loss=6.014, ppl=64.61, wps=11613.2, ups=3.3, wpb=3514.9, bsz=177.5, num_updates=3600, lr=0.00090001, gnorm=1.108, train_wall=30, wall=1129\n",
            "2020-08-03 21:15:38 | INFO | train_inner | epoch 004:    655 / 1015 loss=7.194, nll_loss=6.007, ppl=64.33, wps=11588.9, ups=3.28, wpb=3531.2, bsz=156.5, num_updates=3700, lr=0.000925007, gnorm=1.1, train_wall=30, wall=1160\n",
            "2020-08-03 21:16:08 | INFO | train_inner | epoch 004:    755 / 1015 loss=7.077, nll_loss=5.875, ppl=58.7, wps=11714.4, ups=3.27, wpb=3585.6, bsz=179.1, num_updates=3800, lr=0.000950005, gnorm=1.087, train_wall=31, wall=1191\n",
            "2020-08-03 21:16:39 | INFO | train_inner | epoch 004:    855 / 1015 loss=7.1, nll_loss=5.901, ppl=59.74, wps=11562.6, ups=3.29, wpb=3511, bsz=156, num_updates=3900, lr=0.000975002, gnorm=1.087, train_wall=30, wall=1221\n",
            "2020-08-03 21:17:09 | INFO | train_inner | epoch 004:    955 / 1015 loss=7.036, nll_loss=5.827, ppl=56.77, wps=11662.2, ups=3.26, wpb=3579.4, bsz=156.9, num_updates=4000, lr=0.001, gnorm=1.09, train_wall=31, wall=1252\n",
            "2020-08-03 21:17:28 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:17:34 | INFO | valid | epoch 004 | valid on 'valid' subset | loss 6.687 | nll_loss 5.363 | ppl 41.16 | wps 26938.5 | wpb 2866.6 | bsz 127.8 | num_updates 4060 | best_loss 6.687\n",
            "2020-08-03 21:17:34 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:17:37 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint4.pt (epoch 4 @ 4060 updates, score 6.687) (writing took 2.5782605889999104 seconds)\n",
            "2020-08-03 21:17:37 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:17:37 | INFO | fairseq_cli.train | end of epoch 4 (average epoch stats below)\n",
            "2020-08-03 21:17:37 | INFO | train | epoch 004 | loss 7.262 | nll_loss 6.086 | ppl 67.91 | wps 11305.5 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 4060 | lr 0.000992583 | gnorm 1.069 | train_wall 309 | wall 1279\n",
            "2020-08-03 21:17:37 | INFO | fairseq_cli.train | begin training epoch 4\n",
            "2020-08-03 21:17:49 | INFO | train_inner | epoch 005:     40 / 1015 loss=6.885, nll_loss=5.657, ppl=50.45, wps=8948.6, ups=2.53, wpb=3530.6, bsz=173.2, num_updates=4100, lr=0.00098773, gnorm=1.098, train_wall=31, wall=1291\n",
            "2020-08-03 21:18:20 | INFO | train_inner | epoch 005:    140 / 1015 loss=6.731, nll_loss=5.478, ppl=44.58, wps=11604.2, ups=3.27, wpb=3550.9, bsz=175.8, num_updates=4200, lr=0.0009759, gnorm=1.121, train_wall=30, wall=1322\n",
            "2020-08-03 21:18:50 | INFO | train_inner | epoch 005:    240 / 1015 loss=6.831, nll_loss=5.593, ppl=48.26, wps=11611.4, ups=3.27, wpb=3552.7, bsz=155.4, num_updates=4300, lr=0.000964486, gnorm=1.117, train_wall=30, wall=1352\n",
            "2020-08-03 21:19:21 | INFO | train_inner | epoch 005:    340 / 1015 loss=6.794, nll_loss=5.55, ppl=46.86, wps=11513, ups=3.29, wpb=3496.6, bsz=157.7, num_updates=4400, lr=0.000953463, gnorm=1.13, train_wall=30, wall=1383\n",
            "2020-08-03 21:19:51 | INFO | train_inner | epoch 005:    440 / 1015 loss=6.797, nll_loss=5.552, ppl=46.91, wps=11572.9, ups=3.29, wpb=3517.2, bsz=139.4, num_updates=4500, lr=0.000942809, gnorm=1.111, train_wall=30, wall=1413\n",
            "2020-08-03 21:20:21 | INFO | train_inner | epoch 005:    540 / 1015 loss=6.638, nll_loss=5.37, ppl=41.36, wps=11639.7, ups=3.29, wpb=3537.9, bsz=152.9, num_updates=4600, lr=0.000932505, gnorm=1.101, train_wall=30, wall=1443\n",
            "2020-08-03 21:20:52 | INFO | train_inner | epoch 005:    640 / 1015 loss=6.528, nll_loss=5.246, ppl=37.96, wps=11677.2, ups=3.26, wpb=3583.7, bsz=165.4, num_updates=4700, lr=0.000922531, gnorm=1.053, train_wall=31, wall=1474\n",
            "2020-08-03 21:21:23 | INFO | train_inner | epoch 005:    740 / 1015 loss=6.442, nll_loss=5.147, ppl=35.44, wps=11699.4, ups=3.27, wpb=3581.3, bsz=171.4, num_updates=4800, lr=0.000912871, gnorm=1.114, train_wall=31, wall=1505\n",
            "2020-08-03 21:21:53 | INFO | train_inner | epoch 005:    840 / 1015 loss=6.574, nll_loss=5.297, ppl=39.31, wps=11673.9, ups=3.28, wpb=3557.8, bsz=147.1, num_updates=4900, lr=0.000903508, gnorm=1.127, train_wall=30, wall=1535\n",
            "2020-08-03 21:22:24 | INFO | train_inner | epoch 005:    940 / 1015 loss=6.467, nll_loss=5.174, ppl=36.11, wps=11611, ups=3.28, wpb=3542.8, bsz=151.7, num_updates=5000, lr=0.000894427, gnorm=1.1, train_wall=30, wall=1566\n",
            "2020-08-03 21:22:47 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:22:53 | INFO | valid | epoch 005 | valid on 'valid' subset | loss 6.055 | nll_loss 4.611 | ppl 24.44 | wps 27036.2 | wpb 2866.6 | bsz 127.8 | num_updates 5075 | best_loss 6.055\n",
            "2020-08-03 21:22:53 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:22:55 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint5.pt (epoch 5 @ 5075 updates, score 6.055) (writing took 2.645615469999939 seconds)\n",
            "2020-08-03 21:22:55 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:22:55 | INFO | fairseq_cli.train | end of epoch 5 (average epoch stats below)\n",
            "2020-08-03 21:22:55 | INFO | train | epoch 005 | loss 6.633 | nll_loss 5.365 | ppl 41.21 | wps 11298.5 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 5075 | lr 0.000887794 | gnorm 1.107 | train_wall 309 | wall 1598\n",
            "2020-08-03 21:22:55 | INFO | fairseq_cli.train | begin training epoch 5\n",
            "2020-08-03 21:23:03 | INFO | train_inner | epoch 006:     25 / 1015 loss=6.328, nll_loss=5.016, ppl=32.37, wps=9009.2, ups=2.51, wpb=3582.8, bsz=159.8, num_updates=5100, lr=0.000885615, gnorm=1.095, train_wall=31, wall=1605\n",
            "2020-08-03 21:23:34 | INFO | train_inner | epoch 006:    125 / 1015 loss=6.242, nll_loss=4.916, ppl=30.19, wps=11591.4, ups=3.26, wpb=3559.5, bsz=164.9, num_updates=5200, lr=0.000877058, gnorm=1.123, train_wall=31, wall=1636\n",
            "2020-08-03 21:24:05 | INFO | train_inner | epoch 006:    225 / 1015 loss=6.29, nll_loss=4.971, ppl=31.35, wps=11562.8, ups=3.29, wpb=3519.6, bsz=140.8, num_updates=5300, lr=0.000868744, gnorm=1.122, train_wall=30, wall=1667\n",
            "2020-08-03 21:24:35 | INFO | train_inner | epoch 006:    325 / 1015 loss=6.263, nll_loss=4.938, ppl=30.66, wps=11527.1, ups=3.28, wpb=3517.6, bsz=148.2, num_updates=5400, lr=0.000860663, gnorm=1.105, train_wall=30, wall=1697\n",
            "2020-08-03 21:25:06 | INFO | train_inner | epoch 006:    425 / 1015 loss=6.181, nll_loss=4.845, ppl=28.74, wps=11541.7, ups=3.25, wpb=3548.2, bsz=150.7, num_updates=5500, lr=0.000852803, gnorm=1.087, train_wall=31, wall=1728\n",
            "2020-08-03 21:25:36 | INFO | train_inner | epoch 006:    525 / 1015 loss=6.096, nll_loss=4.748, ppl=26.87, wps=11539.9, ups=3.26, wpb=3534.8, bsz=167.8, num_updates=5600, lr=0.000845154, gnorm=1.113, train_wall=31, wall=1759\n",
            "2020-08-03 21:26:07 | INFO | train_inner | epoch 006:    625 / 1015 loss=5.946, nll_loss=4.576, ppl=23.85, wps=11755, ups=3.29, wpb=3577.8, bsz=179.4, num_updates=5700, lr=0.000837708, gnorm=1.036, train_wall=30, wall=1789\n",
            "2020-08-03 21:26:38 | INFO | train_inner | epoch 006:    725 / 1015 loss=6.004, nll_loss=4.643, ppl=24.98, wps=11488.3, ups=3.25, wpb=3534.2, bsz=159.8, num_updates=5800, lr=0.000830455, gnorm=1.079, train_wall=31, wall=1820\n",
            "2020-08-03 21:27:09 | INFO | train_inner | epoch 006:    825 / 1015 loss=6.106, nll_loss=4.757, ppl=27.04, wps=11489.7, ups=3.23, wpb=3555.2, bsz=149.8, num_updates=5900, lr=0.000823387, gnorm=1.159, train_wall=31, wall=1851\n",
            "2020-08-03 21:27:39 | INFO | train_inner | epoch 006:    925 / 1015 loss=5.96, nll_loss=4.591, ppl=24.1, wps=11801.1, ups=3.25, wpb=3626.1, bsz=165.4, num_updates=6000, lr=0.000816497, gnorm=1.106, train_wall=31, wall=1881\n",
            "2020-08-03 21:28:07 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:28:13 | INFO | valid | epoch 006 | valid on 'valid' subset | loss 5.651 | nll_loss 4.159 | ppl 17.86 | wps 27019.5 | wpb 2866.6 | bsz 127.8 | num_updates 6090 | best_loss 5.651\n",
            "2020-08-03 21:28:13 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:28:15 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint6.pt (epoch 6 @ 6090 updates, score 5.651) (writing took 2.5987908580000294 seconds)\n",
            "2020-08-03 21:28:15 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:28:15 | INFO | fairseq_cli.train | end of epoch 6 (average epoch stats below)\n",
            "2020-08-03 21:28:15 | INFO | train | epoch 006 | loss 6.115 | nll_loss 4.769 | ppl 27.27 | wps 11267.1 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 6090 | lr 0.000810441 | gnorm 1.103 | train_wall 310 | wall 1917\n",
            "2020-08-03 21:28:15 | INFO | fairseq_cli.train | begin training epoch 6\n",
            "2020-08-03 21:28:19 | INFO | train_inner | epoch 007:     10 / 1015 loss=6.019, nll_loss=4.658, ppl=25.25, wps=9014.8, ups=2.55, wpb=3537.3, bsz=149.6, num_updates=6100, lr=0.000809776, gnorm=1.086, train_wall=30, wall=1921\n",
            "2020-08-03 21:28:49 | INFO | train_inner | epoch 007:    110 / 1015 loss=5.774, nll_loss=4.379, ppl=20.81, wps=11700.8, ups=3.25, wpb=3596.6, bsz=164, num_updates=6200, lr=0.000803219, gnorm=1.081, train_wall=31, wall=1951\n",
            "2020-08-03 21:29:20 | INFO | train_inner | epoch 007:    210 / 1015 loss=5.849, nll_loss=4.462, ppl=22.04, wps=11493.3, ups=3.27, wpb=3516.8, bsz=151.4, num_updates=6300, lr=0.000796819, gnorm=1.15, train_wall=30, wall=1982\n",
            "2020-08-03 21:29:50 | INFO | train_inner | epoch 007:    310 / 1015 loss=5.746, nll_loss=4.344, ppl=20.31, wps=11816.5, ups=3.26, wpb=3621.6, bsz=161.4, num_updates=6400, lr=0.000790569, gnorm=1.074, train_wall=31, wall=2013\n",
            "2020-08-03 21:30:21 | INFO | train_inner | epoch 007:    410 / 1015 loss=5.817, nll_loss=4.425, ppl=21.48, wps=11338, ups=3.28, wpb=3459.4, bsz=159.4, num_updates=6500, lr=0.000784465, gnorm=1.115, train_wall=30, wall=2043\n",
            "2020-08-03 21:30:52 | INFO | train_inner | epoch 007:    510 / 1015 loss=5.801, nll_loss=4.406, ppl=21.2, wps=11597.9, ups=3.27, wpb=3541.6, bsz=155.9, num_updates=6600, lr=0.000778499, gnorm=1.132, train_wall=30, wall=2074\n",
            "2020-08-03 21:31:22 | INFO | train_inner | epoch 007:    610 / 1015 loss=5.744, nll_loss=4.341, ppl=20.27, wps=11550.8, ups=3.28, wpb=3524.7, bsz=160, num_updates=6700, lr=0.000772667, gnorm=1.084, train_wall=30, wall=2104\n",
            "2020-08-03 21:31:53 | INFO | train_inner | epoch 007:    710 / 1015 loss=5.808, nll_loss=4.412, ppl=21.29, wps=11551.7, ups=3.27, wpb=3534.6, bsz=140.6, num_updates=6800, lr=0.000766965, gnorm=1.076, train_wall=30, wall=2135\n",
            "2020-08-03 21:32:23 | INFO | train_inner | epoch 007:    810 / 1015 loss=5.617, nll_loss=4.196, ppl=18.33, wps=11881.9, ups=3.24, wpb=3665.3, bsz=169.5, num_updates=6900, lr=0.000761387, gnorm=1.047, train_wall=31, wall=2166\n",
            "2020-08-03 21:32:54 | INFO | train_inner | epoch 007:    910 / 1015 loss=5.688, nll_loss=4.277, ppl=19.38, wps=11461.3, ups=3.29, wpb=3486.5, bsz=157.4, num_updates=7000, lr=0.000755929, gnorm=1.094, train_wall=30, wall=2196\n",
            "2020-08-03 21:33:24 | INFO | train_inner | epoch 007:   1010 / 1015 loss=5.673, nll_loss=4.26, ppl=19.16, wps=11642.2, ups=3.28, wpb=3546.1, bsz=162.8, num_updates=7100, lr=0.000750587, gnorm=1.113, train_wall=30, wall=2226\n",
            "2020-08-03 21:33:26 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:33:32 | INFO | valid | epoch 007 | valid on 'valid' subset | loss 5.362 | nll_loss 3.827 | ppl 14.19 | wps 27084.8 | wpb 2866.6 | bsz 127.8 | num_updates 7105 | best_loss 5.362\n",
            "2020-08-03 21:33:32 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:33:35 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint7.pt (epoch 7 @ 7105 updates, score 5.362) (writing took 2.63321993999989 seconds)\n",
            "2020-08-03 21:33:35 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:33:35 | INFO | fairseq_cli.train | end of epoch 7 (average epoch stats below)\n",
            "2020-08-03 21:33:35 | INFO | train | epoch 007 | loss 5.753 | nll_loss 4.352 | ppl 20.42 | wps 11280 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 7105 | lr 0.000750322 | gnorm 1.096 | train_wall 310 | wall 2237\n",
            "2020-08-03 21:33:35 | INFO | fairseq_cli.train | begin training epoch 7\n",
            "2020-08-03 21:34:04 | INFO | train_inner | epoch 008:     95 / 1015 loss=5.567, nll_loss=4.138, ppl=17.61, wps=8911.8, ups=2.51, wpb=3545.8, bsz=154.2, num_updates=7200, lr=0.000745356, gnorm=1.092, train_wall=31, wall=2266\n",
            "2020-08-03 21:34:34 | INFO | train_inner | epoch 008:    195 / 1015 loss=5.544, nll_loss=4.111, ppl=17.28, wps=11594.9, ups=3.3, wpb=3513.8, bsz=163.6, num_updates=7300, lr=0.000740233, gnorm=1.145, train_wall=30, wall=2297\n",
            "2020-08-03 21:35:05 | INFO | train_inner | epoch 008:    295 / 1015 loss=5.47, nll_loss=4.026, ppl=16.29, wps=11687.1, ups=3.29, wpb=3554.5, bsz=171.4, num_updates=7400, lr=0.000735215, gnorm=1.075, train_wall=30, wall=2327\n",
            "2020-08-03 21:35:35 | INFO | train_inner | epoch 008:    395 / 1015 loss=5.49, nll_loss=4.049, ppl=16.55, wps=11674.8, ups=3.27, wpb=3571.1, bsz=155.8, num_updates=7500, lr=0.000730297, gnorm=1.061, train_wall=30, wall=2358\n",
            "2020-08-03 21:36:06 | INFO | train_inner | epoch 008:    495 / 1015 loss=5.462, nll_loss=4.017, ppl=16.19, wps=11602.6, ups=3.28, wpb=3538.7, bsz=159.6, num_updates=7600, lr=0.000725476, gnorm=1.074, train_wall=30, wall=2388\n",
            "2020-08-03 21:36:36 | INFO | train_inner | epoch 008:    595 / 1015 loss=5.448, nll_loss=4.001, ppl=16.01, wps=11700.6, ups=3.3, wpb=3549.7, bsz=164.6, num_updates=7700, lr=0.00072075, gnorm=1.074, train_wall=30, wall=2418\n",
            "2020-08-03 21:37:07 | INFO | train_inner | epoch 008:    695 / 1015 loss=5.488, nll_loss=4.045, ppl=16.5, wps=11726.6, ups=3.27, wpb=3585.2, bsz=147.7, num_updates=7800, lr=0.000716115, gnorm=1.091, train_wall=30, wall=2449\n",
            "2020-08-03 21:37:38 | INFO | train_inner | epoch 008:    795 / 1015 loss=5.428, nll_loss=3.977, ppl=15.75, wps=11611.6, ups=3.25, wpb=3576.2, bsz=158, num_updates=7900, lr=0.000711568, gnorm=1.081, train_wall=31, wall=2480\n",
            "2020-08-03 21:38:08 | INFO | train_inner | epoch 008:    895 / 1015 loss=5.504, nll_loss=4.063, ppl=16.71, wps=11533, ups=3.28, wpb=3518, bsz=154.3, num_updates=8000, lr=0.000707107, gnorm=1.142, train_wall=30, wall=2510\n",
            "2020-08-03 21:38:39 | INFO | train_inner | epoch 008:    995 / 1015 loss=5.452, nll_loss=4.004, ppl=16.05, wps=11555.3, ups=3.28, wpb=3523.1, bsz=152.4, num_updates=8100, lr=0.000702728, gnorm=1.097, train_wall=30, wall=2541\n",
            "2020-08-03 21:38:45 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:38:51 | INFO | valid | epoch 008 | valid on 'valid' subset | loss 5.164 | nll_loss 3.588 | ppl 12.02 | wps 26891.2 | wpb 2866.6 | bsz 127.8 | num_updates 8120 | best_loss 5.164\n",
            "2020-08-03 21:38:51 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:38:54 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint8.pt (epoch 8 @ 8120 updates, score 5.164) (writing took 2.6421811989998787 seconds)\n",
            "2020-08-03 21:38:54 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:38:54 | INFO | fairseq_cli.train | end of epoch 8 (average epoch stats below)\n",
            "2020-08-03 21:38:54 | INFO | train | epoch 008 | loss 5.483 | nll_loss 4.04 | ppl 16.45 | wps 11301.3 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 8120 | lr 0.000701862 | gnorm 1.092 | train_wall 309 | wall 2556\n",
            "2020-08-03 21:38:54 | INFO | fairseq_cli.train | begin training epoch 8\n",
            "2020-08-03 21:39:18 | INFO | train_inner | epoch 009:     80 / 1015 loss=5.319, nll_loss=3.852, ppl=14.44, wps=8924.2, ups=2.54, wpb=3516.8, bsz=147, num_updates=8200, lr=0.00069843, gnorm=1.065, train_wall=30, wall=2580\n",
            "2020-08-03 21:39:49 | INFO | train_inner | epoch 009:    180 / 1015 loss=5.334, nll_loss=3.867, ppl=14.59, wps=11584.7, ups=3.27, wpb=3542.4, bsz=162.6, num_updates=8300, lr=0.00069421, gnorm=1.164, train_wall=30, wall=2611\n",
            "2020-08-03 21:40:19 | INFO | train_inner | epoch 009:    280 / 1015 loss=5.324, nll_loss=3.856, ppl=14.48, wps=11601.5, ups=3.29, wpb=3529.4, bsz=153.5, num_updates=8400, lr=0.000690066, gnorm=1.08, train_wall=30, wall=2641\n",
            "2020-08-03 21:40:50 | INFO | train_inner | epoch 009:    380 / 1015 loss=5.226, nll_loss=3.745, ppl=13.4, wps=11778.8, ups=3.26, wpb=3616.9, bsz=173.4, num_updates=8500, lr=0.000685994, gnorm=1.062, train_wall=31, wall=2672\n",
            "2020-08-03 21:41:20 | INFO | train_inner | epoch 009:    480 / 1015 loss=5.299, nll_loss=3.827, ppl=14.19, wps=11535.3, ups=3.27, wpb=3531.9, bsz=163.3, num_updates=8600, lr=0.000681994, gnorm=1.102, train_wall=31, wall=2703\n",
            "2020-08-03 21:41:51 | INFO | train_inner | epoch 009:    580 / 1015 loss=5.303, nll_loss=3.831, ppl=14.23, wps=11676.1, ups=3.29, wpb=3552.8, bsz=150.5, num_updates=8700, lr=0.000678064, gnorm=1.107, train_wall=30, wall=2733\n",
            "2020-08-03 21:42:21 | INFO | train_inner | epoch 009:    680 / 1015 loss=5.249, nll_loss=3.77, ppl=13.65, wps=11738.7, ups=3.28, wpb=3576.8, bsz=162.4, num_updates=8800, lr=0.0006742, gnorm=1.141, train_wall=30, wall=2763\n",
            "2020-08-03 21:42:52 | INFO | train_inner | epoch 009:    780 / 1015 loss=5.267, nll_loss=3.79, ppl=13.84, wps=11610, ups=3.27, wpb=3546.9, bsz=161.6, num_updates=8900, lr=0.000670402, gnorm=1.091, train_wall=30, wall=2794\n",
            "2020-08-03 21:43:23 | INFO | train_inner | epoch 009:    880 / 1015 loss=5.265, nll_loss=3.788, ppl=13.81, wps=11622.2, ups=3.26, wpb=3569.7, bsz=148.8, num_updates=9000, lr=0.000666667, gnorm=1.081, train_wall=31, wall=2825\n",
            "2020-08-03 21:43:53 | INFO | train_inner | epoch 009:    980 / 1015 loss=5.285, nll_loss=3.812, ppl=14.04, wps=11661.5, ups=3.3, wpb=3531.1, bsz=144.8, num_updates=9100, lr=0.000662994, gnorm=1.123, train_wall=30, wall=2855\n",
            "2020-08-03 21:44:04 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:44:10 | INFO | valid | epoch 009 | valid on 'valid' subset | loss 5.01 | nll_loss 3.412 | ppl 10.64 | wps 27229.2 | wpb 2866.6 | bsz 127.8 | num_updates 9135 | best_loss 5.01\n",
            "2020-08-03 21:44:10 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:44:12 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint9.pt (epoch 9 @ 9135 updates, score 5.01) (writing took 2.6210734219998812 seconds)\n",
            "2020-08-03 21:44:12 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:44:12 | INFO | fairseq_cli.train | end of epoch 9 (average epoch stats below)\n",
            "2020-08-03 21:44:12 | INFO | train | epoch 009 | loss 5.278 | nll_loss 3.804 | ppl 13.97 | wps 11301.3 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 9135 | lr 0.000661722 | gnorm 1.102 | train_wall 309 | wall 2875\n",
            "2020-08-03 21:44:12 | INFO | fairseq_cli.train | begin training epoch 9\n",
            "2020-08-03 21:44:32 | INFO | train_inner | epoch 010:     65 / 1015 loss=5.085, nll_loss=3.584, ppl=12, wps=9074.8, ups=2.53, wpb=3589.1, bsz=165.8, num_updates=9200, lr=0.00065938, gnorm=1.053, train_wall=31, wall=2895\n",
            "2020-08-03 21:45:03 | INFO | train_inner | epoch 010:    165 / 1015 loss=5.127, nll_loss=3.629, ppl=12.38, wps=11633.4, ups=3.27, wpb=3558.9, bsz=154.4, num_updates=9300, lr=0.000655826, gnorm=1.095, train_wall=30, wall=2925\n",
            "2020-08-03 21:45:33 | INFO | train_inner | epoch 010:    265 / 1015 loss=5.159, nll_loss=3.666, ppl=12.69, wps=11578, ups=3.29, wpb=3521.4, bsz=153.9, num_updates=9400, lr=0.000652328, gnorm=1.154, train_wall=30, wall=2956\n",
            "2020-08-03 21:46:04 | INFO | train_inner | epoch 010:    365 / 1015 loss=5.145, nll_loss=3.65, ppl=12.56, wps=11534.8, ups=3.27, wpb=3529.1, bsz=151.8, num_updates=9500, lr=0.000648886, gnorm=1.093, train_wall=30, wall=2986\n",
            "2020-08-03 21:46:35 | INFO | train_inner | epoch 010:    465 / 1015 loss=5.108, nll_loss=3.607, ppl=12.19, wps=11698.8, ups=3.24, wpb=3611.7, bsz=161, num_updates=9600, lr=0.000645497, gnorm=1.094, train_wall=31, wall=3017\n",
            "2020-08-03 21:47:05 | INFO | train_inner | epoch 010:    565 / 1015 loss=5.21, nll_loss=3.724, ppl=13.21, wps=11406.3, ups=3.29, wpb=3469.9, bsz=147.4, num_updates=9700, lr=0.000642161, gnorm=1.167, train_wall=30, wall=3047\n",
            "2020-08-03 21:47:36 | INFO | train_inner | epoch 010:    665 / 1015 loss=5.047, nll_loss=3.538, ppl=11.62, wps=11807.6, ups=3.25, wpb=3633.8, bsz=165.6, num_updates=9800, lr=0.000638877, gnorm=1.033, train_wall=31, wall=3078\n",
            "2020-08-03 21:48:07 | INFO | train_inner | epoch 010:    765 / 1015 loss=5.114, nll_loss=3.614, ppl=12.25, wps=11621.4, ups=3.27, wpb=3556, bsz=157.1, num_updates=9900, lr=0.000635642, gnorm=1.124, train_wall=30, wall=3109\n",
            "2020-08-03 21:48:37 | INFO | train_inner | epoch 010:    865 / 1015 loss=5.105, nll_loss=3.605, ppl=12.17, wps=11483.5, ups=3.27, wpb=3511.8, bsz=162.2, num_updates=10000, lr=0.000632456, gnorm=1.138, train_wall=30, wall=3139\n",
            "2020-08-03 21:49:08 | INFO | train_inner | epoch 010:    965 / 1015 loss=5.062, nll_loss=3.556, ppl=11.76, wps=11540.6, ups=3.27, wpb=3534.2, bsz=169.5, num_updates=10100, lr=0.000629317, gnorm=1.097, train_wall=31, wall=3170\n",
            "2020-08-03 21:49:23 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:49:29 | INFO | valid | epoch 010 | valid on 'valid' subset | loss 4.901 | nll_loss 3.291 | ppl 9.79 | wps 26983.4 | wpb 2866.6 | bsz 127.8 | num_updates 10150 | best_loss 4.901\n",
            "2020-08-03 21:49:29 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:49:32 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint10.pt (epoch 10 @ 10150 updates, score 4.901) (writing took 2.6020191959996737 seconds)\n",
            "2020-08-03 21:49:32 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:49:32 | INFO | fairseq_cli.train | end of epoch 10 (average epoch stats below)\n",
            "2020-08-03 21:49:32 | INFO | train | epoch 010 | loss 5.115 | nll_loss 3.617 | ppl 12.27 | wps 11279.5 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 10150 | lr 0.000627765 | gnorm 1.104 | train_wall 310 | wall 3194\n",
            "2020-08-03 21:49:32 | INFO | fairseq_cli.train | begin training epoch 10\n",
            "2020-08-03 21:49:47 | INFO | train_inner | epoch 011:     50 / 1015 loss=5.069, nll_loss=3.563, ppl=11.82, wps=8943.5, ups=2.55, wpb=3512.9, bsz=149.7, num_updates=10200, lr=0.000626224, gnorm=1.086, train_wall=30, wall=3209\n",
            "2020-08-03 21:50:18 | INFO | train_inner | epoch 011:    150 / 1015 loss=5.078, nll_loss=3.572, ppl=11.89, wps=11416.4, ups=3.29, wpb=3472.7, bsz=143.2, num_updates=10300, lr=0.000623177, gnorm=1.179, train_wall=30, wall=3240\n",
            "2020-08-03 21:50:48 | INFO | train_inner | epoch 011:    250 / 1015 loss=4.968, nll_loss=3.448, ppl=10.91, wps=11594.3, ups=3.26, wpb=3554.4, bsz=169.2, num_updates=10400, lr=0.000620174, gnorm=1.131, train_wall=31, wall=3270\n",
            "2020-08-03 21:51:19 | INFO | train_inner | epoch 011:    350 / 1015 loss=4.962, nll_loss=3.441, ppl=10.86, wps=11713, ups=3.26, wpb=3595.7, bsz=166.1, num_updates=10500, lr=0.000617213, gnorm=1.086, train_wall=31, wall=3301\n",
            "2020-08-03 21:51:49 | INFO | train_inner | epoch 011:    450 / 1015 loss=5.034, nll_loss=3.522, ppl=11.48, wps=11487.7, ups=3.28, wpb=3502.9, bsz=157.6, num_updates=10600, lr=0.000614295, gnorm=1.123, train_wall=30, wall=3332\n",
            "2020-08-03 21:52:20 | INFO | train_inner | epoch 011:    550 / 1015 loss=4.893, nll_loss=3.364, ppl=10.29, wps=11782.7, ups=3.27, wpb=3604.8, bsz=179.8, num_updates=10700, lr=0.000611418, gnorm=1.046, train_wall=30, wall=3362\n",
            "2020-08-03 21:52:51 | INFO | train_inner | epoch 011:    650 / 1015 loss=4.961, nll_loss=3.44, ppl=10.86, wps=11826.9, ups=3.27, wpb=3615.6, bsz=161, num_updates=10800, lr=0.000608581, gnorm=1.055, train_wall=30, wall=3393\n",
            "2020-08-03 21:53:21 | INFO | train_inner | epoch 011:    750 / 1015 loss=5.026, nll_loss=3.514, ppl=11.42, wps=11620.4, ups=3.28, wpb=3542.8, bsz=148.9, num_updates=10900, lr=0.000605783, gnorm=1.111, train_wall=30, wall=3423\n",
            "2020-08-03 21:53:52 | INFO | train_inner | epoch 011:    850 / 1015 loss=5.005, nll_loss=3.49, ppl=11.23, wps=11590.3, ups=3.28, wpb=3531.2, bsz=149.9, num_updates=11000, lr=0.000603023, gnorm=1.13, train_wall=30, wall=3454\n",
            "2020-08-03 21:54:22 | INFO | train_inner | epoch 011:    950 / 1015 loss=5.005, nll_loss=3.489, ppl=11.23, wps=11496.7, ups=3.25, wpb=3538.4, bsz=149.3, num_updates=11100, lr=0.0006003, gnorm=1.098, train_wall=31, wall=3484\n",
            "2020-08-03 21:54:42 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 21:54:48 | INFO | valid | epoch 011 | valid on 'valid' subset | loss 4.843 | nll_loss 3.217 | ppl 9.3 | wps 27075.1 | wpb 2866.6 | bsz 127.8 | num_updates 11165 | best_loss 4.843\n",
            "2020-08-03 21:54:48 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 21:54:51 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint11.pt (epoch 11 @ 11165 updates, score 4.843) (writing took 2.5035644889999276 seconds)\n",
            "2020-08-03 21:54:51 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 21:54:51 | INFO | fairseq_cli.train | end of epoch 11 (average epoch stats below)\n",
            "2020-08-03 21:54:51 | INFO | train | epoch 011 | loss 4.991 | nll_loss 3.474 | ppl 11.11 | wps 11295.5 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 11165 | lr 0.00059855 | gnorm 1.105 | train_wall 309 | wall 3513\n",
            "2020-08-03 21:54:51 | INFO | fairseq_cli.train | begin training epoch 11\n",
            "2020-08-03 21:55:02 | INFO | train_inner | epoch 012:     35 / 1015 loss=4.904, nll_loss=3.376, ppl=10.38, wps=9173.6, ups=2.54, wpb=3617.5, bsz=165.1, num_updates=11200, lr=0.000597614, gnorm=1.074, train_wall=31, wall=3524\n",
            "2020-08-03 21:55:32 | INFO | train_inner | epoch 012:    135 / 1015 loss=4.916, nll_loss=3.387, ppl=10.46, wps=11538.2, ups=3.3, wpb=3495.3, bsz=148.7, num_updates=11300, lr=0.000594964, gnorm=1.142, train_wall=30, wall=3554\n",
            "2020-08-03 21:56:03 | INFO | train_inner | epoch 012:    235 / 1015 loss=4.832, nll_loss=3.291, ppl=9.79, wps=11838.4, ups=3.25, wpb=3640.1, bsz=166.7, num_updates=11400, lr=0.000592349, gnorm=1.066, train_wall=31, wall=3585\n",
            "2020-08-03 21:56:33 | INFO | train_inner | epoch 012:    335 / 1015 loss=4.903, nll_loss=3.373, ppl=10.36, wps=11546.9, ups=3.29, wpb=3508, bsz=152.1, num_updates=11500, lr=0.000589768, gnorm=1.124, train_wall=30, wall=3615\n",
            "2020-08-03 21:57:04 | INFO | train_inner | epoch 012:    435 / 1015 loss=4.894, nll_loss=3.363, ppl=10.29, wps=11618.6, ups=3.29, wpb=3531.6, bsz=163.2, num_updates=11600, lr=0.00058722, gnorm=1.131, train_wall=30, wall=3646\n",
            "2020-08-03 21:57:34 | INFO | train_inner | epoch 012:    535 / 1015 loss=4.821, nll_loss=3.279, ppl=9.7, wps=11576, ups=3.27, wpb=3538.1, bsz=170.2, num_updates=11700, lr=0.000584705, gnorm=1.073, train_wall=30, wall=3676\n",
            "2020-08-03 21:58:05 | INFO | train_inner | epoch 012:    635 / 1015 loss=4.937, nll_loss=3.41, ppl=10.63, wps=11635.8, ups=3.26, wpb=3571.4, bsz=149.1, num_updates=11800, lr=0.000582223, gnorm=1.158, train_wall=31, wall=3707\n",
            "2020-08-03 21:58:35 | INFO | train_inner | epoch 012:    735 / 1015 loss=4.857, nll_loss=3.321, ppl=9.99, wps=11686.4, ups=3.28, wpb=3567.4, bsz=167, num_updates=11900, lr=0.000579771, gnorm=1.127, train_wall=30, wall=3737\n",
            "2020-08-03 21:59:06 | INFO | train_inner | epoch 012:    835 / 1015 loss=4.874, nll_loss=3.339, ppl=10.12, wps=11655.6, ups=3.28, wpb=3555.3, bsz=158.9, num_updates=12000, lr=0.00057735, gnorm=1.098, train_wall=30, wall=3768\n",
            "2020-08-03 21:59:36 | INFO | train_inner | epoch 012:    935 / 1015 loss=4.943, nll_loss=3.418, ppl=10.69, wps=11604.7, ups=3.31, wpb=3510.8, bsz=146.6, num_updates=12100, lr=0.00057496, gnorm=1.148, train_wall=30, wall=3798\n",
            "2020-08-03 22:00:01 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:00:07 | INFO | valid | epoch 012 | valid on 'valid' subset | loss 4.772 | nll_loss 3.137 | ppl 8.79 | wps 27132.5 | wpb 2866.6 | bsz 127.8 | num_updates 12180 | best_loss 4.772\n",
            "2020-08-03 22:00:07 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:00:09 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint12.pt (epoch 12 @ 12180 updates, score 4.772) (writing took 2.5923936830004095 seconds)\n",
            "2020-08-03 22:00:09 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:00:09 | INFO | fairseq_cli.train | end of epoch 12 (average epoch stats below)\n",
            "2020-08-03 22:00:09 | INFO | train | epoch 012 | loss 4.885 | nll_loss 3.353 | ppl 10.21 | wps 11316.4 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 12180 | lr 0.000573068 | gnorm 1.115 | train_wall 309 | wall 3831\n",
            "2020-08-03 22:00:09 | INFO | fairseq_cli.train | begin training epoch 12\n",
            "2020-08-03 22:00:15 | INFO | train_inner | epoch 013:     20 / 1015 loss=4.882, nll_loss=3.349, ppl=10.19, wps=8884.5, ups=2.55, wpb=3490, bsz=154.6, num_updates=12200, lr=0.000572598, gnorm=1.124, train_wall=30, wall=3838\n",
            "2020-08-03 22:00:46 | INFO | train_inner | epoch 013:    120 / 1015 loss=4.699, nll_loss=3.14, ppl=8.82, wps=11705.8, ups=3.26, wpb=3593.7, bsz=163.4, num_updates=12300, lr=0.000570266, gnorm=1.055, train_wall=31, wall=3868\n",
            "2020-08-03 22:01:17 | INFO | train_inner | epoch 013:    220 / 1015 loss=4.828, nll_loss=3.285, ppl=9.75, wps=11731.7, ups=3.28, wpb=3581.7, bsz=146.2, num_updates=12400, lr=0.000567962, gnorm=1.103, train_wall=30, wall=3899\n",
            "2020-08-03 22:01:47 | INFO | train_inner | epoch 013:    320 / 1015 loss=4.821, nll_loss=3.277, ppl=9.69, wps=11552.3, ups=3.31, wpb=3492.4, bsz=154.7, num_updates=12500, lr=0.000565685, gnorm=1.152, train_wall=30, wall=3929\n",
            "2020-08-03 22:02:17 | INFO | train_inner | epoch 013:    420 / 1015 loss=4.798, nll_loss=3.251, ppl=9.52, wps=11778.7, ups=3.29, wpb=3579.5, bsz=161.8, num_updates=12600, lr=0.000563436, gnorm=1.107, train_wall=30, wall=3959\n",
            "2020-08-03 22:02:48 | INFO | train_inner | epoch 013:    520 / 1015 loss=4.765, nll_loss=3.215, ppl=9.28, wps=11535.6, ups=3.27, wpb=3527.1, bsz=172.4, num_updates=12700, lr=0.000561214, gnorm=1.114, train_wall=30, wall=3990\n",
            "2020-08-03 22:03:18 | INFO | train_inner | epoch 013:    620 / 1015 loss=4.826, nll_loss=3.284, ppl=9.74, wps=11702.4, ups=3.27, wpb=3577.8, bsz=150.2, num_updates=12800, lr=0.000559017, gnorm=1.12, train_wall=30, wall=4021\n",
            "2020-08-03 22:03:49 | INFO | train_inner | epoch 013:    720 / 1015 loss=4.756, nll_loss=3.205, ppl=9.22, wps=11704.2, ups=3.28, wpb=3573.8, bsz=166.1, num_updates=12900, lr=0.000556846, gnorm=1.095, train_wall=30, wall=4051\n",
            "2020-08-03 22:04:20 | INFO | train_inner | epoch 013:    820 / 1015 loss=4.83, nll_loss=3.289, ppl=9.77, wps=11670.5, ups=3.26, wpb=3576.1, bsz=150.5, num_updates=13000, lr=0.0005547, gnorm=1.104, train_wall=31, wall=4082\n",
            "2020-08-03 22:04:50 | INFO | train_inner | epoch 013:    920 / 1015 loss=4.864, nll_loss=3.326, ppl=10.03, wps=11280.7, ups=3.26, wpb=3458.1, bsz=148.3, num_updates=13100, lr=0.000552579, gnorm=1.185, train_wall=31, wall=4112\n",
            "2020-08-03 22:05:19 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:05:25 | INFO | valid | epoch 013 | valid on 'valid' subset | loss 4.705 | nll_loss 3.062 | ppl 8.35 | wps 27072.1 | wpb 2866.6 | bsz 127.8 | num_updates 13195 | best_loss 4.705\n",
            "2020-08-03 22:05:25 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:05:28 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint13.pt (epoch 13 @ 13195 updates, score 4.705) (writing took 2.54281295400051 seconds)\n",
            "2020-08-03 22:05:28 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:05:28 | INFO | fairseq_cli.train | end of epoch 13 (average epoch stats below)\n",
            "2020-08-03 22:05:28 | INFO | train | epoch 013 | loss 4.795 | nll_loss 3.249 | ppl 9.51 | wps 11309.3 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 13195 | lr 0.000550586 | gnorm 1.115 | train_wall 309 | wall 4150\n",
            "2020-08-03 22:05:28 | INFO | fairseq_cli.train | begin training epoch 13\n",
            "2020-08-03 22:05:30 | INFO | train_inner | epoch 014:      5 / 1015 loss=4.767, nll_loss=3.219, ppl=9.31, wps=9133.8, ups=2.54, wpb=3595.5, bsz=160.9, num_updates=13200, lr=0.000550482, gnorm=1.095, train_wall=31, wall=4152\n",
            "2020-08-03 22:06:00 | INFO | train_inner | epoch 014:    105 / 1015 loss=4.677, nll_loss=3.113, ppl=8.65, wps=11657.9, ups=3.27, wpb=3563.4, bsz=158, num_updates=13300, lr=0.000548408, gnorm=1.077, train_wall=30, wall=4182\n",
            "2020-08-03 22:06:31 | INFO | train_inner | epoch 014:    205 / 1015 loss=4.706, nll_loss=3.147, ppl=8.86, wps=11577.7, ups=3.26, wpb=3554.9, bsz=154.2, num_updates=13400, lr=0.000546358, gnorm=1.112, train_wall=31, wall=4213\n",
            "2020-08-03 22:07:02 | INFO | train_inner | epoch 014:    305 / 1015 loss=4.682, nll_loss=3.118, ppl=8.68, wps=11607.9, ups=3.26, wpb=3556.9, bsz=156.3, num_updates=13500, lr=0.000544331, gnorm=1.107, train_wall=31, wall=4244\n",
            "2020-08-03 22:07:32 | INFO | train_inner | epoch 014:    405 / 1015 loss=4.818, nll_loss=3.273, ppl=9.67, wps=11560.8, ups=3.32, wpb=3478.9, bsz=130.8, num_updates=13600, lr=0.000542326, gnorm=1.143, train_wall=30, wall=4274\n",
            "2020-08-03 22:08:02 | INFO | train_inner | epoch 014:    505 / 1015 loss=4.711, nll_loss=3.152, ppl=8.89, wps=11560.5, ups=3.27, wpb=3538.1, bsz=163.8, num_updates=13700, lr=0.000540343, gnorm=1.11, train_wall=31, wall=4304\n",
            "2020-08-03 22:08:32 | INFO | train_inner | epoch 014:    605 / 1015 loss=4.717, nll_loss=3.161, ppl=8.94, wps=11514.5, ups=3.31, wpb=3476.3, bsz=169, num_updates=13800, lr=0.000538382, gnorm=1.16, train_wall=30, wall=4335\n",
            "2020-08-03 22:09:03 | INFO | train_inner | epoch 014:    705 / 1015 loss=4.702, nll_loss=3.143, ppl=8.83, wps=11751.6, ups=3.26, wpb=3610.2, bsz=163.7, num_updates=13900, lr=0.000536442, gnorm=1.069, train_wall=31, wall=4365\n",
            "2020-08-03 22:09:34 | INFO | train_inner | epoch 014:    805 / 1015 loss=4.73, nll_loss=3.177, ppl=9.04, wps=11684.9, ups=3.28, wpb=3567.8, bsz=161.4, num_updates=14000, lr=0.000534522, gnorm=1.113, train_wall=30, wall=4396\n",
            "2020-08-03 22:10:05 | INFO | train_inner | epoch 014:    905 / 1015 loss=4.726, nll_loss=3.17, ppl=9, wps=11610.6, ups=3.24, wpb=3587.8, bsz=157.9, num_updates=14100, lr=0.000532624, gnorm=1.107, train_wall=31, wall=4427\n",
            "2020-08-03 22:10:35 | INFO | train_inner | epoch 014:   1005 / 1015 loss=4.705, nll_loss=3.148, ppl=8.86, wps=11616.1, ups=3.27, wpb=3557.8, bsz=166.5, num_updates=14200, lr=0.000530745, gnorm=1.103, train_wall=31, wall=4457\n",
            "2020-08-03 22:10:38 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:10:44 | INFO | valid | epoch 014 | valid on 'valid' subset | loss 4.65 | nll_loss 2.997 | ppl 7.98 | wps 27029.2 | wpb 2866.6 | bsz 127.8 | num_updates 14210 | best_loss 4.65\n",
            "2020-08-03 22:10:44 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:10:47 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint14.pt (epoch 14 @ 14210 updates, score 4.65) (writing took 2.6397685240008286 seconds)\n",
            "2020-08-03 22:10:47 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:10:47 | INFO | fairseq_cli.train | end of epoch 14 (average epoch stats below)\n",
            "2020-08-03 22:10:47 | INFO | train | epoch 014 | loss 4.717 | nll_loss 3.16 | ppl 8.94 | wps 11290.2 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 14210 | lr 0.000530558 | gnorm 1.109 | train_wall 309 | wall 4469\n",
            "2020-08-03 22:10:47 | INFO | fairseq_cli.train | begin training epoch 14\n",
            "2020-08-03 22:11:15 | INFO | train_inner | epoch 015:     90 / 1015 loss=4.671, nll_loss=3.106, ppl=8.61, wps=8960.1, ups=2.53, wpb=3540.4, bsz=147.7, num_updates=14300, lr=0.000528886, gnorm=1.135, train_wall=31, wall=4497\n",
            "2020-08-03 22:11:45 | INFO | train_inner | epoch 015:    190 / 1015 loss=4.623, nll_loss=3.052, ppl=8.29, wps=11693.5, ups=3.28, wpb=3560.6, bsz=153.4, num_updates=14400, lr=0.000527046, gnorm=1.095, train_wall=30, wall=4527\n",
            "2020-08-03 22:12:15 | INFO | train_inner | epoch 015:    290 / 1015 loss=4.693, nll_loss=3.131, ppl=8.76, wps=11593.1, ups=3.31, wpb=3498.2, bsz=155, num_updates=14500, lr=0.000525226, gnorm=1.167, train_wall=30, wall=4557\n",
            "2020-08-03 22:12:46 | INFO | train_inner | epoch 015:    390 / 1015 loss=4.673, nll_loss=3.109, ppl=8.63, wps=11702.2, ups=3.26, wpb=3586.1, bsz=152.4, num_updates=14600, lr=0.000523424, gnorm=1.116, train_wall=31, wall=4588\n",
            "2020-08-03 22:13:17 | INFO | train_inner | epoch 015:    490 / 1015 loss=4.618, nll_loss=3.047, ppl=8.27, wps=11737, ups=3.24, wpb=3618.9, bsz=167.7, num_updates=14700, lr=0.000521641, gnorm=1.105, train_wall=31, wall=4619\n",
            "2020-08-03 22:13:48 | INFO | train_inner | epoch 015:    590 / 1015 loss=4.592, nll_loss=3.017, ppl=8.09, wps=11567.8, ups=3.26, wpb=3550.8, bsz=164.5, num_updates=14800, lr=0.000519875, gnorm=1.081, train_wall=31, wall=4650\n",
            "2020-08-03 22:14:18 | INFO | train_inner | epoch 015:    690 / 1015 loss=4.693, nll_loss=3.131, ppl=8.76, wps=11691.3, ups=3.3, wpb=3541.3, bsz=147, num_updates=14900, lr=0.000518128, gnorm=1.146, train_wall=30, wall=4680\n",
            "2020-08-03 22:14:48 | INFO | train_inner | epoch 015:    790 / 1015 loss=4.629, nll_loss=3.061, ppl=8.35, wps=11508.9, ups=3.29, wpb=3498.3, bsz=171.5, num_updates=15000, lr=0.000516398, gnorm=1.126, train_wall=30, wall=4710\n",
            "2020-08-03 22:15:19 | INFO | train_inner | epoch 015:    890 / 1015 loss=4.66, nll_loss=3.095, ppl=8.55, wps=11712.6, ups=3.3, wpb=3551.6, bsz=165.9, num_updates=15100, lr=0.000514685, gnorm=1.149, train_wall=30, wall=4741\n",
            "2020-08-03 22:15:49 | INFO | train_inner | epoch 015:    990 / 1015 loss=4.696, nll_loss=3.137, ppl=8.8, wps=11619.7, ups=3.28, wpb=3539.2, bsz=154.1, num_updates=15200, lr=0.000512989, gnorm=1.135, train_wall=30, wall=4771\n",
            "2020-08-03 22:15:57 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:16:03 | INFO | valid | epoch 015 | valid on 'valid' subset | loss 4.596 | nll_loss 2.939 | ppl 7.67 | wps 27108.4 | wpb 2866.6 | bsz 127.8 | num_updates 15225 | best_loss 4.596\n",
            "2020-08-03 22:16:03 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:16:05 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint15.pt (epoch 15 @ 15225 updates, score 4.596) (writing took 2.5497692670005563 seconds)\n",
            "2020-08-03 22:16:05 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:16:05 | INFO | fairseq_cli.train | end of epoch 15 (average epoch stats below)\n",
            "2020-08-03 22:16:05 | INFO | train | epoch 015 | loss 4.655 | nll_loss 3.089 | ppl 8.51 | wps 11327.1 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 15225 | lr 0.000512568 | gnorm 1.125 | train_wall 308 | wall 4787\n",
            "2020-08-03 22:16:05 | INFO | fairseq_cli.train | begin training epoch 15\n",
            "2020-08-03 22:16:28 | INFO | train_inner | epoch 016:     75 / 1015 loss=4.584, nll_loss=3.009, ppl=8.05, wps=9118, ups=2.56, wpb=3561.8, bsz=156.8, num_updates=15300, lr=0.00051131, gnorm=1.077, train_wall=30, wall=4810\n",
            "2020-08-03 22:16:58 | INFO | train_inner | epoch 016:    175 / 1015 loss=4.656, nll_loss=3.088, ppl=8.5, wps=11511.2, ups=3.3, wpb=3492.7, bsz=135, num_updates=15400, lr=0.000509647, gnorm=1.16, train_wall=30, wall=4840\n",
            "2020-08-03 22:17:29 | INFO | train_inner | epoch 016:    275 / 1015 loss=4.624, nll_loss=3.051, ppl=8.29, wps=11619.2, ups=3.29, wpb=3535.4, bsz=143.2, num_updates=15500, lr=0.000508001, gnorm=1.153, train_wall=30, wall=4871\n",
            "2020-08-03 22:17:59 | INFO | train_inner | epoch 016:    375 / 1015 loss=4.537, nll_loss=2.956, ppl=7.76, wps=11729.3, ups=3.27, wpb=3587.6, bsz=174.4, num_updates=15600, lr=0.00050637, gnorm=1.081, train_wall=30, wall=4902\n",
            "2020-08-03 22:18:30 | INFO | train_inner | epoch 016:    475 / 1015 loss=4.582, nll_loss=3.005, ppl=8.03, wps=11657.5, ups=3.27, wpb=3565.8, bsz=159.6, num_updates=15700, lr=0.000504754, gnorm=1.128, train_wall=30, wall=4932\n",
            "2020-08-03 22:19:00 | INFO | train_inner | epoch 016:    575 / 1015 loss=4.594, nll_loss=3.019, ppl=8.11, wps=11539.5, ups=3.29, wpb=3508.3, bsz=165.2, num_updates=15800, lr=0.000503155, gnorm=1.167, train_wall=30, wall=4963\n",
            "2020-08-03 22:19:31 | INFO | train_inner | epoch 016:    675 / 1015 loss=4.615, nll_loss=3.043, ppl=8.24, wps=11753.8, ups=3.24, wpb=3622.9, bsz=149.3, num_updates=15900, lr=0.00050157, gnorm=1.085, train_wall=31, wall=4993\n",
            "2020-08-03 22:20:02 | INFO | train_inner | epoch 016:    775 / 1015 loss=4.615, nll_loss=3.044, ppl=8.25, wps=11688, ups=3.29, wpb=3558, bsz=163.1, num_updates=16000, lr=0.0005, gnorm=1.158, train_wall=30, wall=5024\n",
            "2020-08-03 22:20:32 | INFO | train_inner | epoch 016:    875 / 1015 loss=4.599, nll_loss=3.026, ppl=8.14, wps=11605.6, ups=3.26, wpb=3560.8, bsz=162.9, num_updates=16100, lr=0.000498445, gnorm=1.117, train_wall=31, wall=5054\n",
            "2020-08-03 22:21:03 | INFO | train_inner | epoch 016:    975 / 1015 loss=4.632, nll_loss=3.064, ppl=8.36, wps=11653.8, ups=3.27, wpb=3560.4, bsz=159.7, num_updates=16200, lr=0.000496904, gnorm=1.12, train_wall=30, wall=5085\n",
            "2020-08-03 22:21:15 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:21:21 | INFO | valid | epoch 016 | valid on 'valid' subset | loss 4.593 | nll_loss 2.931 | ppl 7.62 | wps 27056.9 | wpb 2866.6 | bsz 127.8 | num_updates 16240 | best_loss 4.593\n",
            "2020-08-03 22:21:21 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:21:24 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint16.pt (epoch 16 @ 16240 updates, score 4.593) (writing took 2.626778447000106 seconds)\n",
            "2020-08-03 22:21:24 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:21:24 | INFO | fairseq_cli.train | end of epoch 16 (average epoch stats below)\n",
            "2020-08-03 22:21:24 | INFO | train | epoch 016 | loss 4.599 | nll_loss 3.025 | ppl 8.14 | wps 11312.8 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 16240 | lr 0.000496292 | gnorm 1.127 | train_wall 309 | wall 5106\n",
            "2020-08-03 22:21:24 | INFO | fairseq_cli.train | begin training epoch 16\n",
            "2020-08-03 22:21:42 | INFO | train_inner | epoch 017:     60 / 1015 loss=4.495, nll_loss=2.906, ppl=7.5, wps=8994, ups=2.53, wpb=3551.1, bsz=167, num_updates=16300, lr=0.000495377, gnorm=1.121, train_wall=31, wall=5124\n",
            "2020-08-03 22:22:13 | INFO | train_inner | epoch 017:    160 / 1015 loss=4.514, nll_loss=2.929, ppl=7.61, wps=11509.3, ups=3.27, wpb=3522.4, bsz=168.6, num_updates=16400, lr=0.000493865, gnorm=1.14, train_wall=31, wall=5155\n",
            "2020-08-03 22:22:44 | INFO | train_inner | epoch 017:    260 / 1015 loss=4.484, nll_loss=2.893, ppl=7.43, wps=11625.5, ups=3.27, wpb=3552.6, bsz=167.4, num_updates=16500, lr=0.000492366, gnorm=1.092, train_wall=30, wall=5186\n",
            "2020-08-03 22:23:14 | INFO | train_inner | epoch 017:    360 / 1015 loss=4.554, nll_loss=2.973, ppl=7.85, wps=11582.9, ups=3.27, wpb=3547.2, bsz=157.7, num_updates=16600, lr=0.000490881, gnorm=1.11, train_wall=31, wall=5216\n",
            "2020-08-03 22:23:45 | INFO | train_inner | epoch 017:    460 / 1015 loss=4.537, nll_loss=2.954, ppl=7.75, wps=11473.3, ups=3.29, wpb=3489.3, bsz=159.7, num_updates=16700, lr=0.000489409, gnorm=1.128, train_wall=30, wall=5247\n",
            "2020-08-03 22:24:15 | INFO | train_inner | epoch 017:    560 / 1015 loss=4.574, nll_loss=2.995, ppl=7.97, wps=11751.7, ups=3.29, wpb=3572.9, bsz=151.4, num_updates=16800, lr=0.00048795, gnorm=1.131, train_wall=30, wall=5277\n",
            "2020-08-03 22:24:46 | INFO | train_inner | epoch 017:    660 / 1015 loss=4.564, nll_loss=2.984, ppl=7.91, wps=11665.6, ups=3.27, wpb=3562.6, bsz=153.9, num_updates=16900, lr=0.000486504, gnorm=1.103, train_wall=30, wall=5308\n",
            "2020-08-03 22:25:16 | INFO | train_inner | epoch 017:    760 / 1015 loss=4.528, nll_loss=2.945, ppl=7.7, wps=11700.5, ups=3.28, wpb=3567, bsz=163, num_updates=17000, lr=0.000485071, gnorm=1.127, train_wall=30, wall=5338\n",
            "2020-08-03 22:25:47 | INFO | train_inner | epoch 017:    860 / 1015 loss=4.558, nll_loss=2.979, ppl=7.89, wps=11686.1, ups=3.27, wpb=3577.8, bsz=163, num_updates=17100, lr=0.000483651, gnorm=1.106, train_wall=31, wall=5369\n",
            "2020-08-03 22:26:17 | INFO | train_inner | epoch 017:    960 / 1015 loss=4.616, nll_loss=3.045, ppl=8.25, wps=11644.7, ups=3.28, wpb=3546.5, bsz=146.4, num_updates=17200, lr=0.000482243, gnorm=1.153, train_wall=30, wall=5399\n",
            "2020-08-03 22:26:34 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:26:40 | INFO | valid | epoch 017 | valid on 'valid' subset | loss 4.573 | nll_loss 2.903 | ppl 7.48 | wps 27055 | wpb 2866.6 | bsz 127.8 | num_updates 17255 | best_loss 4.573\n",
            "2020-08-03 22:26:40 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:26:42 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint17.pt (epoch 17 @ 17255 updates, score 4.573) (writing took 2.5939480379993256 seconds)\n",
            "2020-08-03 22:26:42 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:26:42 | INFO | fairseq_cli.train | end of epoch 17 (average epoch stats below)\n",
            "2020-08-03 22:26:42 | INFO | train | epoch 017 | loss 4.547 | nll_loss 2.966 | ppl 7.81 | wps 11304 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 17255 | lr 0.000481474 | gnorm 1.12 | train_wall 309 | wall 5425\n",
            "2020-08-03 22:26:42 | INFO | fairseq_cli.train | begin training epoch 17\n",
            "2020-08-03 22:26:56 | INFO | train_inner | epoch 018:     45 / 1015 loss=4.561, nll_loss=2.981, ppl=7.89, wps=8951.6, ups=2.55, wpb=3514.3, bsz=135, num_updates=17300, lr=0.000480847, gnorm=1.146, train_wall=30, wall=5438\n",
            "2020-08-03 22:27:27 | INFO | train_inner | epoch 018:    145 / 1015 loss=4.488, nll_loss=2.897, ppl=7.45, wps=11832.6, ups=3.28, wpb=3611.7, bsz=147.4, num_updates=17400, lr=0.000479463, gnorm=1.107, train_wall=30, wall=5469\n",
            "2020-08-03 22:27:57 | INFO | train_inner | epoch 018:    245 / 1015 loss=4.528, nll_loss=2.943, ppl=7.69, wps=11654.1, ups=3.29, wpb=3538.1, bsz=144.6, num_updates=17500, lr=0.000478091, gnorm=1.148, train_wall=30, wall=5499\n",
            "2020-08-03 22:28:28 | INFO | train_inner | epoch 018:    345 / 1015 loss=4.489, nll_loss=2.898, ppl=7.46, wps=11549.6, ups=3.25, wpb=3553.9, bsz=154.2, num_updates=17600, lr=0.000476731, gnorm=1.144, train_wall=31, wall=5530\n",
            "2020-08-03 22:28:59 | INFO | train_inner | epoch 018:    445 / 1015 loss=4.463, nll_loss=2.87, ppl=7.31, wps=11811.7, ups=3.27, wpb=3609.6, bsz=167.6, num_updates=17700, lr=0.000475383, gnorm=1.082, train_wall=30, wall=5561\n",
            "2020-08-03 22:29:29 | INFO | train_inner | epoch 018:    545 / 1015 loss=4.472, nll_loss=2.881, ppl=7.37, wps=11577.4, ups=3.29, wpb=3516.2, bsz=171.7, num_updates=17800, lr=0.000474045, gnorm=1.134, train_wall=30, wall=5591\n",
            "2020-08-03 22:29:59 | INFO | train_inner | epoch 018:    645 / 1015 loss=4.438, nll_loss=2.844, ppl=7.18, wps=11618.9, ups=3.27, wpb=3549.8, bsz=183, num_updates=17900, lr=0.000472719, gnorm=1.113, train_wall=30, wall=5622\n",
            "2020-08-03 22:30:30 | INFO | train_inner | epoch 018:    745 / 1015 loss=4.571, nll_loss=2.993, ppl=7.96, wps=11529.2, ups=3.29, wpb=3507.7, bsz=149.9, num_updates=18000, lr=0.000471405, gnorm=1.168, train_wall=30, wall=5652\n",
            "2020-08-03 22:31:00 | INFO | train_inner | epoch 018:    845 / 1015 loss=4.554, nll_loss=2.974, ppl=7.86, wps=11356.2, ups=3.27, wpb=3468.9, bsz=158.2, num_updates=18100, lr=0.0004701, gnorm=1.205, train_wall=30, wall=5683\n",
            "2020-08-03 22:31:31 | INFO | train_inner | epoch 018:    945 / 1015 loss=4.509, nll_loss=2.924, ppl=7.59, wps=11780.3, ups=3.25, wpb=3624, bsz=162.1, num_updates=18200, lr=0.000468807, gnorm=1.096, train_wall=31, wall=5713\n",
            "2020-08-03 22:31:53 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:31:59 | INFO | valid | epoch 018 | valid on 'valid' subset | loss 4.533 | nll_loss 2.867 | ppl 7.3 | wps 27126.8 | wpb 2866.6 | bsz 127.8 | num_updates 18270 | best_loss 4.533\n",
            "2020-08-03 22:31:59 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:32:01 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint18.pt (epoch 18 @ 18270 updates, score 4.533) (writing took 2.6455258280002454 seconds)\n",
            "2020-08-03 22:32:01 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:32:01 | INFO | fairseq_cli.train | end of epoch 18 (average epoch stats below)\n",
            "2020-08-03 22:32:01 | INFO | train | epoch 018 | loss 4.503 | nll_loss 2.916 | ppl 7.55 | wps 11301.6 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 18270 | lr 0.000467908 | gnorm 1.135 | train_wall 309 | wall 5743\n",
            "2020-08-03 22:32:01 | INFO | fairseq_cli.train | begin training epoch 18\n",
            "2020-08-03 22:32:11 | INFO | train_inner | epoch 019:     30 / 1015 loss=4.514, nll_loss=2.928, ppl=7.61, wps=8977.5, ups=2.54, wpb=3533.1, bsz=147.3, num_updates=18300, lr=0.000467525, gnorm=1.138, train_wall=30, wall=5753\n",
            "2020-08-03 22:32:41 | INFO | train_inner | epoch 019:    130 / 1015 loss=4.4, nll_loss=2.798, ppl=6.95, wps=11659.7, ups=3.26, wpb=3573.9, bsz=158.7, num_updates=18400, lr=0.000466252, gnorm=1.105, train_wall=31, wall=5783\n",
            "2020-08-03 22:33:12 | INFO | train_inner | epoch 019:    230 / 1015 loss=4.434, nll_loss=2.837, ppl=7.15, wps=11687.5, ups=3.28, wpb=3558.4, bsz=168.6, num_updates=18500, lr=0.000464991, gnorm=1.129, train_wall=30, wall=5814\n",
            "2020-08-03 22:33:42 | INFO | train_inner | epoch 019:    330 / 1015 loss=4.426, nll_loss=2.828, ppl=7.1, wps=11786, ups=3.27, wpb=3607.2, bsz=165.8, num_updates=18600, lr=0.000463739, gnorm=1.096, train_wall=31, wall=5844\n",
            "2020-08-03 22:34:13 | INFO | train_inner | epoch 019:    430 / 1015 loss=4.425, nll_loss=2.826, ppl=7.09, wps=11520.4, ups=3.3, wpb=3492.9, bsz=160.1, num_updates=18700, lr=0.000462497, gnorm=1.14, train_wall=30, wall=5875\n",
            "2020-08-03 22:34:43 | INFO | train_inner | epoch 019:    530 / 1015 loss=4.448, nll_loss=2.853, ppl=7.22, wps=11664.4, ups=3.27, wpb=3566.2, bsz=164.5, num_updates=18800, lr=0.000461266, gnorm=1.136, train_wall=30, wall=5905\n",
            "2020-08-03 22:35:13 | INFO | train_inner | epoch 019:    630 / 1015 loss=4.503, nll_loss=2.914, ppl=7.54, wps=11579.5, ups=3.31, wpb=3496.1, bsz=145.1, num_updates=18900, lr=0.000460044, gnorm=1.187, train_wall=30, wall=5935\n",
            "2020-08-03 22:35:44 | INFO | train_inner | epoch 019:    730 / 1015 loss=4.501, nll_loss=2.914, ppl=7.54, wps=11578.7, ups=3.3, wpb=3513.1, bsz=150.7, num_updates=19000, lr=0.000458831, gnorm=1.137, train_wall=30, wall=5966\n",
            "2020-08-03 22:36:14 | INFO | train_inner | epoch 019:    830 / 1015 loss=4.418, nll_loss=2.82, ppl=7.06, wps=11853.6, ups=3.27, wpb=3623.3, bsz=183.4, num_updates=19100, lr=0.000457629, gnorm=1.117, train_wall=30, wall=5996\n",
            "2020-08-03 22:36:45 | INFO | train_inner | epoch 019:    930 / 1015 loss=4.51, nll_loss=2.924, ppl=7.59, wps=11764.7, ups=3.29, wpb=3576.5, bsz=151, num_updates=19200, lr=0.000456435, gnorm=1.161, train_wall=30, wall=6027\n",
            "2020-08-03 22:37:10 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:37:16 | INFO | valid | epoch 019 | valid on 'valid' subset | loss 4.497 | nll_loss 2.821 | ppl 7.07 | wps 27151.3 | wpb 2866.6 | bsz 127.8 | num_updates 19285 | best_loss 4.497\n",
            "2020-08-03 22:37:16 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:37:19 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint19.pt (epoch 19 @ 19285 updates, score 4.497) (writing took 2.570419850999315 seconds)\n",
            "2020-08-03 22:37:19 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:37:19 | INFO | fairseq_cli.train | end of epoch 19 (average epoch stats below)\n",
            "2020-08-03 22:37:19 | INFO | train | epoch 019 | loss 4.463 | nll_loss 2.87 | ppl 7.31 | wps 11341.4 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 19285 | lr 0.000455428 | gnorm 1.141 | train_wall 308 | wall 6061\n",
            "2020-08-03 22:37:19 | INFO | fairseq_cli.train | begin training epoch 19\n",
            "2020-08-03 22:37:24 | INFO | train_inner | epoch 020:     15 / 1015 loss=4.574, nll_loss=2.995, ppl=7.97, wps=8959.2, ups=2.56, wpb=3499.1, bsz=134, num_updates=19300, lr=0.000455251, gnorm=1.186, train_wall=30, wall=6066\n",
            "2020-08-03 22:37:54 | INFO | train_inner | epoch 020:    115 / 1015 loss=4.396, nll_loss=2.793, ppl=6.93, wps=11784.2, ups=3.29, wpb=3578.4, bsz=146.6, num_updates=19400, lr=0.000454077, gnorm=1.094, train_wall=30, wall=6096\n",
            "2020-08-03 22:38:25 | INFO | train_inner | epoch 020:    215 / 1015 loss=4.353, nll_loss=2.744, ppl=6.7, wps=11770.6, ups=3.26, wpb=3605.2, bsz=169, num_updates=19500, lr=0.000452911, gnorm=1.124, train_wall=31, wall=6127\n",
            "2020-08-03 22:38:55 | INFO | train_inner | epoch 020:    315 / 1015 loss=4.414, nll_loss=2.813, ppl=7.03, wps=11601.5, ups=3.29, wpb=3524.6, bsz=150.8, num_updates=19600, lr=0.000451754, gnorm=1.133, train_wall=30, wall=6157\n",
            "2020-08-03 22:39:26 | INFO | train_inner | epoch 020:    415 / 1015 loss=4.376, nll_loss=2.771, ppl=6.82, wps=11686.4, ups=3.29, wpb=3554, bsz=176.3, num_updates=19700, lr=0.000450606, gnorm=1.115, train_wall=30, wall=6188\n",
            "2020-08-03 22:39:56 | INFO | train_inner | epoch 020:    515 / 1015 loss=4.426, nll_loss=2.828, ppl=7.1, wps=11734, ups=3.28, wpb=3582.3, bsz=158.2, num_updates=19800, lr=0.000449467, gnorm=1.131, train_wall=30, wall=6218\n",
            "2020-08-03 22:40:26 | INFO | train_inner | epoch 020:    615 / 1015 loss=4.427, nll_loss=2.83, ppl=7.11, wps=11804.5, ups=3.3, wpb=3575.2, bsz=157.8, num_updates=19900, lr=0.000448336, gnorm=1.126, train_wall=30, wall=6248\n",
            "2020-08-03 22:40:57 | INFO | train_inner | epoch 020:    715 / 1015 loss=4.492, nll_loss=2.902, ppl=7.47, wps=11543.6, ups=3.3, wpb=3493.5, bsz=146, num_updates=20000, lr=0.000447214, gnorm=1.19, train_wall=30, wall=6279\n",
            "2020-08-03 22:41:27 | INFO | train_inner | epoch 020:    815 / 1015 loss=4.459, nll_loss=2.865, ppl=7.29, wps=11507.9, ups=3.29, wpb=3494.3, bsz=151.5, num_updates=20100, lr=0.0004461, gnorm=1.155, train_wall=30, wall=6309\n",
            "2020-08-03 22:41:57 | INFO | train_inner | epoch 020:    915 / 1015 loss=4.46, nll_loss=2.868, ppl=7.3, wps=11560.2, ups=3.29, wpb=3508.8, bsz=166.6, num_updates=20200, lr=0.000444994, gnorm=1.191, train_wall=30, wall=6339\n",
            "2020-08-03 22:42:28 | INFO | train_inner | epoch 020:   1015 / 1015 loss=4.468, nll_loss=2.877, ppl=7.34, wps=11811.7, ups=3.29, wpb=3589.8, bsz=155.4, num_updates=20300, lr=0.000443897, gnorm=1.118, train_wall=30, wall=6370\n",
            "2020-08-03 22:42:28 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:42:34 | INFO | valid | epoch 020 | valid on 'valid' subset | loss 4.473 | nll_loss 2.798 | ppl 6.95 | wps 27308.5 | wpb 2866.6 | bsz 127.8 | num_updates 20300 | best_loss 4.473\n",
            "2020-08-03 22:42:34 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:42:36 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint20.pt (epoch 20 @ 20300 updates, score 4.473) (writing took 2.632446665000316 seconds)\n",
            "2020-08-03 22:42:36 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:42:36 | INFO | fairseq_cli.train | end of epoch 20 (average epoch stats below)\n",
            "2020-08-03 22:42:36 | INFO | train | epoch 020 | loss 4.426 | nll_loss 2.828 | ppl 7.1 | wps 11354.4 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 20300 | lr 0.000443897 | gnorm 1.137 | train_wall 308 | wall 6379\n",
            "2020-08-03 22:42:36 | INFO | fairseq_cli.train | begin training epoch 20\n",
            "2020-08-03 22:43:07 | INFO | train_inner | epoch 021:    100 / 1015 loss=4.379, nll_loss=2.774, ppl=6.84, wps=9006.7, ups=2.55, wpb=3538.8, bsz=149.8, num_updates=20400, lr=0.000442807, gnorm=1.171, train_wall=30, wall=6409\n",
            "2020-08-03 22:43:37 | INFO | train_inner | epoch 021:    200 / 1015 loss=4.374, nll_loss=2.768, ppl=6.81, wps=11760.4, ups=3.29, wpb=3576.4, bsz=154.2, num_updates=20500, lr=0.000441726, gnorm=1.128, train_wall=30, wall=6440\n",
            "2020-08-03 22:44:08 | INFO | train_inner | epoch 021:    300 / 1015 loss=4.315, nll_loss=2.701, ppl=6.5, wps=11750.7, ups=3.28, wpb=3577.1, bsz=168.1, num_updates=20600, lr=0.000440653, gnorm=1.126, train_wall=30, wall=6470\n",
            "2020-08-03 22:44:38 | INFO | train_inner | epoch 021:    400 / 1015 loss=4.397, nll_loss=2.792, ppl=6.93, wps=11554.6, ups=3.31, wpb=3496.1, bsz=154.4, num_updates=20700, lr=0.000439587, gnorm=1.18, train_wall=30, wall=6500\n",
            "2020-08-03 22:45:08 | INFO | train_inner | epoch 021:    500 / 1015 loss=4.338, nll_loss=2.729, ppl=6.63, wps=11804.7, ups=3.29, wpb=3584.7, bsz=179.9, num_updates=20800, lr=0.000438529, gnorm=1.131, train_wall=30, wall=6531\n",
            "2020-08-03 22:45:39 | INFO | train_inner | epoch 021:    600 / 1015 loss=4.404, nll_loss=2.802, ppl=6.98, wps=11752.4, ups=3.31, wpb=3547.5, bsz=156.5, num_updates=20900, lr=0.000437479, gnorm=1.15, train_wall=30, wall=6561\n",
            "2020-08-03 22:46:09 | INFO | train_inner | epoch 021:    700 / 1015 loss=4.407, nll_loss=2.807, ppl=7, wps=11778.2, ups=3.27, wpb=3603.6, bsz=167.5, num_updates=21000, lr=0.000436436, gnorm=1.125, train_wall=30, wall=6591\n",
            "2020-08-03 22:46:39 | INFO | train_inner | epoch 021:    800 / 1015 loss=4.432, nll_loss=2.834, ppl=7.13, wps=11714.4, ups=3.32, wpb=3525.8, bsz=152.1, num_updates=21100, lr=0.0004354, gnorm=1.163, train_wall=30, wall=6621\n",
            "2020-08-03 22:47:10 | INFO | train_inner | epoch 021:    900 / 1015 loss=4.434, nll_loss=2.836, ppl=7.14, wps=11481.3, ups=3.31, wpb=3465.4, bsz=150.4, num_updates=21200, lr=0.000434372, gnorm=1.162, train_wall=30, wall=6652\n",
            "2020-08-03 22:47:40 | INFO | train_inner | epoch 021:   1000 / 1015 loss=4.454, nll_loss=2.861, ppl=7.26, wps=11793.2, ups=3.29, wpb=3588.8, bsz=151.7, num_updates=21300, lr=0.000433351, gnorm=1.131, train_wall=30, wall=6682\n",
            "2020-08-03 22:47:45 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:47:51 | INFO | valid | epoch 021 | valid on 'valid' subset | loss 4.459 | nll_loss 2.779 | ppl 6.86 | wps 27143.5 | wpb 2866.6 | bsz 127.8 | num_updates 21315 | best_loss 4.459\n",
            "2020-08-03 22:47:51 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:47:53 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint21.pt (epoch 21 @ 21315 updates, score 4.459) (writing took 2.5647365620006894 seconds)\n",
            "2020-08-03 22:47:53 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:47:53 | INFO | fairseq_cli.train | end of epoch 21 (average epoch stats below)\n",
            "2020-08-03 22:47:53 | INFO | train | epoch 021 | loss 4.395 | nll_loss 2.792 | ppl 6.93 | wps 11375.1 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 21315 | lr 0.000433199 | gnorm 1.148 | train_wall 307 | wall 6695\n",
            "2020-08-03 22:47:53 | INFO | fairseq_cli.train | begin training epoch 21\n",
            "2020-08-03 22:48:19 | INFO | train_inner | epoch 022:     85 / 1015 loss=4.359, nll_loss=2.75, ppl=6.73, wps=9093, ups=2.54, wpb=3573.4, bsz=141.2, num_updates=21400, lr=0.000432338, gnorm=1.119, train_wall=30, wall=6721\n",
            "2020-08-03 22:48:50 | INFO | train_inner | epoch 022:    185 / 1015 loss=4.286, nll_loss=2.668, ppl=6.35, wps=11652.3, ups=3.3, wpb=3528.8, bsz=169.9, num_updates=21500, lr=0.000431331, gnorm=1.115, train_wall=30, wall=6752\n",
            "2020-08-03 22:49:20 | INFO | train_inner | epoch 022:    285 / 1015 loss=4.287, nll_loss=2.67, ppl=6.36, wps=11693.7, ups=3.31, wpb=3537.4, bsz=169.1, num_updates=21600, lr=0.000430331, gnorm=1.136, train_wall=30, wall=6782\n",
            "2020-08-03 22:49:50 | INFO | train_inner | epoch 022:    385 / 1015 loss=4.335, nll_loss=2.724, ppl=6.61, wps=11764.8, ups=3.27, wpb=3601.3, bsz=162.9, num_updates=21700, lr=0.000429339, gnorm=1.116, train_wall=31, wall=6813\n",
            "2020-08-03 22:50:21 | INFO | train_inner | epoch 022:    485 / 1015 loss=4.447, nll_loss=2.85, ppl=7.21, wps=11682.9, ups=3.3, wpb=3538.2, bsz=138.1, num_updates=21800, lr=0.000428353, gnorm=1.215, train_wall=30, wall=6843\n",
            "2020-08-03 22:50:51 | INFO | train_inner | epoch 022:    585 / 1015 loss=4.343, nll_loss=2.734, ppl=6.65, wps=11674.9, ups=3.29, wpb=3543.3, bsz=169.8, num_updates=21900, lr=0.000427374, gnorm=1.167, train_wall=30, wall=6873\n",
            "2020-08-03 22:51:21 | INFO | train_inner | epoch 022:    685 / 1015 loss=4.36, nll_loss=2.754, ppl=6.75, wps=11764.1, ups=3.29, wpb=3574.7, bsz=161.4, num_updates=22000, lr=0.000426401, gnorm=1.156, train_wall=30, wall=6904\n",
            "2020-08-03 22:51:52 | INFO | train_inner | epoch 022:    785 / 1015 loss=4.421, nll_loss=2.822, ppl=7.07, wps=11628.2, ups=3.3, wpb=3525.4, bsz=143.3, num_updates=22100, lr=0.000425436, gnorm=1.191, train_wall=30, wall=6934\n",
            "2020-08-03 22:52:22 | INFO | train_inner | epoch 022:    885 / 1015 loss=4.416, nll_loss=2.816, ppl=7.04, wps=11529.2, ups=3.26, wpb=3539.6, bsz=148.7, num_updates=22200, lr=0.000424476, gnorm=1.148, train_wall=31, wall=6965\n",
            "2020-08-03 22:52:53 | INFO | train_inner | epoch 022:    985 / 1015 loss=4.393, nll_loss=2.792, ppl=6.92, wps=11691.5, ups=3.3, wpb=3542.7, bsz=162.6, num_updates=22300, lr=0.000423524, gnorm=1.155, train_wall=30, wall=6995\n",
            "2020-08-03 22:53:02 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:53:08 | INFO | valid | epoch 022 | valid on 'valid' subset | loss 4.448 | nll_loss 2.765 | ppl 6.8 | wps 27190.7 | wpb 2866.6 | bsz 127.8 | num_updates 22330 | best_loss 4.448\n",
            "2020-08-03 22:53:08 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:53:10 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint22.pt (epoch 22 @ 22330 updates, score 4.448) (writing took 2.574693512999147 seconds)\n",
            "2020-08-03 22:53:10 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:53:10 | INFO | fairseq_cli.train | end of epoch 22 (average epoch stats below)\n",
            "2020-08-03 22:53:10 | INFO | train | epoch 022 | loss 4.362 | nll_loss 2.755 | ppl 6.75 | wps 11359.2 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 22330 | lr 0.000423239 | gnorm 1.151 | train_wall 307 | wall 7013\n",
            "2020-08-03 22:53:10 | INFO | fairseq_cli.train | begin training epoch 22\n",
            "2020-08-03 22:53:32 | INFO | train_inner | epoch 023:     70 / 1015 loss=4.342, nll_loss=2.731, ppl=6.64, wps=9006, ups=2.56, wpb=3522.3, bsz=150.6, num_updates=22400, lr=0.000422577, gnorm=1.17, train_wall=30, wall=7034\n",
            "2020-08-03 22:54:03 | INFO | train_inner | epoch 023:    170 / 1015 loss=4.27, nll_loss=2.649, ppl=6.27, wps=11665, ups=3.25, wpb=3587.2, bsz=162.2, num_updates=22500, lr=0.000421637, gnorm=1.132, train_wall=31, wall=7065\n",
            "2020-08-03 22:54:33 | INFO | train_inner | epoch 023:    270 / 1015 loss=4.312, nll_loss=2.697, ppl=6.49, wps=11717, ups=3.28, wpb=3573.3, bsz=162.2, num_updates=22600, lr=0.000420703, gnorm=1.152, train_wall=30, wall=7095\n",
            "2020-08-03 22:55:03 | INFO | train_inner | epoch 023:    370 / 1015 loss=4.364, nll_loss=2.757, ppl=6.76, wps=11758.9, ups=3.31, wpb=3554.6, bsz=148.3, num_updates=22700, lr=0.000419775, gnorm=1.197, train_wall=30, wall=7125\n",
            "2020-08-03 22:55:34 | INFO | train_inner | epoch 023:    470 / 1015 loss=4.346, nll_loss=2.737, ppl=6.67, wps=11915.3, ups=3.27, wpb=3649.1, bsz=152.6, num_updates=22800, lr=0.000418854, gnorm=1.118, train_wall=31, wall=7156\n",
            "2020-08-03 22:56:04 | INFO | train_inner | epoch 023:    570 / 1015 loss=4.366, nll_loss=2.759, ppl=6.77, wps=11671.6, ups=3.31, wpb=3527, bsz=149.6, num_updates=22900, lr=0.000417938, gnorm=1.16, train_wall=30, wall=7186\n",
            "2020-08-03 22:56:34 | INFO | train_inner | epoch 023:    670 / 1015 loss=4.321, nll_loss=2.709, ppl=6.54, wps=11639.2, ups=3.31, wpb=3519, bsz=169.1, num_updates=23000, lr=0.000417029, gnorm=1.135, train_wall=30, wall=7217\n",
            "2020-08-03 22:57:05 | INFO | train_inner | epoch 023:    770 / 1015 loss=4.393, nll_loss=2.789, ppl=6.91, wps=11548, ups=3.3, wpb=3499.4, bsz=145.7, num_updates=23100, lr=0.000416125, gnorm=1.192, train_wall=30, wall=7247\n",
            "2020-08-03 22:57:35 | INFO | train_inner | epoch 023:    870 / 1015 loss=4.325, nll_loss=2.714, ppl=6.56, wps=11664.8, ups=3.28, wpb=3561.7, bsz=168.2, num_updates=23200, lr=0.000415227, gnorm=1.136, train_wall=30, wall=7277\n",
            "2020-08-03 22:58:05 | INFO | train_inner | epoch 023:    970 / 1015 loss=4.323, nll_loss=2.713, ppl=6.56, wps=11644.4, ups=3.31, wpb=3521.1, bsz=173.7, num_updates=23300, lr=0.000414335, gnorm=1.147, train_wall=30, wall=7308\n",
            "2020-08-03 22:58:19 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 22:58:25 | INFO | valid | epoch 023 | valid on 'valid' subset | loss 4.431 | nll_loss 2.757 | ppl 6.76 | wps 27205.4 | wpb 2866.6 | bsz 127.8 | num_updates 23345 | best_loss 4.431\n",
            "2020-08-03 22:58:25 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 22:58:28 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint23.pt (epoch 23 @ 23345 updates, score 4.431) (writing took 2.5361906300004193 seconds)\n",
            "2020-08-03 22:58:28 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 22:58:28 | INFO | fairseq_cli.train | end of epoch 23 (average epoch stats below)\n",
            "2020-08-03 22:58:28 | INFO | train | epoch 023 | loss 4.336 | nll_loss 2.725 | ppl 6.61 | wps 11355 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 23345 | lr 0.000413936 | gnorm 1.154 | train_wall 308 | wall 7330\n",
            "2020-08-03 22:58:28 | INFO | fairseq_cli.train | begin training epoch 23\n",
            "2020-08-03 22:58:45 | INFO | train_inner | epoch 024:     55 / 1015 loss=4.316, nll_loss=2.703, ppl=6.51, wps=8958.7, ups=2.55, wpb=3507.3, bsz=157.6, num_updates=23400, lr=0.000413449, gnorm=1.171, train_wall=30, wall=7347\n",
            "2020-08-03 22:59:15 | INFO | train_inner | epoch 024:    155 / 1015 loss=4.224, nll_loss=2.598, ppl=6.05, wps=11677.1, ups=3.29, wpb=3550.5, bsz=169.6, num_updates=23500, lr=0.000412568, gnorm=1.12, train_wall=30, wall=7377\n",
            "2020-08-03 22:59:45 | INFO | train_inner | epoch 024:    255 / 1015 loss=4.329, nll_loss=2.716, ppl=6.57, wps=11563.1, ups=3.32, wpb=3484, bsz=148.9, num_updates=23600, lr=0.000411693, gnorm=1.19, train_wall=30, wall=7407\n",
            "2020-08-03 23:00:15 | INFO | train_inner | epoch 024:    355 / 1015 loss=4.314, nll_loss=2.699, ppl=6.49, wps=11788.9, ups=3.3, wpb=3572.3, bsz=151.7, num_updates=23700, lr=0.000410824, gnorm=1.158, train_wall=30, wall=7438\n",
            "2020-08-03 23:00:46 | INFO | train_inner | epoch 024:    455 / 1015 loss=4.278, nll_loss=2.66, ppl=6.32, wps=11825.9, ups=3.29, wpb=3594.2, bsz=162.8, num_updates=23800, lr=0.00040996, gnorm=1.125, train_wall=30, wall=7468\n",
            "2020-08-03 23:01:16 | INFO | train_inner | epoch 024:    555 / 1015 loss=4.358, nll_loss=2.749, ppl=6.72, wps=11913.6, ups=3.31, wpb=3596.3, bsz=142, num_updates=23900, lr=0.000409101, gnorm=1.149, train_wall=30, wall=7498\n",
            "2020-08-03 23:01:47 | INFO | train_inner | epoch 024:    655 / 1015 loss=4.257, nll_loss=2.635, ppl=6.21, wps=11565.3, ups=3.26, wpb=3544.1, bsz=167.6, num_updates=24000, lr=0.000408248, gnorm=1.139, train_wall=31, wall=7529\n",
            "2020-08-03 23:02:17 | INFO | train_inner | epoch 024:    755 / 1015 loss=4.369, nll_loss=2.763, ppl=6.79, wps=11576.8, ups=3.3, wpb=3509.8, bsz=152, num_updates=24100, lr=0.0004074, gnorm=1.231, train_wall=30, wall=7559\n",
            "2020-08-03 23:02:47 | INFO | train_inner | epoch 024:    855 / 1015 loss=4.359, nll_loss=2.753, ppl=6.74, wps=11719.3, ups=3.3, wpb=3546.2, bsz=161.9, num_updates=24200, lr=0.000406558, gnorm=1.149, train_wall=30, wall=7589\n",
            "2020-08-03 23:03:18 | INFO | train_inner | epoch 024:    955 / 1015 loss=4.296, nll_loss=2.681, ppl=6.42, wps=11743.1, ups=3.29, wpb=3571.9, bsz=171.6, num_updates=24300, lr=0.00040572, gnorm=1.167, train_wall=30, wall=7620\n",
            "2020-08-03 23:03:36 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:03:42 | INFO | valid | epoch 024 | valid on 'valid' subset | loss 4.425 | nll_loss 2.745 | ppl 6.71 | wps 27219.4 | wpb 2866.6 | bsz 127.8 | num_updates 24360 | best_loss 4.425\n",
            "2020-08-03 23:03:42 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:03:45 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint24.pt (epoch 24 @ 24360 updates, score 4.425) (writing took 2.5952500170005806 seconds)\n",
            "2020-08-03 23:03:45 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:03:45 | INFO | fairseq_cli.train | end of epoch 24 (average epoch stats below)\n",
            "2020-08-03 23:03:45 | INFO | train | epoch 024 | loss 4.311 | nll_loss 2.697 | ppl 6.48 | wps 11372.2 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 24360 | lr 0.00040522 | gnorm 1.16 | train_wall 307 | wall 7647\n",
            "2020-08-03 23:03:45 | INFO | fairseq_cli.train | begin training epoch 24\n",
            "2020-08-03 23:03:57 | INFO | train_inner | epoch 025:     40 / 1015 loss=4.302, nll_loss=2.687, ppl=6.44, wps=9100.9, ups=2.56, wpb=3559.7, bsz=152, num_updates=24400, lr=0.000404888, gnorm=1.155, train_wall=30, wall=7659\n",
            "2020-08-03 23:04:27 | INFO | train_inner | epoch 025:    140 / 1015 loss=4.202, nll_loss=2.573, ppl=5.95, wps=11659.3, ups=3.26, wpb=3571.3, bsz=169.1, num_updates=24500, lr=0.000404061, gnorm=1.116, train_wall=31, wall=7690\n",
            "2020-08-03 23:04:58 | INFO | train_inner | epoch 025:    240 / 1015 loss=4.264, nll_loss=2.642, ppl=6.24, wps=11684.5, ups=3.3, wpb=3537.3, bsz=157.2, num_updates=24600, lr=0.000403239, gnorm=1.155, train_wall=30, wall=7720\n",
            "2020-08-03 23:05:28 | INFO | train_inner | epoch 025:    340 / 1015 loss=4.247, nll_loss=2.624, ppl=6.16, wps=11818.3, ups=3.29, wpb=3594.6, bsz=167.8, num_updates=24700, lr=0.000402422, gnorm=1.117, train_wall=30, wall=7750\n",
            "2020-08-03 23:05:58 | INFO | train_inner | epoch 025:    440 / 1015 loss=4.33, nll_loss=2.718, ppl=6.58, wps=11739.8, ups=3.32, wpb=3537.3, bsz=147.3, num_updates=24800, lr=0.00040161, gnorm=1.175, train_wall=30, wall=7780\n",
            "2020-08-03 23:06:29 | INFO | train_inner | epoch 025:    540 / 1015 loss=4.286, nll_loss=2.668, ppl=6.36, wps=11813.8, ups=3.29, wpb=3595.5, bsz=156.6, num_updates=24900, lr=0.000400802, gnorm=1.146, train_wall=30, wall=7811\n",
            "2020-08-03 23:06:59 | INFO | train_inner | epoch 025:    640 / 1015 loss=4.33, nll_loss=2.718, ppl=6.58, wps=11817.2, ups=3.28, wpb=3602.9, bsz=144.8, num_updates=25000, lr=0.0004, gnorm=1.155, train_wall=30, wall=7841\n",
            "2020-08-03 23:07:29 | INFO | train_inner | epoch 025:    740 / 1015 loss=4.297, nll_loss=2.682, ppl=6.42, wps=11586.7, ups=3.3, wpb=3507.7, bsz=165.5, num_updates=25100, lr=0.000399202, gnorm=1.195, train_wall=30, wall=7872\n",
            "2020-08-03 23:08:00 | INFO | train_inner | epoch 025:    840 / 1015 loss=4.255, nll_loss=2.635, ppl=6.21, wps=11725, ups=3.28, wpb=3579.6, bsz=173.1, num_updates=25200, lr=0.00039841, gnorm=1.149, train_wall=30, wall=7902\n",
            "2020-08-03 23:08:30 | INFO | train_inner | epoch 025:    940 / 1015 loss=4.338, nll_loss=2.728, ppl=6.62, wps=11568.4, ups=3.32, wpb=3484.5, bsz=152.5, num_updates=25300, lr=0.000397621, gnorm=1.232, train_wall=30, wall=7932\n",
            "2020-08-03 23:08:53 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:08:59 | INFO | valid | epoch 025 | valid on 'valid' subset | loss 4.441 | nll_loss 2.748 | ppl 6.72 | wps 27083.1 | wpb 2866.6 | bsz 127.8 | num_updates 25375 | best_loss 4.425\n",
            "2020-08-03 23:08:59 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:09:00 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint25.pt (epoch 25 @ 25375 updates, score 4.441) (writing took 1.513985292000143 seconds)\n",
            "2020-08-03 23:09:00 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:09:00 | INFO | fairseq_cli.train | end of epoch 25 (average epoch stats below)\n",
            "2020-08-03 23:09:00 | INFO | train | epoch 025 | loss 4.286 | nll_loss 2.669 | ppl 6.36 | wps 11416.6 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 25375 | lr 0.000397033 | gnorm 1.165 | train_wall 307 | wall 7962\n",
            "2020-08-03 23:09:00 | INFO | fairseq_cli.train | begin training epoch 25\n",
            "2020-08-03 23:09:08 | INFO | train_inner | epoch 026:     25 / 1015 loss=4.315, nll_loss=2.702, ppl=6.51, wps=9168.9, ups=2.64, wpb=3472, bsz=146.6, num_updates=25400, lr=0.000396838, gnorm=1.203, train_wall=30, wall=7970\n",
            "2020-08-03 23:09:38 | INFO | train_inner | epoch 026:    125 / 1015 loss=4.172, nll_loss=2.539, ppl=5.81, wps=11737.6, ups=3.29, wpb=3571.8, bsz=170.8, num_updates=25500, lr=0.000396059, gnorm=1.125, train_wall=30, wall=8001\n",
            "2020-08-03 23:10:09 | INFO | train_inner | epoch 026:    225 / 1015 loss=4.242, nll_loss=2.617, ppl=6.13, wps=11759.2, ups=3.28, wpb=3590.2, bsz=149.4, num_updates=25600, lr=0.000395285, gnorm=1.153, train_wall=30, wall=8031\n",
            "2020-08-03 23:10:39 | INFO | train_inner | epoch 026:    325 / 1015 loss=4.286, nll_loss=2.667, ppl=6.35, wps=11714.7, ups=3.32, wpb=3526.9, bsz=145.3, num_updates=25700, lr=0.000394515, gnorm=1.197, train_wall=30, wall=8061\n",
            "2020-08-03 23:11:09 | INFO | train_inner | epoch 026:    425 / 1015 loss=4.27, nll_loss=2.649, ppl=6.27, wps=11610.6, ups=3.29, wpb=3534.3, bsz=153.2, num_updates=25800, lr=0.00039375, gnorm=1.177, train_wall=30, wall=8092\n",
            "2020-08-03 23:11:40 | INFO | train_inner | epoch 026:    525 / 1015 loss=4.269, nll_loss=2.648, ppl=6.27, wps=11749.6, ups=3.28, wpb=3585.1, bsz=154.3, num_updates=25900, lr=0.000392989, gnorm=1.165, train_wall=30, wall=8122\n",
            "2020-08-03 23:12:10 | INFO | train_inner | epoch 026:    625 / 1015 loss=4.345, nll_loss=2.735, ppl=6.66, wps=11687.1, ups=3.3, wpb=3541.4, bsz=138.6, num_updates=26000, lr=0.000392232, gnorm=1.225, train_wall=30, wall=8152\n",
            "2020-08-03 23:12:41 | INFO | train_inner | epoch 026:    725 / 1015 loss=4.28, nll_loss=2.662, ppl=6.33, wps=11369.3, ups=3.3, wpb=3446.4, bsz=161.8, num_updates=26100, lr=0.00039148, gnorm=1.204, train_wall=30, wall=8183\n",
            "2020-08-03 23:13:11 | INFO | train_inner | epoch 026:    825 / 1015 loss=4.253, nll_loss=2.633, ppl=6.2, wps=11849.4, ups=3.26, wpb=3638.9, bsz=173.8, num_updates=26200, lr=0.000390732, gnorm=1.16, train_wall=31, wall=8213\n",
            "2020-08-03 23:13:42 | INFO | train_inner | epoch 026:    925 / 1015 loss=4.265, nll_loss=2.647, ppl=6.26, wps=11804, ups=3.27, wpb=3607.9, bsz=167.7, num_updates=26300, lr=0.000389989, gnorm=1.13, train_wall=30, wall=8244\n",
            "2020-08-03 23:14:09 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:14:15 | INFO | valid | epoch 026 | valid on 'valid' subset | loss 4.417 | nll_loss 2.729 | ppl 6.63 | wps 27161.5 | wpb 2866.6 | bsz 127.8 | num_updates 26390 | best_loss 4.417\n",
            "2020-08-03 23:14:15 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:14:18 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint26.pt (epoch 26 @ 26390 updates, score 4.417) (writing took 2.5729291530005867 seconds)\n",
            "2020-08-03 23:14:18 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:14:18 | INFO | fairseq_cli.train | end of epoch 26 (average epoch stats below)\n",
            "2020-08-03 23:14:18 | INFO | train | epoch 026 | loss 4.266 | nll_loss 2.646 | ppl 6.26 | wps 11351.3 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 26390 | lr 0.000389323 | gnorm 1.172 | train_wall 308 | wall 8280\n",
            "2020-08-03 23:14:18 | INFO | fairseq_cli.train | begin training epoch 26\n",
            "2020-08-03 23:14:21 | INFO | train_inner | epoch 027:     10 / 1015 loss=4.286, nll_loss=2.67, ppl=6.37, wps=8930.4, ups=2.56, wpb=3481.7, bsz=164.6, num_updates=26400, lr=0.000389249, gnorm=1.198, train_wall=30, wall=8283\n",
            "2020-08-03 23:14:51 | INFO | train_inner | epoch 027:    110 / 1015 loss=4.172, nll_loss=2.54, ppl=5.81, wps=11810.3, ups=3.3, wpb=3577.4, bsz=169.1, num_updates=26500, lr=0.000388514, gnorm=1.15, train_wall=30, wall=8313\n",
            "2020-08-03 23:15:22 | INFO | train_inner | epoch 027:    210 / 1015 loss=4.23, nll_loss=2.603, ppl=6.07, wps=11730.7, ups=3.29, wpb=3565.8, bsz=153.4, num_updates=26600, lr=0.000387783, gnorm=1.168, train_wall=30, wall=8344\n",
            "2020-08-03 23:15:52 | INFO | train_inner | epoch 027:    310 / 1015 loss=4.215, nll_loss=2.587, ppl=6.01, wps=11958.7, ups=3.29, wpb=3630.3, bsz=163.9, num_updates=26700, lr=0.000387056, gnorm=1.14, train_wall=30, wall=8374\n",
            "2020-08-03 23:16:22 | INFO | train_inner | epoch 027:    410 / 1015 loss=4.214, nll_loss=2.587, ppl=6.01, wps=11546.7, ups=3.29, wpb=3504.8, bsz=169, num_updates=26800, lr=0.000386334, gnorm=1.204, train_wall=30, wall=8404\n",
            "2020-08-03 23:16:53 | INFO | train_inner | epoch 027:    510 / 1015 loss=4.225, nll_loss=2.6, ppl=6.06, wps=11711.8, ups=3.29, wpb=3559.8, bsz=160.2, num_updates=26900, lr=0.000385615, gnorm=1.16, train_wall=30, wall=8435\n",
            "2020-08-03 23:17:23 | INFO | train_inner | epoch 027:    610 / 1015 loss=4.273, nll_loss=2.654, ppl=6.29, wps=11671.3, ups=3.29, wpb=3551.2, bsz=154.7, num_updates=27000, lr=0.0003849, gnorm=1.188, train_wall=30, wall=8465\n",
            "2020-08-03 23:17:54 | INFO | train_inner | epoch 027:    710 / 1015 loss=4.258, nll_loss=2.637, ppl=6.22, wps=11675.2, ups=3.28, wpb=3564.9, bsz=154.9, num_updates=27100, lr=0.000384189, gnorm=1.171, train_wall=30, wall=8496\n",
            "2020-08-03 23:18:24 | INFO | train_inner | epoch 027:    810 / 1015 loss=4.28, nll_loss=2.662, ppl=6.33, wps=11715.8, ups=3.32, wpb=3531.1, bsz=151, num_updates=27200, lr=0.000383482, gnorm=1.189, train_wall=30, wall=8526\n",
            "2020-08-03 23:18:54 | INFO | train_inner | epoch 027:    910 / 1015 loss=4.281, nll_loss=2.664, ppl=6.34, wps=11588.8, ups=3.3, wpb=3506.6, bsz=157.6, num_updates=27300, lr=0.00038278, gnorm=1.234, train_wall=30, wall=8556\n",
            "2020-08-03 23:19:24 | INFO | train_inner | epoch 027:   1010 / 1015 loss=4.331, nll_loss=2.72, ppl=6.59, wps=11530.4, ups=3.31, wpb=3485.8, bsz=140.3, num_updates=27400, lr=0.00038208, gnorm=1.198, train_wall=30, wall=8586\n",
            "2020-08-03 23:19:26 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:19:32 | INFO | valid | epoch 027 | valid on 'valid' subset | loss 4.379 | nll_loss 2.697 | ppl 6.48 | wps 27240 | wpb 2866.6 | bsz 127.8 | num_updates 27405 | best_loss 4.379\n",
            "2020-08-03 23:19:32 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:19:34 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint27.pt (epoch 27 @ 27405 updates, score 4.379) (writing took 2.550953419000507 seconds)\n",
            "2020-08-03 23:19:34 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:19:34 | INFO | fairseq_cli.train | end of epoch 27 (average epoch stats below)\n",
            "2020-08-03 23:19:34 | INFO | train | epoch 027 | loss 4.246 | nll_loss 2.623 | ppl 6.16 | wps 11374.9 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 27405 | lr 0.000382046 | gnorm 1.18 | train_wall 307 | wall 8597\n",
            "2020-08-03 23:19:34 | INFO | fairseq_cli.train | begin training epoch 27\n",
            "2020-08-03 23:20:04 | INFO | train_inner | epoch 028:     95 / 1015 loss=4.168, nll_loss=2.534, ppl=5.79, wps=9201.9, ups=2.55, wpb=3614, bsz=164.3, num_updates=27500, lr=0.000381385, gnorm=1.153, train_wall=30, wall=8626\n",
            "2020-08-03 23:20:34 | INFO | train_inner | epoch 028:    195 / 1015 loss=4.145, nll_loss=2.508, ppl=5.69, wps=11649.1, ups=3.28, wpb=3556.2, bsz=164.6, num_updates=27600, lr=0.000380693, gnorm=1.149, train_wall=30, wall=8656\n",
            "2020-08-03 23:21:04 | INFO | train_inner | epoch 028:    295 / 1015 loss=4.272, nll_loss=2.65, ppl=6.28, wps=11516.9, ups=3.31, wpb=3478.7, bsz=138.2, num_updates=27700, lr=0.000380006, gnorm=1.228, train_wall=30, wall=8686\n",
            "2020-08-03 23:21:34 | INFO | train_inner | epoch 028:    395 / 1015 loss=4.288, nll_loss=2.668, ppl=6.36, wps=11606.3, ups=3.34, wpb=3480.1, bsz=139, num_updates=27800, lr=0.000379322, gnorm=1.212, train_wall=30, wall=8716\n",
            "2020-08-03 23:22:05 | INFO | train_inner | epoch 028:    495 / 1015 loss=4.21, nll_loss=2.582, ppl=5.99, wps=11615.3, ups=3.3, wpb=3516.1, bsz=166, num_updates=27900, lr=0.000378641, gnorm=1.179, train_wall=30, wall=8747\n",
            "2020-08-03 23:22:35 | INFO | train_inner | epoch 028:    595 / 1015 loss=4.165, nll_loss=2.532, ppl=5.78, wps=11818.4, ups=3.29, wpb=3594.2, bsz=178.2, num_updates=28000, lr=0.000377964, gnorm=1.162, train_wall=30, wall=8777\n",
            "2020-08-03 23:23:05 | INFO | train_inner | epoch 028:    695 / 1015 loss=4.254, nll_loss=2.633, ppl=6.2, wps=11720.3, ups=3.28, wpb=3578.6, bsz=152.9, num_updates=28100, lr=0.000377291, gnorm=1.177, train_wall=30, wall=8808\n",
            "2020-08-03 23:23:36 | INFO | train_inner | epoch 028:    795 / 1015 loss=4.222, nll_loss=2.596, ppl=6.05, wps=11755.9, ups=3.3, wpb=3567.8, bsz=157, num_updates=28200, lr=0.000376622, gnorm=1.133, train_wall=30, wall=8838\n",
            "2020-08-03 23:24:06 | INFO | train_inner | epoch 028:    895 / 1015 loss=4.223, nll_loss=2.599, ppl=6.06, wps=11788.2, ups=3.3, wpb=3569.8, bsz=166.3, num_updates=28300, lr=0.000375956, gnorm=1.173, train_wall=30, wall=8868\n",
            "2020-08-03 23:24:37 | INFO | train_inner | epoch 028:    995 / 1015 loss=4.273, nll_loss=2.656, ppl=6.3, wps=11764.5, ups=3.28, wpb=3587.8, bsz=160.3, num_updates=28400, lr=0.000375293, gnorm=1.178, train_wall=30, wall=8899\n",
            "2020-08-03 23:24:43 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:24:49 | INFO | valid | epoch 028 | valid on 'valid' subset | loss 4.395 | nll_loss 2.707 | ppl 6.53 | wps 27198.9 | wpb 2866.6 | bsz 127.8 | num_updates 28420 | best_loss 4.379\n",
            "2020-08-03 23:24:49 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:24:50 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint28.pt (epoch 28 @ 28420 updates, score 4.395) (writing took 1.537754015000246 seconds)\n",
            "2020-08-03 23:24:50 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:24:50 | INFO | fairseq_cli.train | end of epoch 28 (average epoch stats below)\n",
            "2020-08-03 23:24:50 | INFO | train | epoch 028 | loss 4.224 | nll_loss 2.598 | ppl 6.06 | wps 11413.1 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 28420 | lr 0.000375161 | gnorm 1.179 | train_wall 307 | wall 8912\n",
            "2020-08-03 23:24:50 | INFO | fairseq_cli.train | begin training epoch 28\n",
            "2020-08-03 23:25:14 | INFO | train_inner | epoch 029:     80 / 1015 loss=4.187, nll_loss=2.555, ppl=5.88, wps=9183, ups=2.65, wpb=3466.6, bsz=156.4, num_updates=28500, lr=0.000374634, gnorm=1.242, train_wall=30, wall=8936\n",
            "2020-08-03 23:25:45 | INFO | train_inner | epoch 029:    180 / 1015 loss=4.179, nll_loss=2.546, ppl=5.84, wps=11833.4, ups=3.3, wpb=3589.9, bsz=159.1, num_updates=28600, lr=0.000373979, gnorm=1.159, train_wall=30, wall=8967\n",
            "2020-08-03 23:26:15 | INFO | train_inner | epoch 029:    280 / 1015 loss=4.195, nll_loss=2.564, ppl=5.91, wps=11662.2, ups=3.28, wpb=3550.9, bsz=153.1, num_updates=28700, lr=0.000373327, gnorm=1.175, train_wall=30, wall=8997\n",
            "2020-08-03 23:26:46 | INFO | train_inner | epoch 029:    380 / 1015 loss=4.162, nll_loss=2.527, ppl=5.77, wps=11603.7, ups=3.28, wpb=3532.9, bsz=161.3, num_updates=28800, lr=0.000372678, gnorm=1.186, train_wall=30, wall=9028\n",
            "2020-08-03 23:27:16 | INFO | train_inner | epoch 029:    480 / 1015 loss=4.171, nll_loss=2.539, ppl=5.81, wps=11883.4, ups=3.3, wpb=3600.2, bsz=161.2, num_updates=28900, lr=0.000372033, gnorm=1.174, train_wall=30, wall=9058\n",
            "2020-08-03 23:27:46 | INFO | train_inner | epoch 029:    580 / 1015 loss=4.192, nll_loss=2.564, ppl=5.91, wps=11658.4, ups=3.31, wpb=3520.1, bsz=170.4, num_updates=29000, lr=0.000371391, gnorm=1.195, train_wall=30, wall=9088\n",
            "2020-08-03 23:28:16 | INFO | train_inner | epoch 029:    680 / 1015 loss=4.231, nll_loss=2.605, ppl=6.08, wps=11720, ups=3.29, wpb=3560.6, bsz=149.6, num_updates=29100, lr=0.000370752, gnorm=1.195, train_wall=30, wall=9119\n",
            "2020-08-03 23:28:47 | INFO | train_inner | epoch 029:    780 / 1015 loss=4.263, nll_loss=2.643, ppl=6.25, wps=11704.7, ups=3.29, wpb=3553.1, bsz=157.8, num_updates=29200, lr=0.000370117, gnorm=1.242, train_wall=30, wall=9149\n",
            "2020-08-03 23:29:17 | INFO | train_inner | epoch 029:    880 / 1015 loss=4.24, nll_loss=2.617, ppl=6.14, wps=11545.5, ups=3.29, wpb=3510, bsz=156.2, num_updates=29300, lr=0.000369484, gnorm=1.193, train_wall=30, wall=9179\n",
            "2020-08-03 23:29:48 | INFO | train_inner | epoch 029:    980 / 1015 loss=4.263, nll_loss=2.643, ppl=6.24, wps=11716.3, ups=3.27, wpb=3582.9, bsz=148.5, num_updates=29400, lr=0.000368856, gnorm=1.164, train_wall=30, wall=9210\n",
            "2020-08-03 23:29:58 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:30:04 | INFO | valid | epoch 029 | valid on 'valid' subset | loss 4.382 | nll_loss 2.69 | ppl 6.46 | wps 27200.3 | wpb 2866.6 | bsz 127.8 | num_updates 29435 | best_loss 4.379\n",
            "2020-08-03 23:30:04 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:30:06 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint29.pt (epoch 29 @ 29435 updates, score 4.382) (writing took 1.5057816480002657 seconds)\n",
            "2020-08-03 23:30:06 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:30:06 | INFO | fairseq_cli.train | end of epoch 29 (average epoch stats below)\n",
            "2020-08-03 23:30:06 | INFO | train | epoch 029 | loss 4.209 | nll_loss 2.581 | ppl 5.98 | wps 11412 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 29435 | lr 0.000368636 | gnorm 1.19 | train_wall 307 | wall 9228\n",
            "2020-08-03 23:30:06 | INFO | fairseq_cli.train | begin training epoch 29\n",
            "2020-08-03 23:30:26 | INFO | train_inner | epoch 030:     65 / 1015 loss=4.216, nll_loss=2.589, ppl=6.02, wps=9340.1, ups=2.64, wpb=3544.4, bsz=151.8, num_updates=29500, lr=0.00036823, gnorm=1.218, train_wall=30, wall=9248\n",
            "2020-08-03 23:30:56 | INFO | train_inner | epoch 030:    165 / 1015 loss=4.137, nll_loss=2.499, ppl=5.65, wps=11746.9, ups=3.28, wpb=3578.6, bsz=160.2, num_updates=29600, lr=0.000367607, gnorm=1.154, train_wall=30, wall=9278\n",
            "2020-08-03 23:31:27 | INFO | train_inner | epoch 030:    265 / 1015 loss=4.121, nll_loss=2.482, ppl=5.59, wps=11830.8, ups=3.3, wpb=3587.3, bsz=176.7, num_updates=29700, lr=0.000366988, gnorm=1.151, train_wall=30, wall=9309\n",
            "2020-08-03 23:31:57 | INFO | train_inner | epoch 030:    365 / 1015 loss=4.209, nll_loss=2.58, ppl=5.98, wps=11694.6, ups=3.28, wpb=3562.2, bsz=151.9, num_updates=29800, lr=0.000366372, gnorm=1.219, train_wall=30, wall=9339\n",
            "2020-08-03 23:32:27 | INFO | train_inner | epoch 030:    465 / 1015 loss=4.149, nll_loss=2.513, ppl=5.71, wps=11485.2, ups=3.28, wpb=3501.9, bsz=167.5, num_updates=29900, lr=0.000365758, gnorm=1.187, train_wall=30, wall=9370\n",
            "2020-08-03 23:32:58 | INFO | train_inner | epoch 030:    565 / 1015 loss=4.226, nll_loss=2.599, ppl=6.06, wps=11588.8, ups=3.31, wpb=3505.6, bsz=148.7, num_updates=30000, lr=0.000365148, gnorm=1.212, train_wall=30, wall=9400\n",
            "2020-08-03 23:33:28 | INFO | train_inner | epoch 030:    665 / 1015 loss=4.224, nll_loss=2.598, ppl=6.06, wps=11550.5, ups=3.29, wpb=3508.7, bsz=148.8, num_updates=30100, lr=0.000364541, gnorm=1.199, train_wall=30, wall=9430\n",
            "2020-08-03 23:33:58 | INFO | train_inner | epoch 030:    765 / 1015 loss=4.163, nll_loss=2.53, ppl=5.78, wps=11769.5, ups=3.29, wpb=3572.3, bsz=166.5, num_updates=30200, lr=0.000363937, gnorm=1.168, train_wall=30, wall=9461\n",
            "2020-08-03 23:34:29 | INFO | train_inner | epoch 030:    865 / 1015 loss=4.198, nll_loss=2.571, ppl=5.94, wps=11895.9, ups=3.28, wpb=3622.5, bsz=168.9, num_updates=30300, lr=0.000363336, gnorm=1.173, train_wall=30, wall=9491\n",
            "2020-08-03 23:34:59 | INFO | train_inner | epoch 030:    965 / 1015 loss=4.247, nll_loss=2.625, ppl=6.17, wps=11553.2, ups=3.29, wpb=3513.6, bsz=149.7, num_updates=30400, lr=0.000362738, gnorm=1.231, train_wall=30, wall=9521\n",
            "2020-08-03 23:35:15 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:35:21 | INFO | valid | epoch 030 | valid on 'valid' subset | loss 4.364 | nll_loss 2.671 | ppl 6.37 | wps 27216 | wpb 2866.6 | bsz 127.8 | num_updates 30450 | best_loss 4.364\n",
            "2020-08-03 23:35:21 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:35:23 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint30.pt (epoch 30 @ 30450 updates, score 4.364) (writing took 2.557249433999459 seconds)\n",
            "2020-08-03 23:35:23 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:35:23 | INFO | fairseq_cli.train | end of epoch 30 (average epoch stats below)\n",
            "2020-08-03 23:35:23 | INFO | train | epoch 030 | loss 4.189 | nll_loss 2.558 | ppl 5.89 | wps 11357.7 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 30450 | lr 0.00036244 | gnorm 1.189 | train_wall 308 | wall 9545\n",
            "2020-08-03 23:35:23 | INFO | fairseq_cli.train | begin training epoch 30\n",
            "2020-08-03 23:35:38 | INFO | train_inner | epoch 031:     50 / 1015 loss=4.22, nll_loss=2.592, ppl=6.03, wps=8998.4, ups=2.56, wpb=3514.8, bsz=141.4, num_updates=30500, lr=0.000362143, gnorm=1.203, train_wall=30, wall=9561\n",
            "2020-08-03 23:36:09 | INFO | train_inner | epoch 031:    150 / 1015 loss=4.107, nll_loss=2.467, ppl=5.53, wps=11946, ups=3.27, wpb=3651.4, bsz=180.7, num_updates=30600, lr=0.000361551, gnorm=1.149, train_wall=30, wall=9591\n",
            "2020-08-03 23:36:39 | INFO | train_inner | epoch 031:    250 / 1015 loss=4.145, nll_loss=2.508, ppl=5.69, wps=11806.4, ups=3.32, wpb=3560.2, bsz=154.2, num_updates=30700, lr=0.000360961, gnorm=1.17, train_wall=30, wall=9621\n",
            "2020-08-03 23:37:10 | INFO | train_inner | epoch 031:    350 / 1015 loss=4.194, nll_loss=2.562, ppl=5.9, wps=11512.6, ups=3.28, wpb=3508.6, bsz=139.2, num_updates=30800, lr=0.000360375, gnorm=1.225, train_wall=30, wall=9652\n",
            "2020-08-03 23:37:40 | INFO | train_inner | epoch 031:    450 / 1015 loss=4.141, nll_loss=2.504, ppl=5.67, wps=11582.3, ups=3.32, wpb=3492.6, bsz=171.3, num_updates=30900, lr=0.000359791, gnorm=1.231, train_wall=30, wall=9682\n",
            "2020-08-03 23:38:10 | INFO | train_inner | epoch 031:    550 / 1015 loss=4.189, nll_loss=2.557, ppl=5.89, wps=11676.1, ups=3.29, wpb=3553.7, bsz=155.2, num_updates=31000, lr=0.000359211, gnorm=1.211, train_wall=30, wall=9712\n",
            "2020-08-03 23:38:41 | INFO | train_inner | epoch 031:    650 / 1015 loss=4.176, nll_loss=2.543, ppl=5.83, wps=11525.8, ups=3.3, wpb=3495.1, bsz=150.1, num_updates=31100, lr=0.000358633, gnorm=1.241, train_wall=30, wall=9743\n",
            "2020-08-03 23:39:11 | INFO | train_inner | epoch 031:    750 / 1015 loss=4.205, nll_loss=2.577, ppl=5.97, wps=11593.8, ups=3.28, wpb=3539.2, bsz=155, num_updates=31200, lr=0.000358057, gnorm=1.208, train_wall=30, wall=9773\n",
            "2020-08-03 23:39:41 | INFO | train_inner | epoch 031:    850 / 1015 loss=4.191, nll_loss=2.562, ppl=5.91, wps=11852.5, ups=3.29, wpb=3604.8, bsz=158.2, num_updates=31300, lr=0.000357485, gnorm=1.159, train_wall=30, wall=9804\n",
            "2020-08-03 23:40:12 | INFO | train_inner | epoch 031:    950 / 1015 loss=4.199, nll_loss=2.571, ppl=5.94, wps=11826.5, ups=3.28, wpb=3600.3, bsz=158.7, num_updates=31400, lr=0.000356915, gnorm=1.177, train_wall=30, wall=9834\n",
            "2020-08-03 23:40:32 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:40:38 | INFO | valid | epoch 031 | valid on 'valid' subset | loss 4.378 | nll_loss 2.683 | ppl 6.42 | wps 27274.7 | wpb 2866.6 | bsz 127.8 | num_updates 31465 | best_loss 4.364\n",
            "2020-08-03 23:40:38 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:40:39 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint31.pt (epoch 31 @ 31465 updates, score 4.378) (writing took 1.5130572670004767 seconds)\n",
            "2020-08-03 23:40:39 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:40:39 | INFO | fairseq_cli.train | end of epoch 31 (average epoch stats below)\n",
            "2020-08-03 23:40:39 | INFO | train | epoch 031 | loss 4.173 | nll_loss 2.541 | ppl 5.82 | wps 11403.7 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 31465 | lr 0.000356546 | gnorm 1.197 | train_wall 307 | wall 9861\n",
            "2020-08-03 23:40:39 | INFO | fairseq_cli.train | begin training epoch 31\n",
            "2020-08-03 23:40:50 | INFO | train_inner | epoch 032:     35 / 1015 loss=4.187, nll_loss=2.557, ppl=5.88, wps=9252.2, ups=2.64, wpb=3510.7, bsz=152.4, num_updates=31500, lr=0.000356348, gnorm=1.19, train_wall=30, wall=9872\n",
            "2020-08-03 23:41:20 | INFO | train_inner | epoch 032:    135 / 1015 loss=4.129, nll_loss=2.49, ppl=5.62, wps=11727.7, ups=3.28, wpb=3575.4, bsz=151.5, num_updates=31600, lr=0.000355784, gnorm=1.198, train_wall=30, wall=9902\n",
            "2020-08-03 23:41:51 | INFO | train_inner | epoch 032:    235 / 1015 loss=4.078, nll_loss=2.432, ppl=5.39, wps=11829.7, ups=3.27, wpb=3613.3, bsz=162.9, num_updates=31700, lr=0.000355222, gnorm=1.139, train_wall=30, wall=9933\n",
            "2020-08-03 23:42:21 | INFO | train_inner | epoch 032:    335 / 1015 loss=4.137, nll_loss=2.499, ppl=5.65, wps=11542.3, ups=3.33, wpb=3465.3, bsz=161, num_updates=31800, lr=0.000354663, gnorm=1.282, train_wall=30, wall=9963\n",
            "2020-08-03 23:42:51 | INFO | train_inner | epoch 032:    435 / 1015 loss=4.179, nll_loss=2.547, ppl=5.84, wps=11931.3, ups=3.28, wpb=3637.3, bsz=142.3, num_updates=31900, lr=0.000354107, gnorm=1.186, train_wall=30, wall=9993\n",
            "2020-08-03 23:43:22 | INFO | train_inner | epoch 032:    535 / 1015 loss=4.207, nll_loss=2.58, ppl=5.98, wps=11596.6, ups=3.3, wpb=3510.3, bsz=149.5, num_updates=32000, lr=0.000353553, gnorm=1.237, train_wall=30, wall=10024\n",
            "2020-08-03 23:43:52 | INFO | train_inner | epoch 032:    635 / 1015 loss=4.159, nll_loss=2.523, ppl=5.75, wps=11792.2, ups=3.29, wpb=3584.8, bsz=157.9, num_updates=32100, lr=0.000353002, gnorm=1.19, train_wall=30, wall=10054\n",
            "2020-08-03 23:44:22 | INFO | train_inner | epoch 032:    735 / 1015 loss=4.22, nll_loss=2.593, ppl=6.03, wps=11531.9, ups=3.32, wpb=3470.2, bsz=142.9, num_updates=32200, lr=0.000352454, gnorm=1.239, train_wall=30, wall=10084\n",
            "2020-08-03 23:44:53 | INFO | train_inner | epoch 032:    835 / 1015 loss=4.17, nll_loss=2.538, ppl=5.81, wps=11516.6, ups=3.28, wpb=3506.5, bsz=166.6, num_updates=32300, lr=0.000351908, gnorm=1.259, train_wall=30, wall=10115\n",
            "2020-08-03 23:45:23 | INFO | train_inner | epoch 032:    935 / 1015 loss=4.119, nll_loss=2.483, ppl=5.59, wps=11862.4, ups=3.31, wpb=3581.4, bsz=188.4, num_updates=32400, lr=0.000351364, gnorm=1.146, train_wall=30, wall=10145\n",
            "2020-08-03 23:45:47 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:45:53 | INFO | valid | epoch 032 | valid on 'valid' subset | loss 4.37 | nll_loss 2.677 | ppl 6.4 | wps 27078.8 | wpb 2866.6 | bsz 127.8 | num_updates 32480 | best_loss 4.364\n",
            "2020-08-03 23:45:53 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:45:55 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint32.pt (epoch 32 @ 32480 updates, score 4.37) (writing took 1.6016273250006634 seconds)\n",
            "2020-08-03 23:45:55 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:45:55 | INFO | fairseq_cli.train | end of epoch 32 (average epoch stats below)\n",
            "2020-08-03 23:45:55 | INFO | train | epoch 032 | loss 4.157 | nll_loss 2.522 | ppl 5.74 | wps 11417.4 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 32480 | lr 0.000350931 | gnorm 1.206 | train_wall 307 | wall 10177\n",
            "2020-08-03 23:45:55 | INFO | fairseq_cli.train | begin training epoch 32\n",
            "2020-08-03 23:46:01 | INFO | train_inner | epoch 033:     20 / 1015 loss=4.163, nll_loss=2.53, ppl=5.77, wps=9403.8, ups=2.62, wpb=3590.4, bsz=163.9, num_updates=32500, lr=0.000350823, gnorm=1.184, train_wall=30, wall=10183\n",
            "2020-08-03 23:46:31 | INFO | train_inner | epoch 033:    120 / 1015 loss=4.076, nll_loss=2.43, ppl=5.39, wps=11776, ups=3.31, wpb=3560.9, bsz=162.6, num_updates=32600, lr=0.000350285, gnorm=1.162, train_wall=30, wall=10213\n",
            "2020-08-03 23:47:02 | INFO | train_inner | epoch 033:    220 / 1015 loss=4.103, nll_loss=2.46, ppl=5.5, wps=11907, ups=3.27, wpb=3643.3, bsz=151.2, num_updates=32700, lr=0.000349749, gnorm=1.159, train_wall=31, wall=10244\n",
            "2020-08-03 23:47:32 | INFO | train_inner | epoch 033:    320 / 1015 loss=4.129, nll_loss=2.491, ppl=5.62, wps=11558.6, ups=3.3, wpb=3506.6, bsz=154.6, num_updates=32800, lr=0.000349215, gnorm=1.211, train_wall=30, wall=10274\n",
            "2020-08-03 23:48:02 | INFO | train_inner | epoch 033:    420 / 1015 loss=4.18, nll_loss=2.547, ppl=5.84, wps=11751.1, ups=3.3, wpb=3559.2, bsz=145.3, num_updates=32900, lr=0.000348684, gnorm=1.244, train_wall=30, wall=10305\n",
            "2020-08-03 23:48:33 | INFO | train_inner | epoch 033:    520 / 1015 loss=4.131, nll_loss=2.493, ppl=5.63, wps=11549.5, ups=3.29, wpb=3506.7, bsz=165.3, num_updates=33000, lr=0.000348155, gnorm=1.204, train_wall=30, wall=10335\n",
            "2020-08-03 23:49:03 | INFO | train_inner | epoch 033:    620 / 1015 loss=4.183, nll_loss=2.551, ppl=5.86, wps=11667, ups=3.29, wpb=3545.7, bsz=151.9, num_updates=33100, lr=0.000347629, gnorm=1.238, train_wall=30, wall=10365\n",
            "2020-08-03 23:49:34 | INFO | train_inner | epoch 033:    720 / 1015 loss=4.175, nll_loss=2.543, ppl=5.83, wps=11693.3, ups=3.29, wpb=3556.6, bsz=161.4, num_updates=33200, lr=0.000347105, gnorm=1.227, train_wall=30, wall=10396\n",
            "2020-08-03 23:50:04 | INFO | train_inner | epoch 033:    820 / 1015 loss=4.164, nll_loss=2.531, ppl=5.78, wps=11735.8, ups=3.29, wpb=3563.5, bsz=159.3, num_updates=33300, lr=0.000346583, gnorm=1.226, train_wall=30, wall=10426\n",
            "2020-08-03 23:50:34 | INFO | train_inner | epoch 033:    920 / 1015 loss=4.115, nll_loss=2.477, ppl=5.57, wps=11659.4, ups=3.29, wpb=3547.7, bsz=174.1, num_updates=33400, lr=0.000346064, gnorm=1.221, train_wall=30, wall=10457\n",
            "2020-08-03 23:51:03 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:51:09 | INFO | valid | epoch 033 | valid on 'valid' subset | loss 4.347 | nll_loss 2.657 | ppl 6.31 | wps 27078.9 | wpb 2866.6 | bsz 127.8 | num_updates 33495 | best_loss 4.347\n",
            "2020-08-03 23:51:09 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:51:12 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint33.pt (epoch 33 @ 33495 updates, score 4.347) (writing took 2.4600996910012327 seconds)\n",
            "2020-08-03 23:51:12 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:51:12 | INFO | fairseq_cli.train | end of epoch 33 (average epoch stats below)\n",
            "2020-08-03 23:51:12 | INFO | train | epoch 033 | loss 4.145 | nll_loss 2.509 | ppl 5.69 | wps 11376.1 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 33495 | lr 0.000345573 | gnorm 1.212 | train_wall 307 | wall 10494\n",
            "2020-08-03 23:51:12 | INFO | fairseq_cli.train | begin training epoch 33\n",
            "2020-08-03 23:51:13 | INFO | train_inner | epoch 034:      5 / 1015 loss=4.201, nll_loss=2.573, ppl=5.95, wps=9051.1, ups=2.58, wpb=3512.7, bsz=151.1, num_updates=33500, lr=0.000345547, gnorm=1.231, train_wall=30, wall=10495\n",
            "2020-08-03 23:51:44 | INFO | train_inner | epoch 034:    105 / 1015 loss=4.095, nll_loss=2.451, ppl=5.47, wps=11796, ups=3.29, wpb=3586.6, bsz=158, num_updates=33600, lr=0.000345033, gnorm=1.198, train_wall=30, wall=10526\n",
            "2020-08-03 23:52:14 | INFO | train_inner | epoch 034:    205 / 1015 loss=4.044, nll_loss=2.394, ppl=5.26, wps=11835.6, ups=3.28, wpb=3606, bsz=172, num_updates=33700, lr=0.00034452, gnorm=1.146, train_wall=30, wall=10556\n",
            "2020-08-03 23:52:44 | INFO | train_inner | epoch 034:    305 / 1015 loss=4.174, nll_loss=2.539, ppl=5.81, wps=11719.5, ups=3.35, wpb=3501.2, bsz=132.3, num_updates=33800, lr=0.00034401, gnorm=1.248, train_wall=30, wall=10586\n",
            "2020-08-03 23:53:15 | INFO | train_inner | epoch 034:    405 / 1015 loss=4.121, nll_loss=2.479, ppl=5.58, wps=11554.6, ups=3.26, wpb=3541.4, bsz=150.4, num_updates=33900, lr=0.000343503, gnorm=1.231, train_wall=31, wall=10617\n",
            "2020-08-03 23:53:45 | INFO | train_inner | epoch 034:    505 / 1015 loss=4.079, nll_loss=2.435, ppl=5.41, wps=11768.7, ups=3.29, wpb=3577.1, bsz=178.8, num_updates=34000, lr=0.000342997, gnorm=1.174, train_wall=30, wall=10647\n",
            "2020-08-03 23:54:15 | INFO | train_inner | epoch 034:    605 / 1015 loss=4.135, nll_loss=2.498, ppl=5.65, wps=11717, ups=3.31, wpb=3542.8, bsz=168.7, num_updates=34100, lr=0.000342494, gnorm=1.27, train_wall=30, wall=10677\n",
            "2020-08-03 23:54:46 | INFO | train_inner | epoch 034:    705 / 1015 loss=4.133, nll_loss=2.495, ppl=5.64, wps=11706.8, ups=3.29, wpb=3554.7, bsz=166.9, num_updates=34200, lr=0.000341993, gnorm=1.219, train_wall=30, wall=10708\n",
            "2020-08-03 23:55:16 | INFO | train_inner | epoch 034:    805 / 1015 loss=4.188, nll_loss=2.557, ppl=5.89, wps=11776.9, ups=3.3, wpb=3564, bsz=143.4, num_updates=34300, lr=0.000341494, gnorm=1.213, train_wall=30, wall=10738\n",
            "2020-08-03 23:55:46 | INFO | train_inner | epoch 034:    905 / 1015 loss=4.127, nll_loss=2.49, ppl=5.62, wps=11733.6, ups=3.3, wpb=3551.3, bsz=167.4, num_updates=34400, lr=0.000340997, gnorm=1.206, train_wall=30, wall=10768\n",
            "2020-08-03 23:56:17 | INFO | train_inner | epoch 034:   1005 / 1015 loss=4.191, nll_loss=2.561, ppl=5.9, wps=11565.4, ups=3.29, wpb=3518.9, bsz=147.8, num_updates=34500, lr=0.000340503, gnorm=1.247, train_wall=30, wall=10799\n",
            "2020-08-03 23:56:19 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-03 23:56:26 | INFO | valid | epoch 034 | valid on 'valid' subset | loss 4.354 | nll_loss 2.668 | ppl 6.35 | wps 27181 | wpb 2866.6 | bsz 127.8 | num_updates 34510 | best_loss 4.347\n",
            "2020-08-03 23:56:26 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-03 23:56:27 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint34.pt (epoch 34 @ 34510 updates, score 4.354) (writing took 1.6103466090007714 seconds)\n",
            "2020-08-03 23:56:27 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-03 23:56:27 | INFO | fairseq_cli.train | end of epoch 34 (average epoch stats below)\n",
            "2020-08-03 23:56:27 | INFO | train | epoch 034 | loss 4.131 | nll_loss 2.493 | ppl 5.63 | wps 11417.6 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 34510 | lr 0.000340453 | gnorm 1.219 | train_wall 307 | wall 10809\n",
            "2020-08-03 23:56:27 | INFO | fairseq_cli.train | begin training epoch 34\n",
            "2020-08-03 23:56:55 | INFO | train_inner | epoch 035:     90 / 1015 loss=4.148, nll_loss=2.51, ppl=5.7, wps=9254.8, ups=2.64, wpb=3511.7, bsz=138.7, num_updates=34600, lr=0.00034001, gnorm=1.282, train_wall=30, wall=10837\n",
            "2020-08-03 23:57:25 | INFO | train_inner | epoch 035:    190 / 1015 loss=4.071, nll_loss=2.425, ppl=5.37, wps=11670.1, ups=3.3, wpb=3538.3, bsz=160.1, num_updates=34700, lr=0.00033952, gnorm=1.203, train_wall=30, wall=10867\n",
            "2020-08-03 23:57:55 | INFO | train_inner | epoch 035:    290 / 1015 loss=4.096, nll_loss=2.453, ppl=5.48, wps=11838.2, ups=3.31, wpb=3579.1, bsz=159.4, num_updates=34800, lr=0.000339032, gnorm=1.194, train_wall=30, wall=10897\n",
            "2020-08-03 23:58:25 | INFO | train_inner | epoch 035:    390 / 1015 loss=4.123, nll_loss=2.482, ppl=5.59, wps=11645.2, ups=3.31, wpb=3520.6, bsz=151.9, num_updates=34900, lr=0.000338546, gnorm=1.226, train_wall=30, wall=10927\n",
            "2020-08-03 23:58:56 | INFO | train_inner | epoch 035:    490 / 1015 loss=4.135, nll_loss=2.496, ppl=5.64, wps=11817.7, ups=3.3, wpb=3579.3, bsz=153.8, num_updates=35000, lr=0.000338062, gnorm=1.194, train_wall=30, wall=10958\n",
            "2020-08-03 23:59:26 | INFO | train_inner | epoch 035:    590 / 1015 loss=4.111, nll_loss=2.472, ppl=5.55, wps=11802.1, ups=3.28, wpb=3595.2, bsz=169.8, num_updates=35100, lr=0.00033758, gnorm=1.181, train_wall=30, wall=10988\n",
            "2020-08-03 23:59:56 | INFO | train_inner | epoch 035:    690 / 1015 loss=4.088, nll_loss=2.445, ppl=5.44, wps=11656.5, ups=3.29, wpb=3539.3, bsz=171.3, num_updates=35200, lr=0.0003371, gnorm=1.212, train_wall=30, wall=11019\n",
            "2020-08-04 00:00:27 | INFO | train_inner | epoch 035:    790 / 1015 loss=4.175, nll_loss=2.543, ppl=5.83, wps=11585.1, ups=3.29, wpb=3525.9, bsz=146.8, num_updates=35300, lr=0.000336622, gnorm=1.238, train_wall=30, wall=11049\n",
            "2020-08-04 00:00:57 | INFO | train_inner | epoch 035:    890 / 1015 loss=4.101, nll_loss=2.461, ppl=5.51, wps=11619.6, ups=3.3, wpb=3524, bsz=169.6, num_updates=35400, lr=0.000336146, gnorm=1.223, train_wall=30, wall=11079\n",
            "2020-08-04 00:01:27 | INFO | train_inner | epoch 035:    990 / 1015 loss=4.135, nll_loss=2.498, ppl=5.65, wps=11682.7, ups=3.3, wpb=3543.3, bsz=156.9, num_updates=35500, lr=0.000335673, gnorm=1.27, train_wall=30, wall=11110\n",
            "2020-08-04 00:01:35 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:01:41 | INFO | valid | epoch 035 | valid on 'valid' subset | loss 4.338 | nll_loss 2.646 | ppl 6.26 | wps 27126.8 | wpb 2866.6 | bsz 127.8 | num_updates 35525 | best_loss 4.338\n",
            "2020-08-04 00:01:41 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:01:44 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint35.pt (epoch 35 @ 35525 updates, score 4.338) (writing took 2.5593695520001347 seconds)\n",
            "2020-08-04 00:01:44 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:01:44 | INFO | fairseq_cli.train | end of epoch 35 (average epoch stats below)\n",
            "2020-08-04 00:01:44 | INFO | train | epoch 035 | loss 4.117 | nll_loss 2.477 | ppl 5.57 | wps 11380.6 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 35525 | lr 0.000335554 | gnorm 1.219 | train_wall 307 | wall 11126\n",
            "2020-08-04 00:01:44 | INFO | fairseq_cli.train | begin training epoch 35\n",
            "2020-08-04 00:02:07 | INFO | train_inner | epoch 036:     75 / 1015 loss=4.077, nll_loss=2.431, ppl=5.39, wps=9093.7, ups=2.55, wpb=3572.4, bsz=156.2, num_updates=35600, lr=0.000335201, gnorm=1.207, train_wall=30, wall=11149\n",
            "2020-08-04 00:02:37 | INFO | train_inner | epoch 036:    175 / 1015 loss=4.052, nll_loss=2.403, ppl=5.29, wps=11574.8, ups=3.31, wpb=3502.1, bsz=168.2, num_updates=35700, lr=0.000334731, gnorm=1.241, train_wall=30, wall=11179\n",
            "2020-08-04 00:03:08 | INFO | train_inner | epoch 036:    275 / 1015 loss=4.028, nll_loss=2.376, ppl=5.19, wps=11720.9, ups=3.28, wpb=3571.7, bsz=178.5, num_updates=35800, lr=0.000334263, gnorm=1.179, train_wall=30, wall=11210\n",
            "2020-08-04 00:03:38 | INFO | train_inner | epoch 036:    375 / 1015 loss=4.16, nll_loss=2.523, ppl=5.75, wps=11748.1, ups=3.28, wpb=3586.2, bsz=134.8, num_updates=35900, lr=0.000333797, gnorm=1.244, train_wall=30, wall=11240\n",
            "2020-08-04 00:04:08 | INFO | train_inner | epoch 036:    475 / 1015 loss=4.085, nll_loss=2.441, ppl=5.43, wps=11814.2, ups=3.3, wpb=3583.4, bsz=161.7, num_updates=36000, lr=0.000333333, gnorm=1.208, train_wall=30, wall=11270\n",
            "2020-08-04 00:04:39 | INFO | train_inner | epoch 036:    575 / 1015 loss=4.077, nll_loss=2.433, ppl=5.4, wps=11646.6, ups=3.3, wpb=3525.1, bsz=167.4, num_updates=36100, lr=0.000332871, gnorm=1.226, train_wall=30, wall=11301\n",
            "2020-08-04 00:05:09 | INFO | train_inner | epoch 036:    675 / 1015 loss=4.142, nll_loss=2.505, ppl=5.68, wps=11711.3, ups=3.28, wpb=3565.9, bsz=145.8, num_updates=36200, lr=0.000332411, gnorm=1.223, train_wall=30, wall=11331\n",
            "2020-08-04 00:05:39 | INFO | train_inner | epoch 036:    775 / 1015 loss=4.152, nll_loss=2.518, ppl=5.73, wps=11888.6, ups=3.3, wpb=3599.4, bsz=152.1, num_updates=36300, lr=0.000331953, gnorm=1.21, train_wall=30, wall=11361\n",
            "2020-08-04 00:06:09 | INFO | train_inner | epoch 036:    875 / 1015 loss=4.13, nll_loss=2.494, ppl=5.63, wps=11694.9, ups=3.32, wpb=3519.1, bsz=162.8, num_updates=36400, lr=0.000331497, gnorm=1.252, train_wall=30, wall=11392\n",
            "2020-08-04 00:06:40 | INFO | train_inner | epoch 036:    975 / 1015 loss=4.159, nll_loss=2.525, ppl=5.75, wps=11522.4, ups=3.29, wpb=3502.9, bsz=154.7, num_updates=36500, lr=0.000331042, gnorm=1.265, train_wall=30, wall=11422\n",
            "2020-08-04 00:06:52 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:06:58 | INFO | valid | epoch 036 | valid on 'valid' subset | loss 4.369 | nll_loss 2.672 | ppl 6.37 | wps 27145.2 | wpb 2866.6 | bsz 127.8 | num_updates 36540 | best_loss 4.338\n",
            "2020-08-04 00:06:58 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:07:00 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint36.pt (epoch 36 @ 36540 updates, score 4.369) (writing took 1.578139679000742 seconds)\n",
            "2020-08-04 00:07:00 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:07:00 | INFO | fairseq_cli.train | end of epoch 36 (average epoch stats below)\n",
            "2020-08-04 00:07:00 | INFO | train | epoch 036 | loss 4.106 | nll_loss 2.464 | ppl 5.52 | wps 11407.7 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 36540 | lr 0.000330861 | gnorm 1.227 | train_wall 307 | wall 11442\n",
            "2020-08-04 00:07:00 | INFO | fairseq_cli.train | begin training epoch 36\n",
            "2020-08-04 00:07:18 | INFO | train_inner | epoch 037:     60 / 1015 loss=4.053, nll_loss=2.405, ppl=5.3, wps=9331, ups=2.61, wpb=3569.9, bsz=157.4, num_updates=36600, lr=0.00033059, gnorm=1.218, train_wall=30, wall=11460\n",
            "2020-08-04 00:07:48 | INFO | train_inner | epoch 037:    160 / 1015 loss=4.056, nll_loss=2.406, ppl=5.3, wps=11502.8, ups=3.31, wpb=3474.6, bsz=156.7, num_updates=36700, lr=0.000330139, gnorm=1.263, train_wall=30, wall=11490\n",
            "2020-08-04 00:08:18 | INFO | train_inner | epoch 037:    260 / 1015 loss=4.077, nll_loss=2.432, ppl=5.4, wps=11658.1, ups=3.32, wpb=3511.1, bsz=164.3, num_updates=36800, lr=0.00032969, gnorm=1.271, train_wall=30, wall=11521\n",
            "2020-08-04 00:08:49 | INFO | train_inner | epoch 037:    360 / 1015 loss=4.056, nll_loss=2.409, ppl=5.31, wps=11752.2, ups=3.26, wpb=3602.7, bsz=162.3, num_updates=36900, lr=0.000329243, gnorm=1.204, train_wall=31, wall=11551\n",
            "2020-08-04 00:09:19 | INFO | train_inner | epoch 037:    460 / 1015 loss=4.105, nll_loss=2.462, ppl=5.51, wps=11690.7, ups=3.31, wpb=3534.8, bsz=145.4, num_updates=37000, lr=0.000328798, gnorm=1.243, train_wall=30, wall=11581\n",
            "2020-08-04 00:09:50 | INFO | train_inner | epoch 037:    560 / 1015 loss=4.118, nll_loss=2.478, ppl=5.57, wps=11750.3, ups=3.31, wpb=3555.3, bsz=156.9, num_updates=37100, lr=0.000328355, gnorm=1.227, train_wall=30, wall=11612\n",
            "2020-08-04 00:10:20 | INFO | train_inner | epoch 037:    660 / 1015 loss=4.022, nll_loss=2.369, ppl=5.17, wps=11603.4, ups=3.26, wpb=3563.2, bsz=177.8, num_updates=37200, lr=0.000327913, gnorm=1.212, train_wall=31, wall=11642\n",
            "2020-08-04 00:10:51 | INFO | train_inner | epoch 037:    760 / 1015 loss=4.108, nll_loss=2.469, ppl=5.53, wps=11826.1, ups=3.3, wpb=3579.2, bsz=158.8, num_updates=37300, lr=0.000327473, gnorm=1.225, train_wall=30, wall=11673\n",
            "2020-08-04 00:11:21 | INFO | train_inner | epoch 037:    860 / 1015 loss=4.171, nll_loss=2.539, ppl=5.81, wps=11809.3, ups=3.31, wpb=3568.1, bsz=142.7, num_updates=37400, lr=0.000327035, gnorm=1.252, train_wall=30, wall=11703\n",
            "2020-08-04 00:11:51 | INFO | train_inner | epoch 037:    960 / 1015 loss=4.154, nll_loss=2.52, ppl=5.74, wps=11619.3, ups=3.3, wpb=3520.7, bsz=159.3, num_updates=37500, lr=0.000326599, gnorm=1.263, train_wall=30, wall=11733\n",
            "2020-08-04 00:12:08 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:12:14 | INFO | valid | epoch 037 | valid on 'valid' subset | loss 4.332 | nll_loss 2.638 | ppl 6.22 | wps 27148.6 | wpb 2866.6 | bsz 127.8 | num_updates 37555 | best_loss 4.332\n",
            "2020-08-04 00:12:14 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:12:16 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint37.pt (epoch 37 @ 37555 updates, score 4.332) (writing took 2.556094073001077 seconds)\n",
            "2020-08-04 00:12:16 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:12:16 | INFO | fairseq_cli.train | end of epoch 37 (average epoch stats below)\n",
            "2020-08-04 00:12:16 | INFO | train | epoch 037 | loss 4.094 | nll_loss 2.451 | ppl 5.47 | wps 11377.3 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 37555 | lr 0.000326359 | gnorm 1.237 | train_wall 307 | wall 11758\n",
            "2020-08-04 00:12:16 | INFO | fairseq_cli.train | begin training epoch 37\n",
            "2020-08-04 00:12:30 | INFO | train_inner | epoch 038:     45 / 1015 loss=4.091, nll_loss=2.448, ppl=5.46, wps=9120.3, ups=2.56, wpb=3565.5, bsz=151.9, num_updates=37600, lr=0.000326164, gnorm=1.229, train_wall=30, wall=11772\n",
            "2020-08-04 00:13:01 | INFO | train_inner | epoch 038:    145 / 1015 loss=4.029, nll_loss=2.376, ppl=5.19, wps=11608.5, ups=3.28, wpb=3544.3, bsz=164.5, num_updates=37700, lr=0.000325731, gnorm=1.26, train_wall=30, wall=11803\n",
            "2020-08-04 00:13:31 | INFO | train_inner | epoch 038:    245 / 1015 loss=4.068, nll_loss=2.42, ppl=5.35, wps=11868.2, ups=3.26, wpb=3637.1, bsz=154.6, num_updates=37800, lr=0.0003253, gnorm=1.213, train_wall=31, wall=11833\n",
            "2020-08-04 00:14:02 | INFO | train_inner | epoch 038:    345 / 1015 loss=4.077, nll_loss=2.432, ppl=5.4, wps=11547, ups=3.31, wpb=3490.3, bsz=152.1, num_updates=37900, lr=0.000324871, gnorm=1.269, train_wall=30, wall=11864\n",
            "2020-08-04 00:14:32 | INFO | train_inner | epoch 038:    445 / 1015 loss=4.065, nll_loss=2.418, ppl=5.34, wps=11598.6, ups=3.32, wpb=3492.6, bsz=159, num_updates=38000, lr=0.000324443, gnorm=1.251, train_wall=30, wall=11894\n",
            "2020-08-04 00:15:02 | INFO | train_inner | epoch 038:    545 / 1015 loss=4.122, nll_loss=2.481, ppl=5.58, wps=11416.3, ups=3.31, wpb=3448.7, bsz=150, num_updates=38100, lr=0.000324017, gnorm=1.286, train_wall=30, wall=11924\n",
            "2020-08-04 00:15:33 | INFO | train_inner | epoch 038:    645 / 1015 loss=4.063, nll_loss=2.416, ppl=5.34, wps=11806, ups=3.26, wpb=3623.4, bsz=162, num_updates=38200, lr=0.000323592, gnorm=1.184, train_wall=31, wall=11955\n",
            "2020-08-04 00:16:03 | INFO | train_inner | epoch 038:    745 / 1015 loss=4.101, nll_loss=2.46, ppl=5.5, wps=11914, ups=3.3, wpb=3613.5, bsz=161.1, num_updates=38300, lr=0.00032317, gnorm=1.216, train_wall=30, wall=11985\n",
            "2020-08-04 00:16:33 | INFO | train_inner | epoch 038:    845 / 1015 loss=4.158, nll_loss=2.525, ppl=5.76, wps=11749.4, ups=3.33, wpb=3527, bsz=148.5, num_updates=38400, lr=0.000322749, gnorm=1.3, train_wall=30, wall=12015\n",
            "2020-08-04 00:17:03 | INFO | train_inner | epoch 038:    945 / 1015 loss=4.126, nll_loss=2.488, ppl=5.61, wps=11667.8, ups=3.28, wpb=3557.8, bsz=154.4, num_updates=38500, lr=0.000322329, gnorm=1.263, train_wall=30, wall=12046\n",
            "2020-08-04 00:17:25 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:17:31 | INFO | valid | epoch 038 | valid on 'valid' subset | loss 4.328 | nll_loss 2.632 | ppl 6.2 | wps 27215.8 | wpb 2866.6 | bsz 127.8 | num_updates 38570 | best_loss 4.328\n",
            "2020-08-04 00:17:31 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:17:33 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint38.pt (epoch 38 @ 38570 updates, score 4.328) (writing took 2.5008270149992313 seconds)\n",
            "2020-08-04 00:17:33 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:17:33 | INFO | fairseq_cli.train | end of epoch 38 (average epoch stats below)\n",
            "2020-08-04 00:17:33 | INFO | train | epoch 038 | loss 4.084 | nll_loss 2.44 | ppl 5.42 | wps 11376 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 38570 | lr 0.000322037 | gnorm 1.246 | train_wall 307 | wall 12075\n",
            "2020-08-04 00:17:33 | INFO | fairseq_cli.train | begin training epoch 38\n",
            "2020-08-04 00:17:43 | INFO | train_inner | epoch 039:     30 / 1015 loss=4.03, nll_loss=2.379, ppl=5.2, wps=9097.1, ups=2.56, wpb=3559, bsz=172.1, num_updates=38600, lr=0.000321911, gnorm=1.218, train_wall=30, wall=12085\n",
            "2020-08-04 00:18:13 | INFO | train_inner | epoch 039:    130 / 1015 loss=4.055, nll_loss=2.407, ppl=5.3, wps=11710.7, ups=3.31, wpb=3539.1, bsz=155.9, num_updates=38700, lr=0.000321495, gnorm=1.233, train_wall=30, wall=12115\n",
            "2020-08-04 00:18:43 | INFO | train_inner | epoch 039:    230 / 1015 loss=4.05, nll_loss=2.4, ppl=5.28, wps=11548.9, ups=3.28, wpb=3518.9, bsz=160.6, num_updates=38800, lr=0.000321081, gnorm=1.265, train_wall=30, wall=12145\n",
            "2020-08-04 00:19:13 | INFO | train_inner | epoch 039:    330 / 1015 loss=4.068, nll_loss=2.42, ppl=5.35, wps=11692.2, ups=3.32, wpb=3524.2, bsz=149, num_updates=38900, lr=0.000320668, gnorm=1.269, train_wall=30, wall=12176\n",
            "2020-08-04 00:19:44 | INFO | train_inner | epoch 039:    430 / 1015 loss=4.076, nll_loss=2.431, ppl=5.39, wps=11549.9, ups=3.32, wpb=3481, bsz=154.6, num_updates=39000, lr=0.000320256, gnorm=1.292, train_wall=30, wall=12206\n",
            "2020-08-04 00:20:14 | INFO | train_inner | epoch 039:    530 / 1015 loss=3.971, nll_loss=2.315, ppl=4.97, wps=11893.7, ups=3.27, wpb=3637.3, bsz=197, num_updates=39100, lr=0.000319847, gnorm=1.173, train_wall=30, wall=12236\n",
            "2020-08-04 00:20:44 | INFO | train_inner | epoch 039:    630 / 1015 loss=4.114, nll_loss=2.474, ppl=5.56, wps=11818.5, ups=3.29, wpb=3587.9, bsz=147.7, num_updates=39200, lr=0.000319438, gnorm=1.246, train_wall=30, wall=12267\n",
            "2020-08-04 00:21:15 | INFO | train_inner | epoch 039:    730 / 1015 loss=4.07, nll_loss=2.425, ppl=5.37, wps=11787.1, ups=3.27, wpb=3602.3, bsz=162.6, num_updates=39300, lr=0.000319032, gnorm=1.237, train_wall=30, wall=12297\n",
            "2020-08-04 00:21:45 | INFO | train_inner | epoch 039:    830 / 1015 loss=4.138, nll_loss=2.5, ppl=5.66, wps=11680.4, ups=3.32, wpb=3520, bsz=144.1, num_updates=39400, lr=0.000318626, gnorm=1.323, train_wall=30, wall=12327\n",
            "2020-08-04 00:22:15 | INFO | train_inner | epoch 039:    930 / 1015 loss=4.148, nll_loss=2.513, ppl=5.71, wps=11644.5, ups=3.31, wpb=3517.6, bsz=143.6, num_updates=39500, lr=0.000318223, gnorm=1.302, train_wall=30, wall=12357\n",
            "2020-08-04 00:22:41 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:22:47 | INFO | valid | epoch 039 | valid on 'valid' subset | loss 4.308 | nll_loss 2.618 | ppl 6.14 | wps 27205.9 | wpb 2866.6 | bsz 127.8 | num_updates 39585 | best_loss 4.308\n",
            "2020-08-04 00:22:47 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:22:50 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint39.pt (epoch 39 @ 39585 updates, score 4.308) (writing took 2.5430617999991227 seconds)\n",
            "2020-08-04 00:22:50 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:22:50 | INFO | fairseq_cli.train | end of epoch 39 (average epoch stats below)\n",
            "2020-08-04 00:22:50 | INFO | train | epoch 039 | loss 4.073 | nll_loss 2.428 | ppl 5.38 | wps 11377.7 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 39585 | lr 0.000317881 | gnorm 1.256 | train_wall 307 | wall 12392\n",
            "2020-08-04 00:22:50 | INFO | fairseq_cli.train | begin training epoch 39\n",
            "2020-08-04 00:22:55 | INFO | train_inner | epoch 040:     15 / 1015 loss=4.077, nll_loss=2.432, ppl=5.4, wps=9104.9, ups=2.55, wpb=3567.8, bsz=158.2, num_updates=39600, lr=0.000317821, gnorm=1.24, train_wall=30, wall=12397\n",
            "2020-08-04 00:23:25 | INFO | train_inner | epoch 040:    115 / 1015 loss=4.03, nll_loss=2.375, ppl=5.19, wps=11533.7, ups=3.3, wpb=3497.7, bsz=144.2, num_updates=39700, lr=0.00031742, gnorm=1.272, train_wall=30, wall=12427\n",
            "2020-08-04 00:23:55 | INFO | train_inner | epoch 040:    215 / 1015 loss=4.087, nll_loss=2.441, ppl=5.43, wps=11769.9, ups=3.34, wpb=3529.2, bsz=139.9, num_updates=39800, lr=0.000317021, gnorm=1.282, train_wall=30, wall=12457\n",
            "2020-08-04 00:24:25 | INFO | train_inner | epoch 040:    315 / 1015 loss=4.082, nll_loss=2.436, ppl=5.41, wps=11803.6, ups=3.31, wpb=3567.2, bsz=145, num_updates=39900, lr=0.000316624, gnorm=1.252, train_wall=30, wall=12487\n",
            "2020-08-04 00:24:55 | INFO | train_inner | epoch 040:    415 / 1015 loss=4.05, nll_loss=2.401, ppl=5.28, wps=11689.2, ups=3.3, wpb=3542.9, bsz=156.6, num_updates=40000, lr=0.000316228, gnorm=1.234, train_wall=30, wall=12518\n",
            "2020-08-04 00:25:26 | INFO | train_inner | epoch 040:    515 / 1015 loss=4.071, nll_loss=2.425, ppl=5.37, wps=11758.2, ups=3.28, wpb=3582, bsz=165.3, num_updates=40100, lr=0.000315833, gnorm=1.241, train_wall=30, wall=12548\n",
            "2020-08-04 00:25:56 | INFO | train_inner | epoch 040:    615 / 1015 loss=4.013, nll_loss=2.36, ppl=5.14, wps=11649.2, ups=3.27, wpb=3560, bsz=167.3, num_updates=40200, lr=0.00031544, gnorm=1.23, train_wall=30, wall=12579\n",
            "2020-08-04 00:26:26 | INFO | train_inner | epoch 040:    715 / 1015 loss=4.102, nll_loss=2.459, ppl=5.5, wps=11580.2, ups=3.34, wpb=3466.7, bsz=151.9, num_updates=40300, lr=0.000315049, gnorm=1.339, train_wall=30, wall=12608\n",
            "2020-08-04 00:26:57 | INFO | train_inner | epoch 040:    815 / 1015 loss=4.057, nll_loss=2.411, ppl=5.32, wps=11891.7, ups=3.28, wpb=3629.3, bsz=167.5, num_updates=40400, lr=0.000314658, gnorm=1.227, train_wall=30, wall=12639\n",
            "2020-08-04 00:27:27 | INFO | train_inner | epoch 040:    915 / 1015 loss=4.073, nll_loss=2.429, ppl=5.38, wps=11703.3, ups=3.28, wpb=3565.9, bsz=169.9, num_updates=40500, lr=0.00031427, gnorm=1.238, train_wall=30, wall=12669\n",
            "2020-08-04 00:27:58 | INFO | train_inner | epoch 040:   1015 / 1015 loss=4.056, nll_loss=2.41, ppl=5.32, wps=11674.3, ups=3.29, wpb=3550.2, bsz=171.8, num_updates=40600, lr=0.000313882, gnorm=1.231, train_wall=30, wall=12700\n",
            "2020-08-04 00:27:58 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:28:04 | INFO | valid | epoch 040 | valid on 'valid' subset | loss 4.307 | nll_loss 2.62 | ppl 6.15 | wps 27165.7 | wpb 2866.6 | bsz 127.8 | num_updates 40600 | best_loss 4.307\n",
            "2020-08-04 00:28:04 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:28:06 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint40.pt (epoch 40 @ 40600 updates, score 4.307) (writing took 2.59125658999983 seconds)\n",
            "2020-08-04 00:28:06 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:28:06 | INFO | fairseq_cli.train | end of epoch 40 (average epoch stats below)\n",
            "2020-08-04 00:28:06 | INFO | train | epoch 040 | loss 4.061 | nll_loss 2.414 | ppl 5.33 | wps 11380.1 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 40600 | lr 0.000313882 | gnorm 1.254 | train_wall 307 | wall 12709\n",
            "2020-08-04 00:28:06 | INFO | fairseq_cli.train | begin training epoch 40\n",
            "2020-08-04 00:28:37 | INFO | train_inner | epoch 041:    100 / 1015 loss=3.98, nll_loss=2.322, ppl=5, wps=9187.3, ups=2.54, wpb=3618.9, bsz=159.5, num_updates=40700, lr=0.000313497, gnorm=1.217, train_wall=31, wall=12739\n",
            "2020-08-04 00:29:08 | INFO | train_inner | epoch 041:    200 / 1015 loss=4.054, nll_loss=2.404, ppl=5.29, wps=11581.1, ups=3.29, wpb=3522.1, bsz=146.6, num_updates=40800, lr=0.000313112, gnorm=1.272, train_wall=30, wall=12770\n",
            "2020-08-04 00:29:38 | INFO | train_inner | epoch 041:    300 / 1015 loss=4.011, nll_loss=2.357, ppl=5.12, wps=11718.7, ups=3.3, wpb=3553.6, bsz=171.9, num_updates=40900, lr=0.000312729, gnorm=1.251, train_wall=30, wall=12800\n",
            "2020-08-04 00:30:08 | INFO | train_inner | epoch 041:    400 / 1015 loss=4.004, nll_loss=2.35, ppl=5.1, wps=11796.4, ups=3.3, wpb=3578.7, bsz=168.4, num_updates=41000, lr=0.000312348, gnorm=1.236, train_wall=30, wall=12830\n",
            "2020-08-04 00:30:39 | INFO | train_inner | epoch 041:    500 / 1015 loss=4.043, nll_loss=2.394, ppl=5.26, wps=11719.8, ups=3.29, wpb=3564.3, bsz=160.8, num_updates=41100, lr=0.000311967, gnorm=1.257, train_wall=30, wall=12861\n",
            "2020-08-04 00:31:09 | INFO | train_inner | epoch 041:    600 / 1015 loss=4.032, nll_loss=2.382, ppl=5.21, wps=11579.7, ups=3.34, wpb=3470.9, bsz=170.2, num_updates=41200, lr=0.000311588, gnorm=1.31, train_wall=30, wall=12891\n",
            "2020-08-04 00:31:39 | INFO | train_inner | epoch 041:    700 / 1015 loss=4.079, nll_loss=2.435, ppl=5.41, wps=11660.5, ups=3.28, wpb=3550.2, bsz=156.6, num_updates=41300, lr=0.000311211, gnorm=1.255, train_wall=30, wall=12921\n",
            "2020-08-04 00:32:09 | INFO | train_inner | epoch 041:    800 / 1015 loss=4.106, nll_loss=2.463, ppl=5.51, wps=11550.9, ups=3.32, wpb=3480.6, bsz=130.6, num_updates=41400, lr=0.000310835, gnorm=1.295, train_wall=30, wall=12951\n",
            "2020-08-04 00:32:40 | INFO | train_inner | epoch 041:    900 / 1015 loss=4.121, nll_loss=2.483, ppl=5.59, wps=11837.4, ups=3.3, wpb=3592.1, bsz=153.7, num_updates=41500, lr=0.00031046, gnorm=1.276, train_wall=30, wall=12982\n",
            "2020-08-04 00:33:10 | INFO | train_inner | epoch 041:   1000 / 1015 loss=4.109, nll_loss=2.469, ppl=5.54, wps=11659, ups=3.29, wpb=3547.3, bsz=150.6, num_updates=41600, lr=0.000310087, gnorm=1.277, train_wall=30, wall=13012\n",
            "2020-08-04 00:33:15 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:33:21 | INFO | valid | epoch 041 | valid on 'valid' subset | loss 4.312 | nll_loss 2.618 | ppl 6.14 | wps 27121.9 | wpb 2866.6 | bsz 127.8 | num_updates 41615 | best_loss 4.307\n",
            "2020-08-04 00:33:21 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:33:22 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint41.pt (epoch 41 @ 41615 updates, score 4.312) (writing took 1.5105306679997739 seconds)\n",
            "2020-08-04 00:33:22 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:33:22 | INFO | fairseq_cli.train | end of epoch 41 (average epoch stats below)\n",
            "2020-08-04 00:33:22 | INFO | train | epoch 041 | loss 4.052 | nll_loss 2.404 | ppl 5.29 | wps 11414.1 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 41615 | lr 0.000310031 | gnorm 1.263 | train_wall 307 | wall 13024\n",
            "2020-08-04 00:33:22 | INFO | fairseq_cli.train | begin training epoch 41\n",
            "2020-08-04 00:33:48 | INFO | train_inner | epoch 042:     85 / 1015 loss=3.99, nll_loss=2.332, ppl=5.04, wps=9496.5, ups=2.63, wpb=3617, bsz=155, num_updates=41700, lr=0.000309715, gnorm=1.219, train_wall=30, wall=13050\n",
            "2020-08-04 00:34:18 | INFO | train_inner | epoch 042:    185 / 1015 loss=4.007, nll_loss=2.352, ppl=5.11, wps=11799, ups=3.29, wpb=3581.2, bsz=165.6, num_updates=41800, lr=0.000309344, gnorm=1.229, train_wall=30, wall=13081\n",
            "2020-08-04 00:34:49 | INFO | train_inner | epoch 042:    285 / 1015 loss=4.02, nll_loss=2.366, ppl=5.16, wps=11758.8, ups=3.28, wpb=3580.3, bsz=158.6, num_updates=41900, lr=0.000308975, gnorm=1.241, train_wall=30, wall=13111\n",
            "2020-08-04 00:35:19 | INFO | train_inner | epoch 042:    385 / 1015 loss=4.05, nll_loss=2.4, ppl=5.28, wps=11714.2, ups=3.29, wpb=3556.2, bsz=152.1, num_updates=42000, lr=0.000308607, gnorm=1.26, train_wall=30, wall=13141\n",
            "2020-08-04 00:35:49 | INFO | train_inner | epoch 042:    485 / 1015 loss=4.07, nll_loss=2.424, ppl=5.37, wps=11727.8, ups=3.32, wpb=3528.9, bsz=147.9, num_updates=42100, lr=0.00030824, gnorm=1.294, train_wall=30, wall=13171\n",
            "2020-08-04 00:36:20 | INFO | train_inner | epoch 042:    585 / 1015 loss=4.069, nll_loss=2.424, ppl=5.37, wps=11662.7, ups=3.31, wpb=3525.6, bsz=161.7, num_updates=42200, lr=0.000307875, gnorm=1.269, train_wall=30, wall=13202\n",
            "2020-08-04 00:36:50 | INFO | train_inner | epoch 042:    685 / 1015 loss=4.015, nll_loss=2.362, ppl=5.14, wps=11740.8, ups=3.29, wpb=3569.8, bsz=167.8, num_updates=42300, lr=0.00030751, gnorm=1.26, train_wall=30, wall=13232\n",
            "2020-08-04 00:37:20 | INFO | train_inner | epoch 042:    785 / 1015 loss=4.024, nll_loss=2.374, ppl=5.18, wps=11744.9, ups=3.29, wpb=3564.6, bsz=176.9, num_updates=42400, lr=0.000307148, gnorm=1.261, train_wall=30, wall=13262\n",
            "2020-08-04 00:37:51 | INFO | train_inner | epoch 042:    885 / 1015 loss=4.047, nll_loss=2.399, ppl=5.27, wps=11534, ups=3.3, wpb=3498.4, bsz=157.5, num_updates=42500, lr=0.000306786, gnorm=1.264, train_wall=30, wall=13293\n",
            "2020-08-04 00:38:21 | INFO | train_inner | epoch 042:    985 / 1015 loss=4.095, nll_loss=2.452, ppl=5.47, wps=11663.9, ups=3.31, wpb=3528.3, bsz=148.7, num_updates=42600, lr=0.000306426, gnorm=1.317, train_wall=30, wall=13323\n",
            "2020-08-04 00:38:30 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:38:36 | INFO | valid | epoch 042 | valid on 'valid' subset | loss 4.317 | nll_loss 2.628 | ppl 6.18 | wps 27220.6 | wpb 2866.6 | bsz 127.8 | num_updates 42630 | best_loss 4.307\n",
            "2020-08-04 00:38:36 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:38:38 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint42.pt (epoch 42 @ 42630 updates, score 4.317) (writing took 1.6268017690017587 seconds)\n",
            "2020-08-04 00:38:38 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:38:38 | INFO | fairseq_cli.train | end of epoch 42 (average epoch stats below)\n",
            "2020-08-04 00:38:38 | INFO | train | epoch 042 | loss 4.043 | nll_loss 2.393 | ppl 5.25 | wps 11421.9 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 42630 | lr 0.000306318 | gnorm 1.266 | train_wall 307 | wall 13340\n",
            "2020-08-04 00:38:38 | INFO | fairseq_cli.train | begin training epoch 42\n",
            "2020-08-04 00:38:59 | INFO | train_inner | epoch 043:     70 / 1015 loss=3.992, nll_loss=2.337, ppl=5.05, wps=9241.6, ups=2.63, wpb=3512.7, bsz=179.9, num_updates=42700, lr=0.000306067, gnorm=1.289, train_wall=30, wall=13361\n",
            "2020-08-04 00:39:29 | INFO | train_inner | epoch 043:    170 / 1015 loss=3.992, nll_loss=2.334, ppl=5.04, wps=11715.8, ups=3.29, wpb=3559.1, bsz=154.5, num_updates=42800, lr=0.000305709, gnorm=1.251, train_wall=30, wall=13391\n",
            "2020-08-04 00:40:00 | INFO | train_inner | epoch 043:    270 / 1015 loss=4.014, nll_loss=2.359, ppl=5.13, wps=11734, ups=3.29, wpb=3563.5, bsz=151.7, num_updates=42900, lr=0.000305352, gnorm=1.266, train_wall=30, wall=13422\n",
            "2020-08-04 00:40:30 | INFO | train_inner | epoch 043:    370 / 1015 loss=4.028, nll_loss=2.375, ppl=5.19, wps=11715.7, ups=3.31, wpb=3539.5, bsz=156.8, num_updates=43000, lr=0.000304997, gnorm=1.278, train_wall=30, wall=13452\n",
            "2020-08-04 00:41:00 | INFO | train_inner | epoch 043:    470 / 1015 loss=4.044, nll_loss=2.394, ppl=5.25, wps=11691.4, ups=3.33, wpb=3509.6, bsz=156.6, num_updates=43100, lr=0.000304643, gnorm=1.299, train_wall=30, wall=13482\n",
            "2020-08-04 00:41:30 | INFO | train_inner | epoch 043:    570 / 1015 loss=4.054, nll_loss=2.405, ppl=5.3, wps=11663.5, ups=3.3, wpb=3530.6, bsz=151.9, num_updates=43200, lr=0.00030429, gnorm=1.303, train_wall=30, wall=13512\n",
            "2020-08-04 00:42:00 | INFO | train_inner | epoch 043:    670 / 1015 loss=4.073, nll_loss=2.426, ppl=5.38, wps=11732.1, ups=3.29, wpb=3561.5, bsz=151.8, num_updates=43300, lr=0.000303939, gnorm=1.265, train_wall=30, wall=13543\n",
            "2020-08-04 00:42:31 | INFO | train_inner | epoch 043:    770 / 1015 loss=4.053, nll_loss=2.406, ppl=5.3, wps=11886.7, ups=3.3, wpb=3596.9, bsz=164, num_updates=43400, lr=0.000303588, gnorm=1.256, train_wall=30, wall=13573\n",
            "2020-08-04 00:43:01 | INFO | train_inner | epoch 043:    870 / 1015 loss=4.076, nll_loss=2.43, ppl=5.39, wps=11653.2, ups=3.28, wpb=3550.6, bsz=145.4, num_updates=43500, lr=0.000303239, gnorm=1.298, train_wall=30, wall=13603\n",
            "2020-08-04 00:43:32 | INFO | train_inner | epoch 043:    970 / 1015 loss=4.039, nll_loss=2.39, ppl=5.24, wps=11727.3, ups=3.29, wpb=3559.9, bsz=162.2, num_updates=43600, lr=0.000302891, gnorm=1.261, train_wall=30, wall=13634\n",
            "2020-08-04 00:43:45 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:43:51 | INFO | valid | epoch 043 | valid on 'valid' subset | loss 4.324 | nll_loss 2.624 | ppl 6.16 | wps 27150.9 | wpb 2866.6 | bsz 127.8 | num_updates 43645 | best_loss 4.307\n",
            "2020-08-04 00:43:51 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:43:53 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint43.pt (epoch 43 @ 43645 updates, score 4.324) (writing took 1.6400072659998841 seconds)\n",
            "2020-08-04 00:43:53 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:43:53 | INFO | fairseq_cli.train | end of epoch 43 (average epoch stats below)\n",
            "2020-08-04 00:43:53 | INFO | train | epoch 043 | loss 4.034 | nll_loss 2.383 | ppl 5.21 | wps 11423.6 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 43645 | lr 0.000302735 | gnorm 1.276 | train_wall 307 | wall 13655\n",
            "2020-08-04 00:43:53 | INFO | fairseq_cli.train | begin training epoch 43\n",
            "2020-08-04 00:44:10 | INFO | train_inner | epoch 044:     55 / 1015 loss=4.015, nll_loss=2.361, ppl=5.14, wps=9204.1, ups=2.62, wpb=3510.3, bsz=152.2, num_updates=43700, lr=0.000302545, gnorm=1.305, train_wall=30, wall=13672\n",
            "2020-08-04 00:44:40 | INFO | train_inner | epoch 044:    155 / 1015 loss=3.95, nll_loss=2.287, ppl=4.88, wps=11709.5, ups=3.28, wpb=3572.5, bsz=153.9, num_updates=43800, lr=0.000302199, gnorm=1.253, train_wall=30, wall=13702\n",
            "2020-08-04 00:45:11 | INFO | train_inner | epoch 044:    255 / 1015 loss=3.993, nll_loss=2.337, ppl=5.05, wps=11618.4, ups=3.3, wpb=3524.7, bsz=170, num_updates=43900, lr=0.000301855, gnorm=1.282, train_wall=30, wall=13733\n",
            "2020-08-04 00:45:41 | INFO | train_inner | epoch 044:    355 / 1015 loss=3.994, nll_loss=2.337, ppl=5.05, wps=11565.4, ups=3.31, wpb=3494.5, bsz=160.6, num_updates=44000, lr=0.000301511, gnorm=1.299, train_wall=30, wall=13763\n",
            "2020-08-04 00:46:11 | INFO | train_inner | epoch 044:    455 / 1015 loss=4.004, nll_loss=2.348, ppl=5.09, wps=11738.5, ups=3.29, wpb=3570.9, bsz=158.3, num_updates=44100, lr=0.000301169, gnorm=1.259, train_wall=30, wall=13793\n",
            "2020-08-04 00:46:42 | INFO | train_inner | epoch 044:    555 / 1015 loss=4.04, nll_loss=2.39, ppl=5.24, wps=11760.7, ups=3.3, wpb=3565.7, bsz=160, num_updates=44200, lr=0.000300828, gnorm=1.277, train_wall=30, wall=13824\n",
            "2020-08-04 00:47:12 | INFO | train_inner | epoch 044:    655 / 1015 loss=4.072, nll_loss=2.425, ppl=5.37, wps=11592.5, ups=3.32, wpb=3496.6, bsz=142.2, num_updates=44300, lr=0.000300489, gnorm=1.33, train_wall=30, wall=13854\n",
            "2020-08-04 00:47:42 | INFO | train_inner | epoch 044:    755 / 1015 loss=4.037, nll_loss=2.387, ppl=5.23, wps=11780.7, ups=3.33, wpb=3540.2, bsz=161.6, num_updates=44400, lr=0.00030015, gnorm=1.316, train_wall=30, wall=13884\n",
            "2020-08-04 00:48:12 | INFO | train_inner | epoch 044:    855 / 1015 loss=4.111, nll_loss=2.471, ppl=5.54, wps=11671.8, ups=3.33, wpb=3509.5, bsz=142.9, num_updates=44500, lr=0.000299813, gnorm=1.319, train_wall=30, wall=13914\n",
            "2020-08-04 00:48:42 | INFO | train_inner | epoch 044:    955 / 1015 loss=4.038, nll_loss=2.389, ppl=5.24, wps=11846.2, ups=3.27, wpb=3625.9, bsz=173.9, num_updates=44600, lr=0.000299476, gnorm=1.281, train_wall=31, wall=13945\n",
            "2020-08-04 00:49:01 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:49:07 | INFO | valid | epoch 044 | valid on 'valid' subset | loss 4.317 | nll_loss 2.624 | ppl 6.16 | wps 27203 | wpb 2866.6 | bsz 127.8 | num_updates 44660 | best_loss 4.307\n",
            "2020-08-04 00:49:07 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:49:08 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint44.pt (epoch 44 @ 44660 updates, score 4.317) (writing took 1.6292097959994862 seconds)\n",
            "2020-08-04 00:49:08 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:49:08 | INFO | fairseq_cli.train | end of epoch 44 (average epoch stats below)\n",
            "2020-08-04 00:49:08 | INFO | train | epoch 044 | loss 4.026 | nll_loss 2.374 | ppl 5.18 | wps 11425.9 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 44660 | lr 0.000299275 | gnorm 1.287 | train_wall 307 | wall 13971\n",
            "2020-08-04 00:49:08 | INFO | fairseq_cli.train | begin training epoch 44\n",
            "2020-08-04 00:49:21 | INFO | train_inner | epoch 045:     40 / 1015 loss=3.988, nll_loss=2.332, ppl=5.04, wps=9593, ups=2.61, wpb=3679.9, bsz=166.9, num_updates=44700, lr=0.000299141, gnorm=1.214, train_wall=30, wall=13983\n",
            "2020-08-04 00:49:51 | INFO | train_inner | epoch 045:    140 / 1015 loss=3.939, nll_loss=2.275, ppl=4.84, wps=11592.9, ups=3.31, wpb=3501.2, bsz=166, num_updates=44800, lr=0.000298807, gnorm=1.289, train_wall=30, wall=14013\n",
            "2020-08-04 00:50:22 | INFO | train_inner | epoch 045:    240 / 1015 loss=3.99, nll_loss=2.331, ppl=5.03, wps=11586.6, ups=3.26, wpb=3553.3, bsz=145.8, num_updates=44900, lr=0.000298474, gnorm=1.316, train_wall=31, wall=14044\n",
            "2020-08-04 00:50:52 | INFO | train_inner | epoch 045:    340 / 1015 loss=3.986, nll_loss=2.328, ppl=5.02, wps=11885.1, ups=3.29, wpb=3612.6, bsz=163.8, num_updates=45000, lr=0.000298142, gnorm=1.243, train_wall=30, wall=14074\n",
            "2020-08-04 00:51:23 | INFO | train_inner | epoch 045:    440 / 1015 loss=4, nll_loss=2.345, ppl=5.08, wps=11764.9, ups=3.28, wpb=3588.2, bsz=162.8, num_updates=45100, lr=0.000297812, gnorm=1.266, train_wall=30, wall=14105\n",
            "2020-08-04 00:51:53 | INFO | train_inner | epoch 045:    540 / 1015 loss=4.034, nll_loss=2.385, ppl=5.22, wps=11605.6, ups=3.32, wpb=3496.4, bsz=167.3, num_updates=45200, lr=0.000297482, gnorm=1.306, train_wall=30, wall=14135\n",
            "2020-08-04 00:52:23 | INFO | train_inner | epoch 045:    640 / 1015 loss=4.007, nll_loss=2.353, ppl=5.11, wps=11670.4, ups=3.31, wpb=3529.9, bsz=163.9, num_updates=45300, lr=0.000297154, gnorm=1.306, train_wall=30, wall=14165\n",
            "2020-08-04 00:52:53 | INFO | train_inner | epoch 045:    740 / 1015 loss=4.076, nll_loss=2.432, ppl=5.4, wps=11718.3, ups=3.31, wpb=3540.9, bsz=150.2, num_updates=45400, lr=0.000296826, gnorm=1.329, train_wall=30, wall=14195\n",
            "2020-08-04 00:53:23 | INFO | train_inner | epoch 045:    840 / 1015 loss=4.069, nll_loss=2.424, ppl=5.37, wps=11674.2, ups=3.29, wpb=3544.2, bsz=152.9, num_updates=45500, lr=0.0002965, gnorm=1.301, train_wall=30, wall=14226\n",
            "2020-08-04 00:53:54 | INFO | train_inner | epoch 045:    940 / 1015 loss=4.079, nll_loss=2.434, ppl=5.41, wps=11678.2, ups=3.32, wpb=3516.6, bsz=139.8, num_updates=45600, lr=0.000296174, gnorm=1.304, train_wall=30, wall=14256\n",
            "2020-08-04 00:54:16 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:54:22 | INFO | valid | epoch 045 | valid on 'valid' subset | loss 4.31 | nll_loss 2.617 | ppl 6.13 | wps 27095.6 | wpb 2866.6 | bsz 127.8 | num_updates 45675 | best_loss 4.307\n",
            "2020-08-04 00:54:22 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:54:24 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint45.pt (epoch 45 @ 45675 updates, score 4.31) (writing took 1.6271764249995613 seconds)\n",
            "2020-08-04 00:54:24 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:54:24 | INFO | fairseq_cli.train | end of epoch 45 (average epoch stats below)\n",
            "2020-08-04 00:54:24 | INFO | train | epoch 045 | loss 4.017 | nll_loss 2.364 | ppl 5.15 | wps 11419.6 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 45675 | lr 0.000295931 | gnorm 1.291 | train_wall 307 | wall 14286\n",
            "2020-08-04 00:54:24 | INFO | fairseq_cli.train | begin training epoch 45\n",
            "2020-08-04 00:54:32 | INFO | train_inner | epoch 046:     25 / 1015 loss=4.022, nll_loss=2.371, ppl=5.17, wps=9371.6, ups=2.62, wpb=3574.5, bsz=161.4, num_updates=45700, lr=0.00029585, gnorm=1.275, train_wall=30, wall=14294\n",
            "2020-08-04 00:55:02 | INFO | train_inner | epoch 046:    125 / 1015 loss=3.92, nll_loss=2.252, ppl=4.76, wps=11611.3, ups=3.3, wpb=3523.8, bsz=159.6, num_updates=45800, lr=0.000295527, gnorm=1.269, train_wall=30, wall=14324\n",
            "2020-08-04 00:55:33 | INFO | train_inner | epoch 046:    225 / 1015 loss=3.983, nll_loss=2.324, ppl=5.01, wps=11646.6, ups=3.28, wpb=3551.8, bsz=152.6, num_updates=45900, lr=0.000295205, gnorm=1.295, train_wall=30, wall=14355\n",
            "2020-08-04 00:56:03 | INFO | train_inner | epoch 046:    325 / 1015 loss=4.034, nll_loss=2.382, ppl=5.21, wps=11629, ups=3.31, wpb=3509.9, bsz=138.1, num_updates=46000, lr=0.000294884, gnorm=1.35, train_wall=30, wall=14385\n",
            "2020-08-04 00:56:33 | INFO | train_inner | epoch 046:    425 / 1015 loss=4.008, nll_loss=2.353, ppl=5.11, wps=11950.7, ups=3.3, wpb=3621.1, bsz=155.8, num_updates=46100, lr=0.000294564, gnorm=1.264, train_wall=30, wall=14415\n",
            "2020-08-04 00:57:03 | INFO | train_inner | epoch 046:    525 / 1015 loss=4.024, nll_loss=2.371, ppl=5.17, wps=11705.7, ups=3.3, wpb=3542.1, bsz=155.4, num_updates=46200, lr=0.000294245, gnorm=1.293, train_wall=30, wall=14445\n",
            "2020-08-04 00:57:34 | INFO | train_inner | epoch 046:    625 / 1015 loss=4.009, nll_loss=2.357, ppl=5.12, wps=11725.7, ups=3.29, wpb=3563.8, bsz=159.6, num_updates=46300, lr=0.000293927, gnorm=1.282, train_wall=30, wall=14476\n",
            "2020-08-04 00:58:04 | INFO | train_inner | epoch 046:    725 / 1015 loss=3.999, nll_loss=2.345, ppl=5.08, wps=11590.6, ups=3.33, wpb=3483.6, bsz=175.4, num_updates=46400, lr=0.00029361, gnorm=1.325, train_wall=30, wall=14506\n",
            "2020-08-04 00:58:34 | INFO | train_inner | epoch 046:    825 / 1015 loss=4.026, nll_loss=2.375, ppl=5.19, wps=11885.5, ups=3.29, wpb=3614.1, bsz=165.8, num_updates=46500, lr=0.000293294, gnorm=1.273, train_wall=30, wall=14536\n",
            "2020-08-04 00:59:05 | INFO | train_inner | epoch 046:    925 / 1015 loss=4.055, nll_loss=2.408, ppl=5.31, wps=11762.7, ups=3.27, wpb=3593.1, bsz=150.6, num_updates=46600, lr=0.000292979, gnorm=1.318, train_wall=30, wall=14567\n",
            "2020-08-04 00:59:32 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 00:59:38 | INFO | valid | epoch 046 | valid on 'valid' subset | loss 4.33 | nll_loss 2.632 | ppl 6.2 | wps 27153.9 | wpb 2866.6 | bsz 127.8 | num_updates 46690 | best_loss 4.307\n",
            "2020-08-04 00:59:38 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 00:59:40 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint46.pt (epoch 46 @ 46690 updates, score 4.33) (writing took 1.5291647110007034 seconds)\n",
            "2020-08-04 00:59:40 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 00:59:40 | INFO | fairseq_cli.train | end of epoch 46 (average epoch stats below)\n",
            "2020-08-04 00:59:40 | INFO | train | epoch 046 | loss 4.009 | nll_loss 2.355 | ppl 5.12 | wps 11419.2 | ups 3.22 | wpb 3550.1 | bsz 157.9 | num_updates 46690 | lr 0.000292697 | gnorm 1.3 | train_wall 307 | wall 14602\n",
            "2020-08-04 00:59:40 | INFO | fairseq_cli.train | begin training epoch 46\n",
            "2020-08-04 00:59:43 | INFO | train_inner | epoch 047:     10 / 1015 loss=4.037, nll_loss=2.387, ppl=5.23, wps=9185.8, ups=2.63, wpb=3489.6, bsz=164.2, num_updates=46700, lr=0.000292666, gnorm=1.344, train_wall=30, wall=14605\n",
            "2020-08-04 01:00:13 | INFO | train_inner | epoch 047:    110 / 1015 loss=3.965, nll_loss=2.304, ppl=4.94, wps=11668.4, ups=3.31, wpb=3524.8, bsz=146, num_updates=46800, lr=0.000292353, gnorm=1.323, train_wall=30, wall=14635\n",
            "2020-08-04 01:00:43 | INFO | train_inner | epoch 047:    210 / 1015 loss=3.962, nll_loss=2.301, ppl=4.93, wps=11844.1, ups=3.3, wpb=3588.4, bsz=166, num_updates=46900, lr=0.000292041, gnorm=1.262, train_wall=30, wall=14665\n",
            "2020-08-04 01:01:13 | INFO | train_inner | epoch 047:    310 / 1015 loss=3.99, nll_loss=2.334, ppl=5.04, wps=11785.3, ups=3.31, wpb=3565.5, bsz=162.4, num_updates=47000, lr=0.00029173, gnorm=1.276, train_wall=30, wall=14696\n",
            "2020-08-04 01:01:44 | INFO | train_inner | epoch 047:    410 / 1015 loss=3.963, nll_loss=2.303, ppl=4.93, wps=11562.5, ups=3.3, wpb=3504.6, bsz=161.7, num_updates=47100, lr=0.00029142, gnorm=1.302, train_wall=30, wall=14726\n",
            "2020-08-04 01:02:14 | INFO | train_inner | epoch 047:    510 / 1015 loss=3.961, nll_loss=2.301, ppl=4.93, wps=11608.6, ups=3.26, wpb=3556.2, bsz=163.4, num_updates=47200, lr=0.000291111, gnorm=1.324, train_wall=31, wall=14757\n",
            "2020-08-04 01:02:45 | INFO | train_inner | epoch 047:    610 / 1015 loss=3.981, nll_loss=2.323, ppl=5.01, wps=11744.2, ups=3.25, wpb=3616.7, bsz=162.8, num_updates=47300, lr=0.000290803, gnorm=1.283, train_wall=31, wall=14787\n",
            "2020-08-04 01:03:16 | INFO | train_inner | epoch 047:    710 / 1015 loss=4.001, nll_loss=2.346, ppl=5.08, wps=11608.2, ups=3.28, wpb=3544, bsz=162.5, num_updates=47400, lr=0.000290496, gnorm=1.284, train_wall=30, wall=14818\n",
            "2020-08-04 01:03:46 | INFO | train_inner | epoch 047:    810 / 1015 loss=4.052, nll_loss=2.404, ppl=5.29, wps=11464.4, ups=3.33, wpb=3438.6, bsz=158, num_updates=47500, lr=0.000290191, gnorm=1.388, train_wall=30, wall=14848\n",
            "2020-08-04 01:04:16 | INFO | train_inner | epoch 047:    910 / 1015 loss=4.088, nll_loss=2.445, ppl=5.45, wps=11778.6, ups=3.31, wpb=3553.2, bsz=145.5, num_updates=47600, lr=0.000289886, gnorm=1.354, train_wall=30, wall=14878\n",
            "2020-08-04 01:04:46 | INFO | train_inner | epoch 047:   1010 / 1015 loss=4.066, nll_loss=2.42, ppl=5.35, wps=11850.9, ups=3.28, wpb=3609.6, bsz=152.1, num_updates=47700, lr=0.000289581, gnorm=1.291, train_wall=30, wall=14908\n",
            "2020-08-04 01:04:48 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:04:54 | INFO | valid | epoch 047 | valid on 'valid' subset | loss 4.3 | nll_loss 2.608 | ppl 6.1 | wps 27065.5 | wpb 2866.6 | bsz 127.8 | num_updates 47705 | best_loss 4.3\n",
            "2020-08-04 01:04:54 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:04:57 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint47.pt (epoch 47 @ 47705 updates, score 4.3) (writing took 2.5649004820006667 seconds)\n",
            "2020-08-04 01:04:57 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:04:57 | INFO | fairseq_cli.train | end of epoch 47 (average epoch stats below)\n",
            "2020-08-04 01:04:57 | INFO | train | epoch 047 | loss 4.002 | nll_loss 2.347 | ppl 5.09 | wps 11366.5 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 47705 | lr 0.000289566 | gnorm 1.308 | train_wall 307 | wall 14919\n",
            "2020-08-04 01:04:57 | INFO | fairseq_cli.train | begin training epoch 47\n",
            "2020-08-04 01:05:25 | INFO | train_inner | epoch 048:     95 / 1015 loss=3.917, nll_loss=2.251, ppl=4.76, wps=9128.7, ups=2.57, wpb=3555.3, bsz=157.6, num_updates=47800, lr=0.000289278, gnorm=1.27, train_wall=30, wall=14947\n",
            "2020-08-04 01:05:56 | INFO | train_inner | epoch 048:    195 / 1015 loss=4.007, nll_loss=2.35, ppl=5.1, wps=11534.1, ups=3.29, wpb=3506, bsz=137.1, num_updates=47900, lr=0.000288976, gnorm=1.392, train_wall=30, wall=14978\n",
            "2020-08-04 01:06:26 | INFO | train_inner | epoch 048:    295 / 1015 loss=3.982, nll_loss=2.324, ppl=5.01, wps=11529.6, ups=3.3, wpb=3494.1, bsz=158.6, num_updates=48000, lr=0.000288675, gnorm=1.323, train_wall=30, wall=15008\n",
            "2020-08-04 01:06:56 | INFO | train_inner | epoch 048:    395 / 1015 loss=3.992, nll_loss=2.335, ppl=5.05, wps=11746, ups=3.29, wpb=3573.6, bsz=145.9, num_updates=48100, lr=0.000288375, gnorm=1.307, train_wall=30, wall=15039\n",
            "2020-08-04 01:07:27 | INFO | train_inner | epoch 048:    495 / 1015 loss=3.999, nll_loss=2.344, ppl=5.08, wps=11863.6, ups=3.3, wpb=3592.9, bsz=170.7, num_updates=48200, lr=0.000288076, gnorm=1.318, train_wall=30, wall=15069\n",
            "2020-08-04 01:07:57 | INFO | train_inner | epoch 048:    595 / 1015 loss=4.023, nll_loss=2.371, ppl=5.17, wps=11659.1, ups=3.29, wpb=3543.9, bsz=154.3, num_updates=48300, lr=0.000287777, gnorm=1.284, train_wall=30, wall=15099\n",
            "2020-08-04 01:08:27 | INFO | train_inner | epoch 048:    695 / 1015 loss=4.038, nll_loss=2.387, ppl=5.23, wps=11614, ups=3.3, wpb=3523.4, bsz=145, num_updates=48400, lr=0.00028748, gnorm=1.34, train_wall=30, wall=15130\n",
            "2020-08-04 01:08:58 | INFO | train_inner | epoch 048:    795 / 1015 loss=3.973, nll_loss=2.317, ppl=4.98, wps=11649.2, ups=3.29, wpb=3540.9, bsz=178.6, num_updates=48500, lr=0.000287183, gnorm=1.329, train_wall=30, wall=15160\n",
            "2020-08-04 01:09:28 | INFO | train_inner | epoch 048:    895 / 1015 loss=3.955, nll_loss=2.296, ppl=4.91, wps=11650.6, ups=3.28, wpb=3547.3, bsz=177.8, num_updates=48600, lr=0.000286888, gnorm=1.268, train_wall=30, wall=15190\n",
            "2020-08-04 01:09:59 | INFO | train_inner | epoch 048:    995 / 1015 loss=4.028, nll_loss=2.378, ppl=5.2, wps=11876.2, ups=3.28, wpb=3625.2, bsz=160.1, num_updates=48700, lr=0.000286593, gnorm=1.297, train_wall=30, wall=15221\n",
            "2020-08-04 01:10:05 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:10:11 | INFO | valid | epoch 048 | valid on 'valid' subset | loss 4.293 | nll_loss 2.6 | ppl 6.06 | wps 27106.9 | wpb 2866.6 | bsz 127.8 | num_updates 48720 | best_loss 4.293\n",
            "2020-08-04 01:10:11 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:10:13 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint48.pt (epoch 48 @ 48720 updates, score 4.293) (writing took 2.5424217239997233 seconds)\n",
            "2020-08-04 01:10:14 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:10:14 | INFO | fairseq_cli.train | end of epoch 48 (average epoch stats below)\n",
            "2020-08-04 01:10:14 | INFO | train | epoch 048 | loss 3.994 | nll_loss 2.338 | ppl 5.06 | wps 11368.7 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 48720 | lr 0.000286534 | gnorm 1.313 | train_wall 307 | wall 15236\n",
            "2020-08-04 01:10:14 | INFO | fairseq_cli.train | begin training epoch 48\n",
            "2020-08-04 01:10:38 | INFO | train_inner | epoch 049:     80 / 1015 loss=3.999, nll_loss=2.342, ppl=5.07, wps=9000.2, ups=2.55, wpb=3528.6, bsz=130.5, num_updates=48800, lr=0.000286299, gnorm=1.321, train_wall=30, wall=15260\n",
            "2020-08-04 01:11:08 | INFO | train_inner | epoch 049:    180 / 1015 loss=3.927, nll_loss=2.262, ppl=4.8, wps=11711.1, ups=3.28, wpb=3567, bsz=172.4, num_updates=48900, lr=0.000286006, gnorm=1.268, train_wall=30, wall=15291\n",
            "2020-08-04 01:11:39 | INFO | train_inner | epoch 049:    280 / 1015 loss=3.983, nll_loss=2.324, ppl=5.01, wps=11778, ups=3.28, wpb=3588.3, bsz=149.7, num_updates=49000, lr=0.000285714, gnorm=1.31, train_wall=30, wall=15321\n",
            "2020-08-04 01:12:09 | INFO | train_inner | epoch 049:    380 / 1015 loss=3.996, nll_loss=2.34, ppl=5.06, wps=11835.7, ups=3.29, wpb=3602.5, bsz=157, num_updates=49100, lr=0.000285423, gnorm=1.307, train_wall=30, wall=15352\n",
            "2020-08-04 01:12:40 | INFO | train_inner | epoch 049:    480 / 1015 loss=3.936, nll_loss=2.271, ppl=4.83, wps=11731, ups=3.3, wpb=3555.4, bsz=165.4, num_updates=49200, lr=0.000285133, gnorm=1.3, train_wall=30, wall=15382\n",
            "2020-08-04 01:13:10 | INFO | train_inner | epoch 049:    580 / 1015 loss=4.023, nll_loss=2.371, ppl=5.17, wps=11639.2, ups=3.29, wpb=3533.2, bsz=155.7, num_updates=49300, lr=0.000284844, gnorm=1.321, train_wall=30, wall=15412\n",
            "2020-08-04 01:13:40 | INFO | train_inner | epoch 049:    680 / 1015 loss=4.041, nll_loss=2.39, ppl=5.24, wps=11630.9, ups=3.31, wpb=3511.6, bsz=142.9, num_updates=49400, lr=0.000284555, gnorm=1.38, train_wall=30, wall=15442\n",
            "2020-08-04 01:14:11 | INFO | train_inner | epoch 049:    780 / 1015 loss=3.995, nll_loss=2.34, ppl=5.06, wps=11755.1, ups=3.3, wpb=3559.8, bsz=163.3, num_updates=49500, lr=0.000284268, gnorm=1.332, train_wall=30, wall=15473\n",
            "2020-08-04 01:14:41 | INFO | train_inner | epoch 049:    880 / 1015 loss=4.023, nll_loss=2.371, ppl=5.17, wps=11542.3, ups=3.27, wpb=3526.8, bsz=151.8, num_updates=49600, lr=0.000283981, gnorm=1.351, train_wall=30, wall=15503\n",
            "2020-08-04 01:15:12 | INFO | train_inner | epoch 049:    980 / 1015 loss=3.967, nll_loss=2.31, ppl=4.96, wps=11653, ups=3.28, wpb=3550.5, bsz=177.8, num_updates=49700, lr=0.000283695, gnorm=1.284, train_wall=30, wall=15534\n",
            "2020-08-04 01:15:22 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:15:28 | INFO | valid | epoch 049 | valid on 'valid' subset | loss 4.289 | nll_loss 2.592 | ppl 6.03 | wps 27025.6 | wpb 2866.6 | bsz 127.8 | num_updates 49735 | best_loss 4.289\n",
            "2020-08-04 01:15:28 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:15:31 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint49.pt (epoch 49 @ 49735 updates, score 4.289) (writing took 2.5350852939991455 seconds)\n",
            "2020-08-04 01:15:31 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:15:31 | INFO | fairseq_cli.train | end of epoch 49 (average epoch stats below)\n",
            "2020-08-04 01:15:31 | INFO | train | epoch 049 | loss 3.986 | nll_loss 2.329 | ppl 5.03 | wps 11353.7 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 49735 | lr 0.000283595 | gnorm 1.317 | train_wall 308 | wall 15553\n",
            "2020-08-04 01:15:31 | INFO | fairseq_cli.train | begin training epoch 49\n",
            "2020-08-04 01:15:51 | INFO | train_inner | epoch 050:     65 / 1015 loss=3.973, nll_loss=2.315, ppl=4.97, wps=9021.7, ups=2.56, wpb=3529.6, bsz=154.1, num_updates=49800, lr=0.00028341, gnorm=1.313, train_wall=30, wall=15573\n",
            "2020-08-04 01:16:21 | INFO | train_inner | epoch 050:    165 / 1015 loss=3.951, nll_loss=2.287, ppl=4.88, wps=11468.1, ups=3.25, wpb=3523.4, bsz=146.9, num_updates=49900, lr=0.000283126, gnorm=1.353, train_wall=31, wall=15604\n",
            "2020-08-04 01:16:52 | INFO | train_inner | epoch 050:    265 / 1015 loss=3.957, nll_loss=2.295, ppl=4.91, wps=11671, ups=3.27, wpb=3566.3, bsz=157.3, num_updates=50000, lr=0.000282843, gnorm=1.311, train_wall=30, wall=15634\n",
            "2020-08-04 01:17:23 | INFO | train_inner | epoch 050:    365 / 1015 loss=3.994, nll_loss=2.336, ppl=5.05, wps=11617.7, ups=3.27, wpb=3551.1, bsz=135.2, num_updates=50100, lr=0.00028256, gnorm=1.331, train_wall=30, wall=15665\n",
            "2020-08-04 01:17:53 | INFO | train_inner | epoch 050:    465 / 1015 loss=3.953, nll_loss=2.292, ppl=4.9, wps=11619.3, ups=3.3, wpb=3526.1, bsz=165.8, num_updates=50200, lr=0.000282279, gnorm=1.338, train_wall=30, wall=15695\n",
            "2020-08-04 01:18:23 | INFO | train_inner | epoch 050:    565 / 1015 loss=3.944, nll_loss=2.281, ppl=4.86, wps=11881.8, ups=3.27, wpb=3631.4, bsz=174.6, num_updates=50300, lr=0.000281998, gnorm=1.271, train_wall=30, wall=15726\n",
            "2020-08-04 01:18:54 | INFO | train_inner | epoch 050:    665 / 1015 loss=3.984, nll_loss=2.327, ppl=5.02, wps=11666.1, ups=3.29, wpb=3541.1, bsz=169.3, num_updates=50400, lr=0.000281718, gnorm=1.32, train_wall=30, wall=15756\n",
            "2020-08-04 01:19:24 | INFO | train_inner | epoch 050:    765 / 1015 loss=4.009, nll_loss=2.357, ppl=5.12, wps=11827.4, ups=3.31, wpb=3571.4, bsz=166.2, num_updates=50500, lr=0.000281439, gnorm=1.322, train_wall=30, wall=15786\n",
            "2020-08-04 01:19:55 | INFO | train_inner | epoch 050:    865 / 1015 loss=3.984, nll_loss=2.328, ppl=5.02, wps=11684.4, ups=3.27, wpb=3568.9, bsz=161.8, num_updates=50600, lr=0.000281161, gnorm=1.308, train_wall=30, wall=15817\n",
            "2020-08-04 01:20:25 | INFO | train_inner | epoch 050:    965 / 1015 loss=4.023, nll_loss=2.372, ppl=5.18, wps=11516.4, ups=3.29, wpb=3496.6, bsz=158.9, num_updates=50700, lr=0.000280883, gnorm=1.358, train_wall=30, wall=15847\n",
            "2020-08-04 01:20:40 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:20:46 | INFO | valid | epoch 050 | valid on 'valid' subset | loss 4.296 | nll_loss 2.609 | ppl 6.1 | wps 27254.3 | wpb 2866.6 | bsz 127.8 | num_updates 50750 | best_loss 4.289\n",
            "2020-08-04 01:20:46 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:20:48 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint50.pt (epoch 50 @ 50750 updates, score 4.296) (writing took 1.6587511639991135 seconds)\n",
            "2020-08-04 01:20:48 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:20:48 | INFO | fairseq_cli.train | end of epoch 50 (average epoch stats below)\n",
            "2020-08-04 01:20:48 | INFO | train | epoch 050 | loss 3.98 | nll_loss 2.322 | ppl 5 | wps 11367.9 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 50750 | lr 0.000280745 | gnorm 1.325 | train_wall 308 | wall 15870\n",
            "2020-08-04 01:20:48 | INFO | fairseq_cli.train | begin training epoch 50\n",
            "2020-08-04 01:21:03 | INFO | train_inner | epoch 051:     50 / 1015 loss=3.977, nll_loss=2.32, ppl=4.99, wps=9327.4, ups=2.61, wpb=3575.9, bsz=161.3, num_updates=50800, lr=0.000280607, gnorm=1.334, train_wall=30, wall=15885\n",
            "2020-08-04 01:21:34 | INFO | train_inner | epoch 051:    150 / 1015 loss=3.918, nll_loss=2.25, ppl=4.76, wps=11689, ups=3.28, wpb=3563.9, bsz=159.1, num_updates=50900, lr=0.000280331, gnorm=1.337, train_wall=30, wall=15916\n",
            "2020-08-04 01:22:04 | INFO | train_inner | epoch 051:    250 / 1015 loss=3.956, nll_loss=2.294, ppl=4.9, wps=11637.7, ups=3.27, wpb=3557.4, bsz=149.4, num_updates=51000, lr=0.000280056, gnorm=1.347, train_wall=30, wall=15946\n",
            "2020-08-04 01:22:35 | INFO | train_inner | epoch 051:    350 / 1015 loss=3.96, nll_loss=2.299, ppl=4.92, wps=11717.1, ups=3.3, wpb=3547, bsz=151.1, num_updates=51100, lr=0.000279782, gnorm=1.31, train_wall=30, wall=15977\n",
            "2020-08-04 01:23:05 | INFO | train_inner | epoch 051:    450 / 1015 loss=3.962, nll_loss=2.302, ppl=4.93, wps=11585.5, ups=3.28, wpb=3535.9, bsz=164.5, num_updates=51200, lr=0.000279508, gnorm=1.339, train_wall=30, wall=16007\n",
            "2020-08-04 01:23:35 | INFO | train_inner | epoch 051:    550 / 1015 loss=3.939, nll_loss=2.276, ppl=4.84, wps=11608.3, ups=3.3, wpb=3516.7, bsz=173.2, num_updates=51300, lr=0.000279236, gnorm=1.331, train_wall=30, wall=16038\n",
            "2020-08-04 01:24:06 | INFO | train_inner | epoch 051:    650 / 1015 loss=4.003, nll_loss=2.349, ppl=5.09, wps=11809, ups=3.29, wpb=3587, bsz=155.8, num_updates=51400, lr=0.000278964, gnorm=1.326, train_wall=30, wall=16068\n",
            "2020-08-04 01:24:36 | INFO | train_inner | epoch 051:    750 / 1015 loss=4, nll_loss=2.345, ppl=5.08, wps=11558.1, ups=3.31, wpb=3490.2, bsz=150.2, num_updates=51500, lr=0.000278693, gnorm=1.369, train_wall=30, wall=16098\n",
            "2020-08-04 01:25:06 | INFO | train_inner | epoch 051:    850 / 1015 loss=3.988, nll_loss=2.332, ppl=5.03, wps=11663, ups=3.29, wpb=3540.9, bsz=160.5, num_updates=51600, lr=0.000278423, gnorm=1.319, train_wall=30, wall=16128\n",
            "2020-08-04 01:25:37 | INFO | train_inner | epoch 051:    950 / 1015 loss=4.023, nll_loss=2.372, ppl=5.18, wps=11648.9, ups=3.3, wpb=3532.8, bsz=151.3, num_updates=51700, lr=0.000278154, gnorm=1.336, train_wall=30, wall=16159\n",
            "2020-08-04 01:25:56 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:26:03 | INFO | valid | epoch 051 | valid on 'valid' subset | loss 4.286 | nll_loss 2.588 | ppl 6.01 | wps 27127.8 | wpb 2866.6 | bsz 127.8 | num_updates 51765 | best_loss 4.286\n",
            "2020-08-04 01:26:03 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:26:05 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint51.pt (epoch 51 @ 51765 updates, score 4.286) (writing took 2.5300420280000253 seconds)\n",
            "2020-08-04 01:26:05 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:26:05 | INFO | fairseq_cli.train | end of epoch 51 (average epoch stats below)\n",
            "2020-08-04 01:26:05 | INFO | train | epoch 051 | loss 3.972 | nll_loss 2.313 | ppl 4.97 | wps 11359.6 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 51765 | lr 0.000277979 | gnorm 1.334 | train_wall 308 | wall 16187\n",
            "2020-08-04 01:26:05 | INFO | fairseq_cli.train | begin training epoch 51\n",
            "2020-08-04 01:26:16 | INFO | train_inner | epoch 052:     35 / 1015 loss=3.988, nll_loss=2.331, ppl=5.03, wps=9128.8, ups=2.55, wpb=3577.3, bsz=151.2, num_updates=51800, lr=0.000277885, gnorm=1.354, train_wall=30, wall=16198\n",
            "2020-08-04 01:26:46 | INFO | train_inner | epoch 052:    135 / 1015 loss=3.949, nll_loss=2.286, ppl=4.88, wps=11586.8, ups=3.32, wpb=3487.2, bsz=150.5, num_updates=51900, lr=0.000277617, gnorm=1.35, train_wall=30, wall=16228\n",
            "2020-08-04 01:27:16 | INFO | train_inner | epoch 052:    235 / 1015 loss=3.921, nll_loss=2.254, ppl=4.77, wps=11668.8, ups=3.28, wpb=3557.8, bsz=157.4, num_updates=52000, lr=0.00027735, gnorm=1.333, train_wall=30, wall=16259\n",
            "2020-08-04 01:27:47 | INFO | train_inner | epoch 052:    335 / 1015 loss=3.941, nll_loss=2.278, ppl=4.85, wps=11686, ups=3.3, wpb=3539.1, bsz=158.9, num_updates=52100, lr=0.000277084, gnorm=1.358, train_wall=30, wall=16289\n",
            "2020-08-04 01:28:17 | INFO | train_inner | epoch 052:    435 / 1015 loss=4.002, nll_loss=2.346, ppl=5.08, wps=11815.3, ups=3.31, wpb=3567.8, bsz=138.4, num_updates=52200, lr=0.000276818, gnorm=1.326, train_wall=30, wall=16319\n",
            "2020-08-04 01:28:47 | INFO | train_inner | epoch 052:    535 / 1015 loss=3.99, nll_loss=2.332, ppl=5.04, wps=11654.4, ups=3.29, wpb=3538.4, bsz=159, num_updates=52300, lr=0.000276553, gnorm=1.362, train_wall=30, wall=16349\n",
            "2020-08-04 01:29:17 | INFO | train_inner | epoch 052:    635 / 1015 loss=4.01, nll_loss=2.354, ppl=5.11, wps=11555, ups=3.31, wpb=3492.3, bsz=144.6, num_updates=52400, lr=0.000276289, gnorm=1.405, train_wall=30, wall=16380\n",
            "2020-08-04 01:29:48 | INFO | train_inner | epoch 052:    735 / 1015 loss=3.948, nll_loss=2.289, ppl=4.89, wps=11740.4, ups=3.3, wpb=3562.7, bsz=175.1, num_updates=52500, lr=0.000276026, gnorm=1.324, train_wall=30, wall=16410\n",
            "2020-08-04 01:30:19 | INFO | train_inner | epoch 052:    835 / 1015 loss=3.954, nll_loss=2.294, ppl=4.9, wps=11698.2, ups=3.25, wpb=3597.3, bsz=170.5, num_updates=52600, lr=0.000275764, gnorm=1.315, train_wall=31, wall=16441\n",
            "2020-08-04 01:30:49 | INFO | train_inner | epoch 052:    935 / 1015 loss=4.018, nll_loss=2.365, ppl=5.15, wps=11639.1, ups=3.29, wpb=3534.9, bsz=150.9, num_updates=52700, lr=0.000275502, gnorm=1.392, train_wall=30, wall=16471\n",
            "2020-08-04 01:31:13 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:31:19 | INFO | valid | epoch 052 | valid on 'valid' subset | loss 4.301 | nll_loss 2.605 | ppl 6.08 | wps 26975.4 | wpb 2866.6 | bsz 127.8 | num_updates 52780 | best_loss 4.286\n",
            "2020-08-04 01:31:19 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:31:21 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint52.pt (epoch 52 @ 52780 updates, score 4.301) (writing took 1.4907894059979299 seconds)\n",
            "2020-08-04 01:31:21 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:31:21 | INFO | fairseq_cli.train | end of epoch 52 (average epoch stats below)\n",
            "2020-08-04 01:31:21 | INFO | train | epoch 052 | loss 3.969 | nll_loss 2.309 | ppl 4.96 | wps 11410.1 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 52780 | lr 0.000275293 | gnorm 1.348 | train_wall 307 | wall 16503\n",
            "2020-08-04 01:31:21 | INFO | fairseq_cli.train | begin training epoch 52\n",
            "2020-08-04 01:31:27 | INFO | train_inner | epoch 053:     20 / 1015 loss=3.967, nll_loss=2.31, ppl=4.96, wps=9494.2, ups=2.64, wpb=3600.5, bsz=172.2, num_updates=52800, lr=0.000275241, gnorm=1.337, train_wall=30, wall=16509\n",
            "2020-08-04 01:31:57 | INFO | train_inner | epoch 053:    120 / 1015 loss=3.868, nll_loss=2.195, ppl=4.58, wps=11739.5, ups=3.29, wpb=3564.8, bsz=164.2, num_updates=52900, lr=0.000274981, gnorm=1.294, train_wall=30, wall=16539\n",
            "2020-08-04 01:32:27 | INFO | train_inner | epoch 053:    220 / 1015 loss=3.983, nll_loss=2.324, ppl=5.01, wps=11592.5, ups=3.34, wpb=3470.6, bsz=137, num_updates=53000, lr=0.000274721, gnorm=1.435, train_wall=30, wall=16569\n",
            "2020-08-04 01:32:57 | INFO | train_inner | epoch 053:    320 / 1015 loss=3.996, nll_loss=2.34, ppl=5.06, wps=11545.1, ups=3.31, wpb=3488.7, bsz=145, num_updates=53100, lr=0.000274462, gnorm=1.383, train_wall=30, wall=16600\n",
            "2020-08-04 01:33:28 | INFO | train_inner | epoch 053:    420 / 1015 loss=3.972, nll_loss=2.311, ppl=4.96, wps=11752, ups=3.29, wpb=3566.7, bsz=148.3, num_updates=53200, lr=0.000274204, gnorm=1.348, train_wall=30, wall=16630\n",
            "2020-08-04 01:33:58 | INFO | train_inner | epoch 053:    520 / 1015 loss=3.932, nll_loss=2.268, ppl=4.82, wps=11688.5, ups=3.28, wpb=3563.7, bsz=165.4, num_updates=53300, lr=0.000273947, gnorm=1.344, train_wall=30, wall=16660\n",
            "2020-08-04 01:34:29 | INFO | train_inner | epoch 053:    620 / 1015 loss=3.944, nll_loss=2.283, ppl=4.87, wps=11788.9, ups=3.27, wpb=3605.6, bsz=172.8, num_updates=53400, lr=0.00027369, gnorm=1.318, train_wall=30, wall=16691\n",
            "2020-08-04 01:34:59 | INFO | train_inner | epoch 053:    720 / 1015 loss=3.977, nll_loss=2.317, ppl=4.98, wps=11468, ups=3.29, wpb=3489.5, bsz=151.1, num_updates=53500, lr=0.000273434, gnorm=1.391, train_wall=30, wall=16721\n",
            "2020-08-04 01:35:30 | INFO | train_inner | epoch 053:    820 / 1015 loss=3.972, nll_loss=2.313, ppl=4.97, wps=11814.1, ups=3.31, wpb=3573.4, bsz=158.8, num_updates=53600, lr=0.000273179, gnorm=1.346, train_wall=30, wall=16752\n",
            "2020-08-04 01:36:00 | INFO | train_inner | epoch 053:    920 / 1015 loss=4.013, nll_loss=2.361, ppl=5.14, wps=11845, ups=3.28, wpb=3611.5, bsz=160.4, num_updates=53700, lr=0.000272925, gnorm=1.343, train_wall=30, wall=16782\n",
            "2020-08-04 01:36:29 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:36:35 | INFO | valid | epoch 053 | valid on 'valid' subset | loss 4.27 | nll_loss 2.576 | ppl 5.96 | wps 27201.4 | wpb 2866.6 | bsz 127.8 | num_updates 53795 | best_loss 4.27\n",
            "2020-08-04 01:36:35 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:36:38 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint53.pt (epoch 53 @ 53795 updates, score 4.27) (writing took 2.560115248998045 seconds)\n",
            "2020-08-04 01:36:38 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:36:38 | INFO | fairseq_cli.train | end of epoch 53 (average epoch stats below)\n",
            "2020-08-04 01:36:38 | INFO | train | epoch 053 | loss 3.961 | nll_loss 2.301 | ppl 4.93 | wps 11380.9 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 53795 | lr 0.000272684 | gnorm 1.355 | train_wall 307 | wall 16820\n",
            "2020-08-04 01:36:38 | INFO | fairseq_cli.train | begin training epoch 53\n",
            "2020-08-04 01:36:39 | INFO | train_inner | epoch 054:      5 / 1015 loss=3.943, nll_loss=2.284, ppl=4.87, wps=9154.4, ups=2.55, wpb=3590.4, bsz=181.3, num_updates=53800, lr=0.000272671, gnorm=1.317, train_wall=30, wall=16821\n",
            "2020-08-04 01:37:09 | INFO | train_inner | epoch 054:    105 / 1015 loss=3.94, nll_loss=2.275, ppl=4.84, wps=11780.1, ups=3.3, wpb=3564.8, bsz=146.4, num_updates=53900, lr=0.000272418, gnorm=1.328, train_wall=30, wall=16852\n",
            "2020-08-04 01:37:40 | INFO | train_inner | epoch 054:    205 / 1015 loss=3.918, nll_loss=2.251, ppl=4.76, wps=11756.8, ups=3.28, wpb=3588.8, bsz=155.7, num_updates=54000, lr=0.000272166, gnorm=1.329, train_wall=30, wall=16882\n",
            "2020-08-04 01:38:10 | INFO | train_inner | epoch 054:    305 / 1015 loss=3.922, nll_loss=2.255, ppl=4.77, wps=11533.7, ups=3.28, wpb=3514.8, bsz=161.3, num_updates=54100, lr=0.000271914, gnorm=1.366, train_wall=30, wall=16913\n",
            "2020-08-04 01:38:41 | INFO | train_inner | epoch 054:    405 / 1015 loss=3.966, nll_loss=2.306, ppl=4.94, wps=11802.7, ups=3.3, wpb=3578.4, bsz=160, num_updates=54200, lr=0.000271663, gnorm=1.333, train_wall=30, wall=16943\n",
            "2020-08-04 01:39:11 | INFO | train_inner | epoch 054:    505 / 1015 loss=3.957, nll_loss=2.297, ppl=4.91, wps=11570.1, ups=3.28, wpb=3526.9, bsz=155.9, num_updates=54300, lr=0.000271413, gnorm=1.383, train_wall=30, wall=16973\n",
            "2020-08-04 01:39:42 | INFO | train_inner | epoch 054:    605 / 1015 loss=3.96, nll_loss=2.3, ppl=4.93, wps=11939.3, ups=3.3, wpb=3620.4, bsz=162.6, num_updates=54400, lr=0.000271163, gnorm=1.329, train_wall=30, wall=17004\n",
            "2020-08-04 01:40:12 | INFO | train_inner | epoch 054:    705 / 1015 loss=3.973, nll_loss=2.314, ppl=4.97, wps=11618.4, ups=3.28, wpb=3547.5, bsz=154.3, num_updates=54500, lr=0.000270914, gnorm=1.375, train_wall=30, wall=17034\n",
            "2020-08-04 01:40:42 | INFO | train_inner | epoch 054:    805 / 1015 loss=3.981, nll_loss=2.323, ppl=5, wps=11607, ups=3.3, wpb=3522.1, bsz=152.6, num_updates=54600, lr=0.000270666, gnorm=1.399, train_wall=30, wall=17065\n",
            "2020-08-04 01:41:13 | INFO | train_inner | epoch 054:    905 / 1015 loss=3.92, nll_loss=2.257, ppl=4.78, wps=11575.3, ups=3.28, wpb=3527.9, bsz=179, num_updates=54700, lr=0.000270418, gnorm=1.341, train_wall=30, wall=17095\n",
            "2020-08-04 01:41:43 | INFO | train_inner | epoch 054:   1005 / 1015 loss=3.999, nll_loss=2.345, ppl=5.08, wps=11649.6, ups=3.3, wpb=3527.5, bsz=152.9, num_updates=54800, lr=0.000270172, gnorm=1.383, train_wall=30, wall=17125\n",
            "2020-08-04 01:41:46 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:41:52 | INFO | valid | epoch 054 | valid on 'valid' subset | loss 4.278 | nll_loss 2.582 | ppl 5.99 | wps 27124.2 | wpb 2866.6 | bsz 127.8 | num_updates 54810 | best_loss 4.27\n",
            "2020-08-04 01:41:52 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:41:54 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint54.pt (epoch 54 @ 54810 updates, score 4.278) (writing took 1.6034427100021276 seconds)\n",
            "2020-08-04 01:41:54 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:41:54 | INFO | fairseq_cli.train | end of epoch 54 (average epoch stats below)\n",
            "2020-08-04 01:41:54 | INFO | train | epoch 054 | loss 3.955 | nll_loss 2.293 | ppl 4.9 | wps 11387.3 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 54810 | lr 0.000270147 | gnorm 1.358 | train_wall 308 | wall 17136\n",
            "2020-08-04 01:41:54 | INFO | fairseq_cli.train | begin training epoch 54\n",
            "2020-08-04 01:42:21 | INFO | train_inner | epoch 055:     90 / 1015 loss=3.921, nll_loss=2.254, ppl=4.77, wps=9213.1, ups=2.62, wpb=3517.3, bsz=154.1, num_updates=54900, lr=0.000269925, gnorm=1.387, train_wall=30, wall=17164\n",
            "2020-08-04 01:42:52 | INFO | train_inner | epoch 055:    190 / 1015 loss=3.899, nll_loss=2.228, ppl=4.69, wps=11686.5, ups=3.29, wpb=3550.7, bsz=151.8, num_updates=55000, lr=0.00026968, gnorm=1.349, train_wall=30, wall=17194\n",
            "2020-08-04 01:43:22 | INFO | train_inner | epoch 055:    290 / 1015 loss=3.931, nll_loss=2.266, ppl=4.81, wps=11692.1, ups=3.26, wpb=3581.1, bsz=157.9, num_updates=55100, lr=0.000269435, gnorm=1.331, train_wall=31, wall=17225\n",
            "2020-08-04 01:43:53 | INFO | train_inner | epoch 055:    390 / 1015 loss=3.989, nll_loss=2.33, ppl=5.03, wps=11636.9, ups=3.26, wpb=3565.9, bsz=135, num_updates=55200, lr=0.000269191, gnorm=1.372, train_wall=31, wall=17255\n",
            "2020-08-04 01:44:23 | INFO | train_inner | epoch 055:    490 / 1015 loss=3.937, nll_loss=2.274, ppl=4.84, wps=11650.3, ups=3.3, wpb=3529.8, bsz=158.9, num_updates=55300, lr=0.000268947, gnorm=1.403, train_wall=30, wall=17285\n",
            "2020-08-04 01:44:54 | INFO | train_inner | epoch 055:    590 / 1015 loss=3.955, nll_loss=2.295, ppl=4.91, wps=11574.1, ups=3.28, wpb=3530.1, bsz=155.9, num_updates=55400, lr=0.000268705, gnorm=1.377, train_wall=30, wall=17316\n",
            "2020-08-04 01:45:24 | INFO | train_inner | epoch 055:    690 / 1015 loss=3.911, nll_loss=2.245, ppl=4.74, wps=11702.1, ups=3.29, wpb=3555.7, bsz=178.3, num_updates=55500, lr=0.000268462, gnorm=1.367, train_wall=30, wall=17346\n",
            "2020-08-04 01:45:55 | INFO | train_inner | epoch 055:    790 / 1015 loss=3.999, nll_loss=2.345, ppl=5.08, wps=11745.4, ups=3.29, wpb=3564.7, bsz=159.8, num_updates=55600, lr=0.000268221, gnorm=1.388, train_wall=30, wall=17377\n",
            "2020-08-04 01:46:25 | INFO | train_inner | epoch 055:    890 / 1015 loss=3.957, nll_loss=2.297, ppl=4.91, wps=11534.5, ups=3.27, wpb=3528.8, bsz=165, num_updates=55700, lr=0.00026798, gnorm=1.38, train_wall=30, wall=17407\n",
            "2020-08-04 01:46:56 | INFO | train_inner | epoch 055:    990 / 1015 loss=4.01, nll_loss=2.357, ppl=5.12, wps=11615.2, ups=3.27, wpb=3548.5, bsz=149.4, num_updates=55800, lr=0.00026774, gnorm=1.381, train_wall=30, wall=17438\n",
            "2020-08-04 01:47:03 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:47:10 | INFO | valid | epoch 055 | valid on 'valid' subset | loss 4.268 | nll_loss 2.578 | ppl 5.97 | wps 27159.4 | wpb 2866.6 | bsz 127.8 | num_updates 55825 | best_loss 4.268\n",
            "2020-08-04 01:47:10 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:47:12 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint55.pt (epoch 55 @ 55825 updates, score 4.268) (writing took 2.6033867859987367 seconds)\n",
            "2020-08-04 01:47:12 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:47:12 | INFO | fairseq_cli.train | end of epoch 55 (average epoch stats below)\n",
            "2020-08-04 01:47:12 | INFO | train | epoch 055 | loss 3.949 | nll_loss 2.286 | ppl 4.88 | wps 11325.8 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 55825 | lr 0.00026768 | gnorm 1.37 | train_wall 308 | wall 17454\n",
            "2020-08-04 01:47:12 | INFO | fairseq_cli.train | begin training epoch 55\n",
            "2020-08-04 01:47:35 | INFO | train_inner | epoch 056:     75 / 1015 loss=3.877, nll_loss=2.205, ppl=4.61, wps=8913.6, ups=2.53, wpb=3530, bsz=168.1, num_updates=55900, lr=0.0002675, gnorm=1.34, train_wall=31, wall=17477\n",
            "2020-08-04 01:48:06 | INFO | train_inner | epoch 056:    175 / 1015 loss=3.868, nll_loss=2.194, ppl=4.58, wps=11697.5, ups=3.27, wpb=3580, bsz=159.9, num_updates=56000, lr=0.000267261, gnorm=1.318, train_wall=31, wall=17508\n",
            "2020-08-04 01:48:37 | INFO | train_inner | epoch 056:    275 / 1015 loss=3.951, nll_loss=2.288, ppl=4.88, wps=11522, ups=3.27, wpb=3520.7, bsz=152.5, num_updates=56100, lr=0.000267023, gnorm=1.388, train_wall=30, wall=17539\n",
            "2020-08-04 01:49:07 | INFO | train_inner | epoch 056:    375 / 1015 loss=3.923, nll_loss=2.258, ppl=4.78, wps=11732.3, ups=3.25, wpb=3610, bsz=169.4, num_updates=56200, lr=0.000266785, gnorm=1.326, train_wall=31, wall=17569\n",
            "2020-08-04 01:49:38 | INFO | train_inner | epoch 056:    475 / 1015 loss=3.948, nll_loss=2.286, ppl=4.88, wps=11645.5, ups=3.25, wpb=3586.3, bsz=154.8, num_updates=56300, lr=0.000266548, gnorm=1.357, train_wall=31, wall=17600\n",
            "2020-08-04 01:50:09 | INFO | train_inner | epoch 056:    575 / 1015 loss=3.961, nll_loss=2.301, ppl=4.93, wps=11642.7, ups=3.28, wpb=3553.8, bsz=157.3, num_updates=56400, lr=0.000266312, gnorm=1.368, train_wall=30, wall=17631\n",
            "2020-08-04 01:50:40 | INFO | train_inner | epoch 056:    675 / 1015 loss=3.973, nll_loss=2.314, ppl=4.97, wps=11556.6, ups=3.23, wpb=3576.3, bsz=143.8, num_updates=56500, lr=0.000266076, gnorm=1.375, train_wall=31, wall=17662\n",
            "2020-08-04 01:51:10 | INFO | train_inner | epoch 056:    775 / 1015 loss=3.973, nll_loss=2.315, ppl=4.98, wps=11501.8, ups=3.29, wpb=3494.7, bsz=164.8, num_updates=56600, lr=0.000265841, gnorm=1.416, train_wall=30, wall=17692\n",
            "2020-08-04 01:51:41 | INFO | train_inner | epoch 056:    875 / 1015 loss=3.979, nll_loss=2.321, ppl=5, wps=11512.4, ups=3.27, wpb=3522.3, bsz=153.9, num_updates=56700, lr=0.000265606, gnorm=1.385, train_wall=30, wall=17723\n",
            "2020-08-04 01:52:11 | INFO | train_inner | epoch 056:    975 / 1015 loss=3.964, nll_loss=2.303, ppl=4.94, wps=11527.3, ups=3.25, wpb=3547.1, bsz=159.1, num_updates=56800, lr=0.000265372, gnorm=1.447, train_wall=31, wall=17753\n",
            "2020-08-04 01:52:23 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:52:30 | INFO | valid | epoch 056 | valid on 'valid' subset | loss 4.302 | nll_loss 2.6 | ppl 6.06 | wps 26959.3 | wpb 2866.6 | bsz 127.8 | num_updates 56840 | best_loss 4.268\n",
            "2020-08-04 01:52:30 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:52:31 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint56.pt (epoch 56 @ 56840 updates, score 4.302) (writing took 1.542887266001344 seconds)\n",
            "2020-08-04 01:52:31 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:52:31 | INFO | fairseq_cli.train | end of epoch 56 (average epoch stats below)\n",
            "2020-08-04 01:52:31 | INFO | train | epoch 056 | loss 3.943 | nll_loss 2.28 | ppl 4.86 | wps 11293.7 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 56840 | lr 0.000265279 | gnorm 1.375 | train_wall 310 | wall 17773\n",
            "2020-08-04 01:52:31 | INFO | fairseq_cli.train | begin training epoch 56\n",
            "2020-08-04 01:52:50 | INFO | train_inner | epoch 057:     60 / 1015 loss=3.922, nll_loss=2.255, ppl=4.77, wps=9114.9, ups=2.6, wpb=3503.2, bsz=152.7, num_updates=56900, lr=0.000265139, gnorm=1.396, train_wall=31, wall=17792\n",
            "2020-08-04 01:53:20 | INFO | train_inner | epoch 057:    160 / 1015 loss=3.903, nll_loss=2.233, ppl=4.7, wps=11594.2, ups=3.25, wpb=3563.3, bsz=155.2, num_updates=57000, lr=0.000264906, gnorm=1.369, train_wall=31, wall=17823\n",
            "2020-08-04 01:53:51 | INFO | train_inner | epoch 057:    260 / 1015 loss=3.94, nll_loss=2.276, ppl=4.84, wps=11684.4, ups=3.25, wpb=3595.5, bsz=154.6, num_updates=57100, lr=0.000264674, gnorm=1.342, train_wall=31, wall=17853\n",
            "2020-08-04 01:54:22 | INFO | train_inner | epoch 057:    360 / 1015 loss=3.888, nll_loss=2.219, ppl=4.66, wps=11549.9, ups=3.26, wpb=3543.1, bsz=168.1, num_updates=57200, lr=0.000264443, gnorm=1.358, train_wall=31, wall=17884\n",
            "2020-08-04 01:54:53 | INFO | train_inner | epoch 057:    460 / 1015 loss=3.874, nll_loss=2.203, ppl=4.6, wps=11528.1, ups=3.26, wpb=3535, bsz=178.7, num_updates=57300, lr=0.000264212, gnorm=1.404, train_wall=31, wall=17915\n",
            "2020-08-04 01:55:23 | INFO | train_inner | epoch 057:    560 / 1015 loss=3.97, nll_loss=2.31, ppl=4.96, wps=11433.3, ups=3.26, wpb=3502.9, bsz=149.4, num_updates=57400, lr=0.000263982, gnorm=1.415, train_wall=31, wall=17945\n",
            "2020-08-04 01:55:54 | INFO | train_inner | epoch 057:    660 / 1015 loss=3.931, nll_loss=2.267, ppl=4.81, wps=11652.2, ups=3.25, wpb=3580.7, bsz=165.7, num_updates=57500, lr=0.000263752, gnorm=1.351, train_wall=31, wall=17976\n",
            "2020-08-04 01:56:24 | INFO | train_inner | epoch 057:    760 / 1015 loss=4.018, nll_loss=2.365, ppl=5.15, wps=11491.2, ups=3.28, wpb=3504.3, bsz=139.4, num_updates=57600, lr=0.000263523, gnorm=1.424, train_wall=30, wall=18007\n",
            "2020-08-04 01:56:55 | INFO | train_inner | epoch 057:    860 / 1015 loss=3.915, nll_loss=2.249, ppl=4.75, wps=11628.7, ups=3.25, wpb=3583.3, bsz=170.4, num_updates=57700, lr=0.000263295, gnorm=1.333, train_wall=31, wall=18037\n",
            "2020-08-04 01:57:26 | INFO | train_inner | epoch 057:    960 / 1015 loss=3.998, nll_loss=2.343, ppl=5.07, wps=11554.5, ups=3.25, wpb=3557.5, bsz=143, num_updates=57800, lr=0.000263067, gnorm=1.403, train_wall=31, wall=18068\n",
            "2020-08-04 01:57:43 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 01:57:49 | INFO | valid | epoch 057 | valid on 'valid' subset | loss 4.27 | nll_loss 2.571 | ppl 5.94 | wps 26906.7 | wpb 2866.6 | bsz 127.8 | num_updates 57855 | best_loss 4.268\n",
            "2020-08-04 01:57:49 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 01:57:51 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint57.pt (epoch 57 @ 57855 updates, score 4.27) (writing took 1.6797169240016956 seconds)\n",
            "2020-08-04 01:57:51 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 01:57:51 | INFO | fairseq_cli.train | end of epoch 57 (average epoch stats below)\n",
            "2020-08-04 01:57:51 | INFO | train | epoch 057 | loss 3.937 | nll_loss 2.273 | ppl 4.83 | wps 11273.2 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 57855 | lr 0.000262942 | gnorm 1.378 | train_wall 311 | wall 18093\n",
            "2020-08-04 01:57:51 | INFO | fairseq_cli.train | begin training epoch 57\n",
            "2020-08-04 01:58:05 | INFO | train_inner | epoch 058:     45 / 1015 loss=3.89, nll_loss=2.222, ppl=4.66, wps=9268.9, ups=2.59, wpb=3581, bsz=173.3, num_updates=57900, lr=0.00026284, gnorm=1.371, train_wall=31, wall=18107\n",
            "2020-08-04 01:58:35 | INFO | train_inner | epoch 058:    145 / 1015 loss=3.869, nll_loss=2.197, ppl=4.58, wps=11560.7, ups=3.26, wpb=3548.7, bsz=167.7, num_updates=58000, lr=0.000262613, gnorm=1.365, train_wall=31, wall=18138\n",
            "2020-08-04 01:59:06 | INFO | train_inner | epoch 058:    245 / 1015 loss=3.922, nll_loss=2.255, ppl=4.77, wps=11532.9, ups=3.25, wpb=3543.5, bsz=158, num_updates=58100, lr=0.000262387, gnorm=1.393, train_wall=31, wall=18168\n",
            "2020-08-04 01:59:37 | INFO | train_inner | epoch 058:    345 / 1015 loss=3.908, nll_loss=2.24, ppl=4.72, wps=11628.5, ups=3.26, wpb=3569.7, bsz=162.6, num_updates=58200, lr=0.000262161, gnorm=1.383, train_wall=31, wall=18199\n",
            "2020-08-04 02:00:07 | INFO | train_inner | epoch 058:    445 / 1015 loss=3.979, nll_loss=2.32, ppl=4.99, wps=11505.2, ups=3.26, wpb=3524.7, bsz=141.4, num_updates=58300, lr=0.000261936, gnorm=1.439, train_wall=31, wall=18230\n",
            "2020-08-04 02:00:38 | INFO | train_inner | epoch 058:    545 / 1015 loss=3.978, nll_loss=2.319, ppl=4.99, wps=11517.3, ups=3.25, wpb=3538.4, bsz=143.8, num_updates=58400, lr=0.000261712, gnorm=1.415, train_wall=31, wall=18260\n",
            "2020-08-04 02:01:09 | INFO | train_inner | epoch 058:    645 / 1015 loss=3.945, nll_loss=2.283, ppl=4.87, wps=11540.8, ups=3.27, wpb=3530.5, bsz=159, num_updates=58500, lr=0.000261488, gnorm=1.394, train_wall=30, wall=18291\n",
            "2020-08-04 02:01:39 | INFO | train_inner | epoch 058:    745 / 1015 loss=3.944, nll_loss=2.282, ppl=4.86, wps=11556.5, ups=3.27, wpb=3536.9, bsz=161.9, num_updates=58600, lr=0.000261265, gnorm=1.381, train_wall=30, wall=18321\n",
            "2020-08-04 02:02:10 | INFO | train_inner | epoch 058:    845 / 1015 loss=3.964, nll_loss=2.305, ppl=4.94, wps=11574.5, ups=3.24, wpb=3567.6, bsz=153.8, num_updates=58700, lr=0.000261042, gnorm=1.406, train_wall=31, wall=18352\n",
            "2020-08-04 02:02:41 | INFO | train_inner | epoch 058:    945 / 1015 loss=3.956, nll_loss=2.296, ppl=4.91, wps=11531.5, ups=3.23, wpb=3566.4, bsz=157.2, num_updates=58800, lr=0.00026082, gnorm=1.396, train_wall=31, wall=18383\n",
            "2020-08-04 02:03:03 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:03:09 | INFO | valid | epoch 058 | valid on 'valid' subset | loss 4.28 | nll_loss 2.584 | ppl 5.99 | wps 26836.6 | wpb 2866.6 | bsz 127.8 | num_updates 58870 | best_loss 4.268\n",
            "2020-08-04 02:03:09 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:03:10 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint58.pt (epoch 58 @ 58870 updates, score 4.28) (writing took 1.5385673569981009 seconds)\n",
            "2020-08-04 02:03:11 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:03:11 | INFO | fairseq_cli.train | end of epoch 58 (average epoch stats below)\n",
            "2020-08-04 02:03:11 | INFO | train | epoch 058 | loss 3.932 | nll_loss 2.268 | ppl 4.81 | wps 11271.2 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 58870 | lr 0.000260665 | gnorm 1.393 | train_wall 311 | wall 18413\n",
            "2020-08-04 02:03:11 | INFO | fairseq_cli.train | begin training epoch 58\n",
            "2020-08-04 02:03:20 | INFO | train_inner | epoch 059:     30 / 1015 loss=3.904, nll_loss=2.236, ppl=4.71, wps=9285.6, ups=2.58, wpb=3605.2, bsz=156.5, num_updates=58900, lr=0.000260599, gnorm=1.346, train_wall=31, wall=18422\n",
            "2020-08-04 02:03:50 | INFO | train_inner | epoch 059:    130 / 1015 loss=3.91, nll_loss=2.241, ppl=4.73, wps=11522.6, ups=3.27, wpb=3518.6, bsz=145, num_updates=59000, lr=0.000260378, gnorm=1.412, train_wall=30, wall=18453\n",
            "2020-08-04 02:04:21 | INFO | train_inner | epoch 059:    230 / 1015 loss=3.939, nll_loss=2.275, ppl=4.84, wps=11529.2, ups=3.26, wpb=3534.4, bsz=140.4, num_updates=59100, lr=0.000260157, gnorm=1.418, train_wall=31, wall=18483\n",
            "2020-08-04 02:04:52 | INFO | train_inner | epoch 059:    330 / 1015 loss=3.909, nll_loss=2.241, ppl=4.73, wps=11465.9, ups=3.24, wpb=3535, bsz=159.9, num_updates=59200, lr=0.000259938, gnorm=1.421, train_wall=31, wall=18514\n",
            "2020-08-04 02:05:23 | INFO | train_inner | epoch 059:    430 / 1015 loss=3.895, nll_loss=2.225, ppl=4.67, wps=11276.3, ups=3.25, wpb=3467.9, bsz=158.2, num_updates=59300, lr=0.000259718, gnorm=1.406, train_wall=31, wall=18545\n",
            "2020-08-04 02:05:53 | INFO | train_inner | epoch 059:    530 / 1015 loss=3.959, nll_loss=2.297, ppl=4.91, wps=11383.4, ups=3.27, wpb=3479.6, bsz=146.6, num_updates=59400, lr=0.0002595, gnorm=1.471, train_wall=30, wall=18575\n",
            "2020-08-04 02:06:24 | INFO | train_inner | epoch 059:    630 / 1015 loss=3.919, nll_loss=2.254, ppl=4.77, wps=11748.4, ups=3.25, wpb=3614.2, bsz=168.2, num_updates=59500, lr=0.000259281, gnorm=1.349, train_wall=31, wall=18606\n",
            "2020-08-04 02:06:55 | INFO | train_inner | epoch 059:    730 / 1015 loss=3.897, nll_loss=2.229, ppl=4.69, wps=11683.3, ups=3.24, wpb=3602.9, bsz=167, num_updates=59600, lr=0.000259064, gnorm=1.368, train_wall=31, wall=18637\n",
            "2020-08-04 02:07:26 | INFO | train_inner | epoch 059:    830 / 1015 loss=3.928, nll_loss=2.265, ppl=4.81, wps=11702.7, ups=3.25, wpb=3595.5, bsz=171.8, num_updates=59700, lr=0.000258847, gnorm=1.382, train_wall=31, wall=18668\n",
            "2020-08-04 02:07:56 | INFO | train_inner | epoch 059:    930 / 1015 loss=3.993, nll_loss=2.338, ppl=5.06, wps=11565.6, ups=3.24, wpb=3569.9, bsz=153.2, num_updates=59800, lr=0.00025863, gnorm=1.41, train_wall=31, wall=18699\n",
            "2020-08-04 02:08:23 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:08:29 | INFO | valid | epoch 059 | valid on 'valid' subset | loss 4.272 | nll_loss 2.572 | ppl 5.95 | wps 27120.8 | wpb 2866.6 | bsz 127.8 | num_updates 59885 | best_loss 4.268\n",
            "2020-08-04 02:08:29 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:08:30 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint59.pt (epoch 59 @ 59885 updates, score 4.272) (writing took 1.496543790002761 seconds)\n",
            "2020-08-04 02:08:30 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:08:30 | INFO | fairseq_cli.train | end of epoch 59 (average epoch stats below)\n",
            "2020-08-04 02:08:30 | INFO | train | epoch 059 | loss 3.927 | nll_loss 2.262 | ppl 4.8 | wps 11268 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 59885 | lr 0.000258447 | gnorm 1.401 | train_wall 311 | wall 18732\n",
            "2020-08-04 02:08:30 | INFO | fairseq_cli.train | begin training epoch 59\n",
            "2020-08-04 02:08:35 | INFO | train_inner | epoch 060:     15 / 1015 loss=3.912, nll_loss=2.248, ppl=4.75, wps=9184.8, ups=2.59, wpb=3542.1, bsz=180, num_updates=59900, lr=0.000258414, gnorm=1.386, train_wall=31, wall=18737\n",
            "2020-08-04 02:09:06 | INFO | train_inner | epoch 060:    115 / 1015 loss=3.863, nll_loss=2.189, ppl=4.56, wps=11563.6, ups=3.28, wpb=3530, bsz=165.9, num_updates=60000, lr=0.000258199, gnorm=1.381, train_wall=30, wall=18768\n",
            "2020-08-04 02:09:36 | INFO | train_inner | epoch 060:    215 / 1015 loss=3.949, nll_loss=2.284, ppl=4.87, wps=11437.6, ups=3.26, wpb=3507.4, bsz=133.8, num_updates=60100, lr=0.000257984, gnorm=1.48, train_wall=31, wall=18798\n",
            "2020-08-04 02:10:07 | INFO | train_inner | epoch 060:    315 / 1015 loss=3.879, nll_loss=2.207, ppl=4.62, wps=11449.8, ups=3.23, wpb=3540.7, bsz=160.8, num_updates=60200, lr=0.00025777, gnorm=1.385, train_wall=31, wall=18829\n",
            "2020-08-04 02:10:38 | INFO | train_inner | epoch 060:    415 / 1015 loss=3.894, nll_loss=2.225, ppl=4.68, wps=11565.1, ups=3.27, wpb=3536.8, bsz=168.2, num_updates=60300, lr=0.000257556, gnorm=1.381, train_wall=30, wall=18860\n",
            "2020-08-04 02:11:08 | INFO | train_inner | epoch 060:    515 / 1015 loss=3.936, nll_loss=2.272, ppl=4.83, wps=11644.2, ups=3.26, wpb=3573.7, bsz=162.6, num_updates=60400, lr=0.000257343, gnorm=1.408, train_wall=31, wall=18891\n",
            "2020-08-04 02:11:39 | INFO | train_inner | epoch 060:    615 / 1015 loss=3.955, nll_loss=2.293, ppl=4.9, wps=11440.9, ups=3.26, wpb=3514.8, bsz=151.4, num_updates=60500, lr=0.00025713, gnorm=1.447, train_wall=31, wall=18921\n",
            "2020-08-04 02:12:10 | INFO | train_inner | epoch 060:    715 / 1015 loss=3.91, nll_loss=2.243, ppl=4.73, wps=11623.5, ups=3.25, wpb=3575.5, bsz=163.4, num_updates=60600, lr=0.000256917, gnorm=1.387, train_wall=31, wall=18952\n",
            "2020-08-04 02:12:41 | INFO | train_inner | epoch 060:    815 / 1015 loss=3.969, nll_loss=2.31, ppl=4.96, wps=11577.6, ups=3.23, wpb=3579.7, bsz=145, num_updates=60700, lr=0.000256706, gnorm=1.388, train_wall=31, wall=18983\n",
            "2020-08-04 02:13:12 | INFO | train_inner | epoch 060:    915 / 1015 loss=3.942, nll_loss=2.28, ppl=4.86, wps=11522.4, ups=3.25, wpb=3546.5, bsz=157.7, num_updates=60800, lr=0.000256495, gnorm=1.409, train_wall=31, wall=19014\n",
            "2020-08-04 02:13:42 | INFO | train_inner | epoch 060:   1015 / 1015 loss=3.956, nll_loss=2.296, ppl=4.91, wps=11671.5, ups=3.24, wpb=3602.5, bsz=165.6, num_updates=60900, lr=0.000256284, gnorm=1.39, train_wall=31, wall=19045\n",
            "2020-08-04 02:13:42 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:13:49 | INFO | valid | epoch 060 | valid on 'valid' subset | loss 4.284 | nll_loss 2.585 | ppl 6 | wps 26902.7 | wpb 2866.6 | bsz 127.8 | num_updates 60900 | best_loss 4.268\n",
            "2020-08-04 02:13:49 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:13:50 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint60.pt (epoch 60 @ 60900 updates, score 4.284) (writing took 1.5097320579989173 seconds)\n",
            "2020-08-04 02:13:50 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:13:50 | INFO | fairseq_cli.train | end of epoch 60 (average epoch stats below)\n",
            "2020-08-04 02:13:50 | INFO | train | epoch 060 | loss 3.923 | nll_loss 2.258 | ppl 4.78 | wps 11265.3 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 60900 | lr 0.000256284 | gnorm 1.405 | train_wall 311 | wall 19052\n",
            "2020-08-04 02:13:50 | INFO | fairseq_cli.train | begin training epoch 60\n",
            "2020-08-04 02:14:21 | INFO | train_inner | epoch 061:    100 / 1015 loss=3.848, nll_loss=2.173, ppl=4.51, wps=9213.4, ups=2.59, wpb=3551.7, bsz=168, num_updates=61000, lr=0.000256074, gnorm=1.36, train_wall=31, wall=19083\n",
            "2020-08-04 02:14:52 | INFO | train_inner | epoch 061:    200 / 1015 loss=3.94, nll_loss=2.275, ppl=4.84, wps=11505.2, ups=3.26, wpb=3533.9, bsz=139.5, num_updates=61100, lr=0.000255864, gnorm=1.433, train_wall=31, wall=19114\n",
            "2020-08-04 02:15:23 | INFO | train_inner | epoch 061:    300 / 1015 loss=3.893, nll_loss=2.222, ppl=4.67, wps=11594, ups=3.25, wpb=3569.6, bsz=154.2, num_updates=61200, lr=0.000255655, gnorm=1.406, train_wall=31, wall=19145\n",
            "2020-08-04 02:15:53 | INFO | train_inner | epoch 061:    400 / 1015 loss=3.889, nll_loss=2.219, ppl=4.66, wps=11724, ups=3.27, wpb=3589.2, bsz=162.1, num_updates=61300, lr=0.000255446, gnorm=1.373, train_wall=31, wall=19175\n",
            "2020-08-04 02:16:24 | INFO | train_inner | epoch 061:    500 / 1015 loss=3.926, nll_loss=2.259, ppl=4.79, wps=11306.8, ups=3.27, wpb=3457.8, bsz=151, num_updates=61400, lr=0.000255238, gnorm=1.456, train_wall=30, wall=19206\n",
            "2020-08-04 02:16:54 | INFO | train_inner | epoch 061:    600 / 1015 loss=3.939, nll_loss=2.276, ppl=4.84, wps=11556.6, ups=3.26, wpb=3549.6, bsz=156.8, num_updates=61500, lr=0.000255031, gnorm=1.42, train_wall=31, wall=19237\n",
            "2020-08-04 02:17:25 | INFO | train_inner | epoch 061:    700 / 1015 loss=3.932, nll_loss=2.268, ppl=4.82, wps=11663.7, ups=3.24, wpb=3595.5, bsz=156.1, num_updates=61600, lr=0.000254824, gnorm=1.377, train_wall=31, wall=19267\n",
            "2020-08-04 02:17:56 | INFO | train_inner | epoch 061:    800 / 1015 loss=3.929, nll_loss=2.265, ppl=4.81, wps=11499.3, ups=3.25, wpb=3541.1, bsz=163.1, num_updates=61700, lr=0.000254617, gnorm=1.428, train_wall=31, wall=19298\n",
            "2020-08-04 02:18:27 | INFO | train_inner | epoch 061:    900 / 1015 loss=3.958, nll_loss=2.298, ppl=4.92, wps=11559.8, ups=3.26, wpb=3541.6, bsz=157.4, num_updates=61800, lr=0.000254411, gnorm=1.431, train_wall=31, wall=19329\n",
            "2020-08-04 02:18:57 | INFO | train_inner | epoch 061:   1000 / 1015 loss=3.938, nll_loss=2.277, ppl=4.85, wps=11620, ups=3.25, wpb=3577.1, bsz=159.9, num_updates=61900, lr=0.000254205, gnorm=1.419, train_wall=31, wall=19360\n",
            "2020-08-04 02:19:02 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:19:08 | INFO | valid | epoch 061 | valid on 'valid' subset | loss 4.28 | nll_loss 2.58 | ppl 5.98 | wps 27025.3 | wpb 2866.6 | bsz 127.8 | num_updates 61915 | best_loss 4.268\n",
            "2020-08-04 02:19:08 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:19:10 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint61.pt (epoch 61 @ 61915 updates, score 4.28) (writing took 1.578238299000077 seconds)\n",
            "2020-08-04 02:19:10 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:19:10 | INFO | fairseq_cli.train | end of epoch 61 (average epoch stats below)\n",
            "2020-08-04 02:19:10 | INFO | train | epoch 061 | loss 3.917 | nll_loss 2.25 | ppl 4.76 | wps 11271.8 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 61915 | lr 0.000254175 | gnorm 1.41 | train_wall 311 | wall 19372\n",
            "2020-08-04 02:19:10 | INFO | fairseq_cli.train | begin training epoch 61\n",
            "2020-08-04 02:19:36 | INFO | train_inner | epoch 062:     85 / 1015 loss=3.777, nll_loss=2.095, ppl=4.27, wps=9195.7, ups=2.59, wpb=3546, bsz=198.7, num_updates=62000, lr=0.000254, gnorm=1.358, train_wall=31, wall=19398\n",
            "2020-08-04 02:20:07 | INFO | train_inner | epoch 062:    185 / 1015 loss=3.815, nll_loss=2.136, ppl=4.39, wps=11749.2, ups=3.25, wpb=3615.5, bsz=177.3, num_updates=62100, lr=0.000253796, gnorm=1.345, train_wall=31, wall=19429\n",
            "2020-08-04 02:20:38 | INFO | train_inner | epoch 062:    285 / 1015 loss=3.945, nll_loss=2.28, ppl=4.86, wps=11819.8, ups=3.26, wpb=3625.5, bsz=138, num_updates=62200, lr=0.000253592, gnorm=1.415, train_wall=31, wall=19460\n",
            "2020-08-04 02:21:08 | INFO | train_inner | epoch 062:    385 / 1015 loss=3.904, nll_loss=2.234, ppl=4.71, wps=11298.2, ups=3.24, wpb=3483.2, bsz=150, num_updates=62300, lr=0.000253388, gnorm=1.445, train_wall=31, wall=19490\n",
            "2020-08-04 02:21:39 | INFO | train_inner | epoch 062:    485 / 1015 loss=3.953, nll_loss=2.292, ppl=4.9, wps=11292.4, ups=3.28, wpb=3442.8, bsz=148.6, num_updates=62400, lr=0.000253185, gnorm=1.474, train_wall=30, wall=19521\n",
            "2020-08-04 02:22:10 | INFO | train_inner | epoch 062:    585 / 1015 loss=3.896, nll_loss=2.227, ppl=4.68, wps=11529.8, ups=3.25, wpb=3550.3, bsz=156.2, num_updates=62500, lr=0.000252982, gnorm=1.414, train_wall=31, wall=19552\n",
            "2020-08-04 02:22:40 | INFO | train_inner | epoch 062:    685 / 1015 loss=3.889, nll_loss=2.219, ppl=4.65, wps=11375, ups=3.25, wpb=3497.5, bsz=159.6, num_updates=62600, lr=0.00025278, gnorm=1.454, train_wall=31, wall=19582\n",
            "2020-08-04 02:23:11 | INFO | train_inner | epoch 062:    785 / 1015 loss=3.963, nll_loss=2.303, ppl=4.94, wps=11580.6, ups=3.25, wpb=3567.7, bsz=144.9, num_updates=62700, lr=0.000252578, gnorm=1.442, train_wall=31, wall=19613\n",
            "2020-08-04 02:23:42 | INFO | train_inner | epoch 062:    885 / 1015 loss=3.946, nll_loss=2.285, ppl=4.87, wps=11542.2, ups=3.24, wpb=3559.1, bsz=167, num_updates=62800, lr=0.000252377, gnorm=1.428, train_wall=31, wall=19644\n",
            "2020-08-04 02:24:13 | INFO | train_inner | epoch 062:    985 / 1015 loss=3.996, nll_loss=2.342, ppl=5.07, wps=11681.7, ups=3.25, wpb=3590.4, bsz=145.8, num_updates=62900, lr=0.000252177, gnorm=1.419, train_wall=31, wall=19675\n",
            "2020-08-04 02:24:22 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:24:28 | INFO | valid | epoch 062 | valid on 'valid' subset | loss 4.271 | nll_loss 2.569 | ppl 5.93 | wps 27044.2 | wpb 2866.6 | bsz 127.8 | num_updates 62930 | best_loss 4.268\n",
            "2020-08-04 02:24:28 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:24:30 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint62.pt (epoch 62 @ 62930 updates, score 4.271) (writing took 1.499708355000621 seconds)\n",
            "2020-08-04 02:24:30 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:24:30 | INFO | fairseq_cli.train | end of epoch 62 (average epoch stats below)\n",
            "2020-08-04 02:24:30 | INFO | train | epoch 062 | loss 3.912 | nll_loss 2.245 | ppl 4.74 | wps 11270.4 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 62930 | lr 0.000252116 | gnorm 1.419 | train_wall 311 | wall 19692\n",
            "2020-08-04 02:24:30 | INFO | fairseq_cli.train | begin training epoch 62\n",
            "2020-08-04 02:24:51 | INFO | train_inner | epoch 063:     70 / 1015 loss=3.849, nll_loss=2.175, ppl=4.52, wps=9311.7, ups=2.6, wpb=3585.4, bsz=171.1, num_updates=63000, lr=0.000251976, gnorm=1.39, train_wall=31, wall=19713\n",
            "2020-08-04 02:25:22 | INFO | train_inner | epoch 063:    170 / 1015 loss=3.85, nll_loss=2.175, ppl=4.51, wps=11361.8, ups=3.25, wpb=3490.9, bsz=154.6, num_updates=63100, lr=0.000251777, gnorm=1.432, train_wall=31, wall=19744\n",
            "2020-08-04 02:25:53 | INFO | train_inner | epoch 063:    270 / 1015 loss=3.899, nll_loss=2.228, ppl=4.68, wps=11501.6, ups=3.25, wpb=3544.4, bsz=147, num_updates=63200, lr=0.000251577, gnorm=1.447, train_wall=31, wall=19775\n",
            "2020-08-04 02:26:24 | INFO | train_inner | epoch 063:    370 / 1015 loss=3.901, nll_loss=2.232, ppl=4.7, wps=11618.4, ups=3.24, wpb=3585, bsz=151.3, num_updates=63300, lr=0.000251379, gnorm=1.421, train_wall=31, wall=19806\n",
            "2020-08-04 02:26:54 | INFO | train_inner | epoch 063:    470 / 1015 loss=3.9, nll_loss=2.231, ppl=4.69, wps=11540, ups=3.26, wpb=3536.3, bsz=157.3, num_updates=63400, lr=0.00025118, gnorm=1.437, train_wall=31, wall=19836\n",
            "2020-08-04 02:27:25 | INFO | train_inner | epoch 063:    570 / 1015 loss=3.927, nll_loss=2.262, ppl=4.8, wps=11750.5, ups=3.25, wpb=3618.5, bsz=154.3, num_updates=63500, lr=0.000250982, gnorm=1.399, train_wall=31, wall=19867\n",
            "2020-08-04 02:27:56 | INFO | train_inner | epoch 063:    670 / 1015 loss=3.924, nll_loss=2.26, ppl=4.79, wps=11616.6, ups=3.25, wpb=3569.2, bsz=168.2, num_updates=63600, lr=0.000250785, gnorm=1.43, train_wall=31, wall=19898\n",
            "2020-08-04 02:28:27 | INFO | train_inner | epoch 063:    770 / 1015 loss=3.926, nll_loss=2.262, ppl=4.8, wps=11557.6, ups=3.26, wpb=3549.1, bsz=156.6, num_updates=63700, lr=0.000250588, gnorm=1.432, train_wall=31, wall=19929\n",
            "2020-08-04 02:28:57 | INFO | train_inner | epoch 063:    870 / 1015 loss=3.946, nll_loss=2.285, ppl=4.87, wps=11626.3, ups=3.26, wpb=3567.2, bsz=154.9, num_updates=63800, lr=0.000250392, gnorm=1.443, train_wall=31, wall=19959\n",
            "2020-08-04 02:29:28 | INFO | train_inner | epoch 063:    970 / 1015 loss=3.923, nll_loss=2.259, ppl=4.79, wps=11421.6, ups=3.26, wpb=3506.1, bsz=165.8, num_updates=63900, lr=0.000250196, gnorm=1.426, train_wall=31, wall=19990\n",
            "2020-08-04 02:29:42 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:29:48 | INFO | valid | epoch 063 | valid on 'valid' subset | loss 4.277 | nll_loss 2.577 | ppl 5.97 | wps 27020 | wpb 2866.6 | bsz 127.8 | num_updates 63945 | best_loss 4.268\n",
            "2020-08-04 02:29:48 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:29:49 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint63.pt (epoch 63 @ 63945 updates, score 4.277) (writing took 1.6586262919990986 seconds)\n",
            "2020-08-04 02:29:49 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:29:49 | INFO | fairseq_cli.train | end of epoch 63 (average epoch stats below)\n",
            "2020-08-04 02:29:49 | INFO | train | epoch 063 | loss 3.908 | nll_loss 2.241 | ppl 4.73 | wps 11267.2 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 63945 | lr 0.000250107 | gnorm 1.43 | train_wall 311 | wall 20012\n",
            "2020-08-04 02:29:49 | INFO | fairseq_cli.train | begin training epoch 63\n",
            "2020-08-04 02:30:07 | INFO | train_inner | epoch 064:     55 / 1015 loss=3.896, nll_loss=2.229, ppl=4.69, wps=9200.3, ups=2.58, wpb=3565.3, bsz=166.8, num_updates=64000, lr=0.00025, gnorm=1.405, train_wall=31, wall=20029\n",
            "2020-08-04 02:30:38 | INFO | train_inner | epoch 064:    155 / 1015 loss=3.839, nll_loss=2.161, ppl=4.47, wps=11492.4, ups=3.24, wpb=3548.4, bsz=165.8, num_updates=64100, lr=0.000249805, gnorm=1.405, train_wall=31, wall=20060\n",
            "2020-08-04 02:31:08 | INFO | train_inner | epoch 064:    255 / 1015 loss=3.831, nll_loss=2.152, ppl=4.44, wps=11498.9, ups=3.23, wpb=3560.8, bsz=165.4, num_updates=64200, lr=0.00024961, gnorm=1.421, train_wall=31, wall=20091\n",
            "2020-08-04 02:31:39 | INFO | train_inner | epoch 064:    355 / 1015 loss=3.845, nll_loss=2.169, ppl=4.5, wps=11628.2, ups=3.23, wpb=3600.5, bsz=171, num_updates=64300, lr=0.000249416, gnorm=1.398, train_wall=31, wall=20122\n",
            "2020-08-04 02:32:10 | INFO | train_inner | epoch 064:    455 / 1015 loss=3.946, nll_loss=2.283, ppl=4.87, wps=11498.2, ups=3.27, wpb=3518.8, bsz=150.4, num_updates=64400, lr=0.000249222, gnorm=1.455, train_wall=30, wall=20152\n",
            "2020-08-04 02:32:41 | INFO | train_inner | epoch 064:    555 / 1015 loss=3.907, nll_loss=2.238, ppl=4.72, wps=11522.6, ups=3.25, wpb=3543.9, bsz=155.7, num_updates=64500, lr=0.000249029, gnorm=1.474, train_wall=31, wall=20183\n",
            "2020-08-04 02:33:12 | INFO | train_inner | epoch 064:    655 / 1015 loss=4.004, nll_loss=2.349, ppl=5.09, wps=11327.4, ups=3.26, wpb=3479.4, bsz=137.8, num_updates=64600, lr=0.000248836, gnorm=1.552, train_wall=31, wall=20214\n",
            "2020-08-04 02:33:42 | INFO | train_inner | epoch 064:    755 / 1015 loss=3.935, nll_loss=2.272, ppl=4.83, wps=11639.2, ups=3.25, wpb=3584.7, bsz=161, num_updates=64700, lr=0.000248644, gnorm=1.42, train_wall=31, wall=20244\n",
            "2020-08-04 02:34:13 | INFO | train_inner | epoch 064:    855 / 1015 loss=3.954, nll_loss=2.293, ppl=4.9, wps=11374.1, ups=3.27, wpb=3481, bsz=151.1, num_updates=64800, lr=0.000248452, gnorm=1.473, train_wall=30, wall=20275\n",
            "2020-08-04 02:34:44 | INFO | train_inner | epoch 064:    955 / 1015 loss=3.948, nll_loss=2.286, ppl=4.88, wps=11672.4, ups=3.25, wpb=3589.4, bsz=149.6, num_updates=64900, lr=0.000248261, gnorm=1.469, train_wall=31, wall=20306\n",
            "2020-08-04 02:35:02 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:35:08 | INFO | valid | epoch 064 | valid on 'valid' subset | loss 4.28 | nll_loss 2.579 | ppl 5.98 | wps 26914.2 | wpb 2866.6 | bsz 127.8 | num_updates 64960 | best_loss 4.268\n",
            "2020-08-04 02:35:08 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:35:10 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint64.pt (epoch 64 @ 64960 updates, score 4.28) (writing took 1.6322140400006901 seconds)\n",
            "2020-08-04 02:35:10 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:35:10 | INFO | fairseq_cli.train | end of epoch 64 (average epoch stats below)\n",
            "2020-08-04 02:35:10 | INFO | train | epoch 064 | loss 3.904 | nll_loss 2.236 | ppl 4.71 | wps 11243.9 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 64960 | lr 0.000248146 | gnorm 1.444 | train_wall 311 | wall 20332\n",
            "2020-08-04 02:35:10 | INFO | fairseq_cli.train | begin training epoch 64\n",
            "2020-08-04 02:35:22 | INFO | train_inner | epoch 065:     40 / 1015 loss=3.86, nll_loss=2.185, ppl=4.55, wps=9167.4, ups=2.6, wpb=3530.7, bsz=158.7, num_updates=65000, lr=0.000248069, gnorm=1.433, train_wall=31, wall=20344\n",
            "2020-08-04 02:35:53 | INFO | train_inner | epoch 065:    140 / 1015 loss=3.828, nll_loss=2.149, ppl=4.44, wps=11394.7, ups=3.24, wpb=3512, bsz=162.6, num_updates=65100, lr=0.000247879, gnorm=1.445, train_wall=31, wall=20375\n",
            "2020-08-04 02:36:23 | INFO | train_inner | epoch 065:    240 / 1015 loss=3.933, nll_loss=2.267, ppl=4.81, wps=11393.5, ups=3.29, wpb=3460.9, bsz=134.3, num_updates=65200, lr=0.000247689, gnorm=1.491, train_wall=30, wall=20406\n",
            "2020-08-04 02:36:54 | INFO | train_inner | epoch 065:    340 / 1015 loss=3.908, nll_loss=2.24, ppl=4.72, wps=11509.1, ups=3.28, wpb=3513.3, bsz=153.8, num_updates=65300, lr=0.000247499, gnorm=1.469, train_wall=30, wall=20436\n",
            "2020-08-04 02:37:25 | INFO | train_inner | epoch 065:    440 / 1015 loss=3.893, nll_loss=2.223, ppl=4.67, wps=11427.8, ups=3.25, wpb=3512.6, bsz=163.9, num_updates=65400, lr=0.00024731, gnorm=1.433, train_wall=31, wall=20467\n",
            "2020-08-04 02:37:55 | INFO | train_inner | epoch 065:    540 / 1015 loss=3.923, nll_loss=2.258, ppl=4.78, wps=11680.9, ups=3.25, wpb=3592.8, bsz=150.2, num_updates=65500, lr=0.000247121, gnorm=1.425, train_wall=31, wall=20498\n",
            "2020-08-04 02:38:26 | INFO | train_inner | epoch 065:    640 / 1015 loss=3.89, nll_loss=2.221, ppl=4.66, wps=11638.6, ups=3.24, wpb=3587.7, bsz=165.4, num_updates=65600, lr=0.000246932, gnorm=1.422, train_wall=31, wall=20528\n",
            "2020-08-04 02:38:57 | INFO | train_inner | epoch 065:    740 / 1015 loss=3.875, nll_loss=2.204, ppl=4.61, wps=11602.6, ups=3.27, wpb=3552, bsz=168.2, num_updates=65700, lr=0.000246744, gnorm=1.424, train_wall=31, wall=20559\n",
            "2020-08-04 02:39:28 | INFO | train_inner | epoch 065:    840 / 1015 loss=3.893, nll_loss=2.225, ppl=4.67, wps=11718.5, ups=3.26, wpb=3593.3, bsz=174.9, num_updates=65800, lr=0.000246557, gnorm=1.457, train_wall=31, wall=20590\n",
            "2020-08-04 02:39:58 | INFO | train_inner | epoch 065:    940 / 1015 loss=3.933, nll_loss=2.27, ppl=4.82, wps=11674.2, ups=3.24, wpb=3601, bsz=157.4, num_updates=65900, lr=0.00024637, gnorm=1.437, train_wall=31, wall=20620\n",
            "2020-08-04 02:40:21 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:40:27 | INFO | valid | epoch 065 | valid on 'valid' subset | loss 4.28 | nll_loss 2.587 | ppl 6.01 | wps 26926.4 | wpb 2866.6 | bsz 127.8 | num_updates 65975 | best_loss 4.268\n",
            "2020-08-04 02:40:27 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:40:29 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint65.pt (epoch 65 @ 65975 updates, score 4.28) (writing took 1.5732920599984936 seconds)\n",
            "2020-08-04 02:40:29 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:40:29 | INFO | fairseq_cli.train | end of epoch 65 (average epoch stats below)\n",
            "2020-08-04 02:40:29 | INFO | train | epoch 065 | loss 3.898 | nll_loss 2.23 | ppl 4.69 | wps 11287.7 | ups 3.18 | wpb 3550.1 | bsz 157.9 | num_updates 65975 | lr 0.00024623 | gnorm 1.443 | train_wall 310 | wall 20651\n",
            "2020-08-04 02:40:29 | INFO | fairseq_cli.train | begin training epoch 65\n",
            "2020-08-04 02:40:37 | INFO | train_inner | epoch 066:     25 / 1015 loss=3.912, nll_loss=2.246, ppl=4.74, wps=9269.7, ups=2.59, wpb=3579.4, bsz=157.2, num_updates=66000, lr=0.000246183, gnorm=1.423, train_wall=31, wall=20659\n",
            "2020-08-04 02:41:08 | INFO | train_inner | epoch 066:    125 / 1015 loss=3.785, nll_loss=2.103, ppl=4.29, wps=11605.6, ups=3.24, wpb=3576.6, bsz=184.2, num_updates=66100, lr=0.000245997, gnorm=1.383, train_wall=31, wall=20690\n",
            "2020-08-04 02:41:38 | INFO | train_inner | epoch 066:    225 / 1015 loss=3.861, nll_loss=2.186, ppl=4.55, wps=11392.3, ups=3.26, wpb=3495.2, bsz=158.4, num_updates=66200, lr=0.000245811, gnorm=1.511, train_wall=31, wall=20721\n",
            "2020-08-04 02:42:09 | INFO | train_inner | epoch 066:    325 / 1015 loss=3.933, nll_loss=2.267, ppl=4.81, wps=11611.9, ups=3.3, wpb=3517.3, bsz=130.4, num_updates=66300, lr=0.000245625, gnorm=1.493, train_wall=30, wall=20751\n",
            "2020-08-04 02:42:39 | INFO | train_inner | epoch 066:    425 / 1015 loss=3.937, nll_loss=2.272, ppl=4.83, wps=11625, ups=3.28, wpb=3546.6, bsz=141.4, num_updates=66400, lr=0.00024544, gnorm=1.469, train_wall=30, wall=20781\n",
            "2020-08-04 02:43:10 | INFO | train_inner | epoch 066:    525 / 1015 loss=3.898, nll_loss=2.229, ppl=4.69, wps=11606.5, ups=3.24, wpb=3577.4, bsz=153.7, num_updates=66500, lr=0.000245256, gnorm=1.446, train_wall=31, wall=20812\n",
            "2020-08-04 02:43:41 | INFO | train_inner | epoch 066:    625 / 1015 loss=3.92, nll_loss=2.254, ppl=4.77, wps=11515, ups=3.27, wpb=3524.7, bsz=157.8, num_updates=66600, lr=0.000245072, gnorm=1.483, train_wall=30, wall=20843\n",
            "2020-08-04 02:44:11 | INFO | train_inner | epoch 066:    725 / 1015 loss=3.917, nll_loss=2.251, ppl=4.76, wps=11579.4, ups=3.27, wpb=3545.9, bsz=161.4, num_updates=66700, lr=0.000244888, gnorm=1.452, train_wall=31, wall=20873\n",
            "2020-08-04 02:44:42 | INFO | train_inner | epoch 066:    825 / 1015 loss=3.897, nll_loss=2.229, ppl=4.69, wps=11589.7, ups=3.22, wpb=3594.5, bsz=165.4, num_updates=66800, lr=0.000244704, gnorm=1.419, train_wall=31, wall=20904\n",
            "2020-08-04 02:45:13 | INFO | train_inner | epoch 066:    925 / 1015 loss=3.905, nll_loss=2.239, ppl=4.72, wps=11731.7, ups=3.24, wpb=3620.9, bsz=161.3, num_updates=66900, lr=0.000244521, gnorm=1.399, train_wall=31, wall=20935\n",
            "2020-08-04 02:45:41 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:45:47 | INFO | valid | epoch 066 | valid on 'valid' subset | loss 4.267 | nll_loss 2.573 | ppl 5.95 | wps 26980.3 | wpb 2866.6 | bsz 127.8 | num_updates 66990 | best_loss 4.267\n",
            "2020-08-04 02:45:47 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:45:50 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint66.pt (epoch 66 @ 66990 updates, score 4.267) (writing took 2.69097003600109 seconds)\n",
            "2020-08-04 02:45:50 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:45:50 | INFO | fairseq_cli.train | end of epoch 66 (average epoch stats below)\n",
            "2020-08-04 02:45:50 | INFO | train | epoch 066 | loss 3.895 | nll_loss 2.226 | ppl 4.68 | wps 11239.6 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 66990 | lr 0.000244357 | gnorm 1.455 | train_wall 311 | wall 20972\n",
            "2020-08-04 02:45:50 | INFO | fairseq_cli.train | begin training epoch 66\n",
            "2020-08-04 02:45:53 | INFO | train_inner | epoch 067:     10 / 1015 loss=3.911, nll_loss=2.246, ppl=4.74, wps=8858.4, ups=2.52, wpb=3514.5, bsz=162.6, num_updates=67000, lr=0.000244339, gnorm=1.484, train_wall=31, wall=20975\n",
            "2020-08-04 02:46:24 | INFO | train_inner | epoch 067:    110 / 1015 loss=3.799, nll_loss=2.117, ppl=4.34, wps=11652.7, ups=3.24, wpb=3600.8, bsz=172.3, num_updates=67100, lr=0.000244157, gnorm=1.376, train_wall=31, wall=21006\n",
            "2020-08-04 02:46:55 | INFO | train_inner | epoch 067:    210 / 1015 loss=3.858, nll_loss=2.184, ppl=4.54, wps=11706.6, ups=3.25, wpb=3605.1, bsz=163.2, num_updates=67200, lr=0.000243975, gnorm=1.442, train_wall=31, wall=21037\n",
            "2020-08-04 02:47:25 | INFO | train_inner | epoch 067:    310 / 1015 loss=3.898, nll_loss=2.228, ppl=4.68, wps=11516.5, ups=3.26, wpb=3535.8, bsz=144.7, num_updates=67300, lr=0.000243794, gnorm=1.474, train_wall=31, wall=21067\n",
            "2020-08-04 02:47:56 | INFO | train_inner | epoch 067:    410 / 1015 loss=3.88, nll_loss=2.208, ppl=4.62, wps=11571.3, ups=3.26, wpb=3548.8, bsz=157.9, num_updates=67400, lr=0.000243613, gnorm=1.473, train_wall=31, wall=21098\n",
            "2020-08-04 02:48:27 | INFO | train_inner | epoch 067:    510 / 1015 loss=3.918, nll_loss=2.252, ppl=4.76, wps=11451.8, ups=3.26, wpb=3516.5, bsz=158.3, num_updates=67500, lr=0.000243432, gnorm=1.493, train_wall=31, wall=21129\n",
            "2020-08-04 02:48:57 | INFO | train_inner | epoch 067:    610 / 1015 loss=3.898, nll_loss=2.229, ppl=4.69, wps=11631.7, ups=3.25, wpb=3579.7, bsz=157.3, num_updates=67600, lr=0.000243252, gnorm=1.437, train_wall=31, wall=21160\n",
            "2020-08-04 02:49:28 | INFO | train_inner | epoch 067:    710 / 1015 loss=3.92, nll_loss=2.254, ppl=4.77, wps=11302.6, ups=3.26, wpb=3466.9, bsz=151.6, num_updates=67700, lr=0.000243072, gnorm=1.548, train_wall=31, wall=21190\n",
            "2020-08-04 02:49:59 | INFO | train_inner | epoch 067:    810 / 1015 loss=3.898, nll_loss=2.23, ppl=4.69, wps=11585.7, ups=3.24, wpb=3573.3, bsz=158.6, num_updates=67800, lr=0.000242893, gnorm=1.448, train_wall=31, wall=21221\n",
            "2020-08-04 02:50:30 | INFO | train_inner | epoch 067:    910 / 1015 loss=3.948, nll_loss=2.286, ppl=4.88, wps=11473.4, ups=3.27, wpb=3512.2, bsz=146.7, num_updates=67900, lr=0.000242714, gnorm=1.489, train_wall=31, wall=21252\n",
            "2020-08-04 02:51:00 | INFO | train_inner | epoch 067:   1010 / 1015 loss=3.906, nll_loss=2.241, ppl=4.73, wps=11584.7, ups=3.25, wpb=3559.9, bsz=166.6, num_updates=68000, lr=0.000242536, gnorm=1.433, train_wall=31, wall=21282\n",
            "2020-08-04 02:51:02 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:51:08 | INFO | valid | epoch 067 | valid on 'valid' subset | loss 4.259 | nll_loss 2.556 | ppl 5.88 | wps 26895.1 | wpb 2866.6 | bsz 127.8 | num_updates 68005 | best_loss 4.259\n",
            "2020-08-04 02:51:08 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:51:11 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint67.pt (epoch 67 @ 68005 updates, score 4.259) (writing took 2.585023256000568 seconds)\n",
            "2020-08-04 02:51:11 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:51:11 | INFO | fairseq_cli.train | end of epoch 67 (average epoch stats below)\n",
            "2020-08-04 02:51:11 | INFO | train | epoch 067 | loss 3.891 | nll_loss 2.221 | ppl 4.66 | wps 11230.2 | ups 3.16 | wpb 3550.1 | bsz 157.9 | num_updates 68005 | lr 0.000242527 | gnorm 1.46 | train_wall 311 | wall 21293\n",
            "2020-08-04 02:51:11 | INFO | fairseq_cli.train | begin training epoch 67\n",
            "2020-08-04 02:51:40 | INFO | train_inner | epoch 068:     95 / 1015 loss=3.844, nll_loss=2.167, ppl=4.49, wps=8838.4, ups=2.54, wpb=3481, bsz=152.2, num_updates=68100, lr=0.000242357, gnorm=1.491, train_wall=30, wall=21322\n",
            "2020-08-04 02:52:11 | INFO | train_inner | epoch 068:    195 / 1015 loss=3.839, nll_loss=2.161, ppl=4.47, wps=11557.5, ups=3.23, wpb=3581.1, bsz=153.6, num_updates=68200, lr=0.00024218, gnorm=1.473, train_wall=31, wall=21353\n",
            "2020-08-04 02:52:41 | INFO | train_inner | epoch 068:    295 / 1015 loss=3.88, nll_loss=2.209, ppl=4.62, wps=11711.4, ups=3.28, wpb=3569.1, bsz=157, num_updates=68300, lr=0.000242002, gnorm=1.47, train_wall=30, wall=21383\n",
            "2020-08-04 02:53:12 | INFO | train_inner | epoch 068:    395 / 1015 loss=3.843, nll_loss=2.168, ppl=4.49, wps=11549.5, ups=3.26, wpb=3544.3, bsz=174.7, num_updates=68400, lr=0.000241825, gnorm=1.444, train_wall=31, wall=21414\n",
            "2020-08-04 02:53:43 | INFO | train_inner | epoch 068:    495 / 1015 loss=3.897, nll_loss=2.228, ppl=4.69, wps=11781.9, ups=3.26, wpb=3618.8, bsz=152.3, num_updates=68500, lr=0.000241649, gnorm=1.441, train_wall=31, wall=21445\n",
            "2020-08-04 02:54:13 | INFO | train_inner | epoch 068:    595 / 1015 loss=3.863, nll_loss=2.191, ppl=4.57, wps=11568.8, ups=3.24, wpb=3570.4, bsz=174.1, num_updates=68600, lr=0.000241473, gnorm=1.463, train_wall=31, wall=21476\n",
            "2020-08-04 02:54:44 | INFO | train_inner | epoch 068:    695 / 1015 loss=3.848, nll_loss=2.174, ppl=4.51, wps=11640.7, ups=3.23, wpb=3598.9, bsz=177.8, num_updates=68700, lr=0.000241297, gnorm=1.393, train_wall=31, wall=21506\n",
            "2020-08-04 02:55:15 | INFO | train_inner | epoch 068:    795 / 1015 loss=3.96, nll_loss=2.298, ppl=4.92, wps=11484.3, ups=3.25, wpb=3529.1, bsz=138.1, num_updates=68800, lr=0.000241121, gnorm=1.481, train_wall=31, wall=21537\n",
            "2020-08-04 02:55:46 | INFO | train_inner | epoch 068:    895 / 1015 loss=3.943, nll_loss=2.281, ppl=4.86, wps=11545.8, ups=3.25, wpb=3554.9, bsz=151, num_updates=68900, lr=0.000240946, gnorm=1.479, train_wall=31, wall=21568\n",
            "2020-08-04 02:56:17 | INFO | train_inner | epoch 068:    995 / 1015 loss=3.958, nll_loss=2.297, ppl=4.92, wps=11352.8, ups=3.26, wpb=3485.9, bsz=143.6, num_updates=69000, lr=0.000240772, gnorm=1.529, train_wall=31, wall=21599\n",
            "2020-08-04 02:56:23 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 02:56:29 | INFO | valid | epoch 068 | valid on 'valid' subset | loss 4.273 | nll_loss 2.579 | ppl 5.97 | wps 27010.6 | wpb 2866.6 | bsz 127.8 | num_updates 69020 | best_loss 4.259\n",
            "2020-08-04 02:56:29 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 02:56:30 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint68.pt (epoch 68 @ 69020 updates, score 4.273) (writing took 1.5405759599998419 seconds)\n",
            "2020-08-04 02:56:30 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 02:56:30 | INFO | fairseq_cli.train | end of epoch 68 (average epoch stats below)\n",
            "2020-08-04 02:56:30 | INFO | train | epoch 068 | loss 3.887 | nll_loss 2.217 | ppl 4.65 | wps 11269.6 | ups 3.17 | wpb 3550.1 | bsz 157.9 | num_updates 69020 | lr 0.000240737 | gnorm 1.467 | train_wall 311 | wall 21612\n",
            "2020-08-04 02:56:30 | INFO | fairseq_cli.train | begin training epoch 68\n",
            "2020-08-04 02:56:55 | INFO | train_inner | epoch 069:     80 / 1015 loss=3.843, nll_loss=2.168, ppl=4.49, wps=9138.1, ups=2.59, wpb=3527.8, bsz=164.2, num_updates=69100, lr=0.000240597, gnorm=1.46, train_wall=31, wall=21637\n",
            "2020-08-04 02:57:26 | INFO | train_inner | epoch 069:    180 / 1015 loss=3.849, nll_loss=2.172, ppl=4.51, wps=11504.9, ups=3.24, wpb=3546.2, bsz=150.9, num_updates=69200, lr=0.000240424, gnorm=1.465, train_wall=31, wall=21668\n",
            "2020-08-04 02:57:57 | INFO | train_inner | epoch 069:    280 / 1015 loss=3.874, nll_loss=2.201, ppl=4.6, wps=11553.4, ups=3.27, wpb=3532.3, bsz=154.6, num_updates=69300, lr=0.00024025, gnorm=1.481, train_wall=30, wall=21699\n",
            "2020-08-04 02:58:27 | INFO | train_inner | epoch 069:    380 / 1015 loss=3.873, nll_loss=2.2, ppl=4.6, wps=11641.8, ups=3.25, wpb=3587.4, bsz=153.1, num_updates=69400, lr=0.000240077, gnorm=1.472, train_wall=31, wall=21729\n",
            "2020-08-04 02:58:58 | INFO | train_inner | epoch 069:    480 / 1015 loss=3.827, nll_loss=2.151, ppl=4.44, wps=11657.4, ups=3.25, wpb=3590.8, bsz=177.9, num_updates=69500, lr=0.000239904, gnorm=1.438, train_wall=31, wall=21760\n",
            "2020-08-04 02:59:29 | INFO | train_inner | epoch 069:    580 / 1015 loss=3.89, nll_loss=2.22, ppl=4.66, wps=11553.5, ups=3.26, wpb=3549.3, bsz=162.5, num_updates=69600, lr=0.000239732, gnorm=1.492, train_wall=31, wall=21791\n",
            "2020-08-04 03:00:00 | INFO | train_inner | epoch 069:    680 / 1015 loss=3.891, nll_loss=2.221, ppl=4.66, wps=11394.3, ups=3.25, wpb=3504.9, bsz=158.7, num_updates=69700, lr=0.00023956, gnorm=1.469, train_wall=31, wall=21822\n",
            "2020-08-04 03:00:30 | INFO | train_inner | epoch 069:    780 / 1015 loss=3.907, nll_loss=2.241, ppl=4.73, wps=11598.4, ups=3.26, wpb=3554.7, bsz=157.8, num_updates=69800, lr=0.000239388, gnorm=1.479, train_wall=31, wall=21852\n",
            "2020-08-04 03:01:01 | INFO | train_inner | epoch 069:    880 / 1015 loss=3.966, nll_loss=2.306, ppl=4.95, wps=11512, ups=3.26, wpb=3531.5, bsz=140.7, num_updates=69900, lr=0.000239217, gnorm=1.515, train_wall=31, wall=21883\n",
            "2020-08-04 03:01:32 | INFO | train_inner | epoch 069:    980 / 1015 loss=3.893, nll_loss=2.226, ppl=4.68, wps=11509, ups=3.24, wpb=3549.7, bsz=166.5, num_updates=70000, lr=0.000239046, gnorm=1.474, train_wall=31, wall=21914\n",
            "2020-08-04 03:01:43 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:01:49 | INFO | valid | epoch 069 | valid on 'valid' subset | loss 4.259 | nll_loss 2.564 | ppl 5.92 | wps 27062.3 | wpb 2866.6 | bsz 127.8 | num_updates 70035 | best_loss 4.259\n",
            "2020-08-04 03:01:49 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:01:51 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint69.pt (epoch 69 @ 70035 updates, score 4.259) (writing took 2.724310294001043 seconds)\n",
            "2020-08-04 03:01:51 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:01:51 | INFO | fairseq_cli.train | end of epoch 69 (average epoch stats below)\n",
            "2020-08-04 03:01:51 | INFO | train | epoch 069 | loss 3.883 | nll_loss 2.212 | ppl 4.63 | wps 11222.1 | ups 3.16 | wpb 3550.1 | bsz 157.9 | num_updates 70035 | lr 0.000238986 | gnorm 1.476 | train_wall 311 | wall 21934\n",
            "2020-08-04 03:01:51 | INFO | fairseq_cli.train | begin training epoch 69\n",
            "2020-08-04 03:02:11 | INFO | train_inner | epoch 070:     65 / 1015 loss=3.913, nll_loss=2.245, ppl=4.74, wps=8972.3, ups=2.54, wpb=3533.7, bsz=141.4, num_updates=70100, lr=0.000238875, gnorm=1.496, train_wall=30, wall=21953\n",
            "2020-08-04 03:02:42 | INFO | train_inner | epoch 070:    165 / 1015 loss=3.873, nll_loss=2.2, ppl=4.59, wps=11654.9, ups=3.26, wpb=3574.2, bsz=143.1, num_updates=70200, lr=0.000238705, gnorm=1.462, train_wall=31, wall=21984\n",
            "2020-08-04 03:03:13 | INFO | train_inner | epoch 070:    265 / 1015 loss=3.874, nll_loss=2.201, ppl=4.6, wps=11525.8, ups=3.26, wpb=3539.7, bsz=156, num_updates=70300, lr=0.000238535, gnorm=1.528, train_wall=31, wall=22015\n",
            "2020-08-04 03:03:43 | INFO | train_inner | epoch 070:    365 / 1015 loss=3.849, nll_loss=2.173, ppl=4.51, wps=11481.3, ups=3.26, wpb=3525.3, bsz=155, num_updates=70400, lr=0.000238366, gnorm=1.469, train_wall=31, wall=22045\n",
            "2020-08-04 03:04:14 | INFO | train_inner | epoch 070:    465 / 1015 loss=3.861, nll_loss=2.187, ppl=4.55, wps=11641.3, ups=3.25, wpb=3579.1, bsz=164.2, num_updates=70500, lr=0.000238197, gnorm=1.436, train_wall=31, wall=22076\n",
            "2020-08-04 03:04:45 | INFO | train_inner | epoch 070:    565 / 1015 loss=3.884, nll_loss=2.214, ppl=4.64, wps=11624.7, ups=3.28, wpb=3548.1, bsz=163.3, num_updates=70600, lr=0.000238028, gnorm=1.497, train_wall=30, wall=22107\n",
            "2020-08-04 03:05:15 | INFO | train_inner | epoch 070:    665 / 1015 loss=3.892, nll_loss=2.223, ppl=4.67, wps=11666.8, ups=3.25, wpb=3589, bsz=167.1, num_updates=70700, lr=0.000237859, gnorm=1.52, train_wall=31, wall=22137\n",
            "2020-08-04 03:05:46 | INFO | train_inner | epoch 070:    765 / 1015 loss=3.882, nll_loss=2.212, ppl=4.63, wps=11697.1, ups=3.26, wpb=3582.9, bsz=158.3, num_updates=70800, lr=0.000237691, gnorm=1.463, train_wall=31, wall=22168\n",
            "2020-08-04 03:06:16 | INFO | train_inner | epoch 070:    865 / 1015 loss=3.906, nll_loss=2.239, ppl=4.72, wps=11488.8, ups=3.29, wpb=3491.1, bsz=156.7, num_updates=70900, lr=0.000237524, gnorm=1.537, train_wall=30, wall=22198\n",
            "2020-08-04 03:06:47 | INFO | train_inner | epoch 070:    965 / 1015 loss=3.906, nll_loss=2.239, ppl=4.72, wps=11602.6, ups=3.27, wpb=3550.2, bsz=156.6, num_updates=71000, lr=0.000237356, gnorm=1.504, train_wall=30, wall=22229\n",
            "2020-08-04 03:07:02 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:07:08 | INFO | valid | epoch 070 | valid on 'valid' subset | loss 4.277 | nll_loss 2.57 | ppl 5.94 | wps 27066 | wpb 2866.6 | bsz 127.8 | num_updates 71050 | best_loss 4.259\n",
            "2020-08-04 03:07:08 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:07:10 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint70.pt (epoch 70 @ 71050 updates, score 4.277) (writing took 1.6489478719995532 seconds)\n",
            "2020-08-04 03:07:10 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:07:10 | INFO | fairseq_cli.train | end of epoch 70 (average epoch stats below)\n",
            "2020-08-04 03:07:10 | INFO | train | epoch 070 | loss 3.88 | nll_loss 2.209 | ppl 4.62 | wps 11313 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 71050 | lr 0.000237273 | gnorm 1.488 | train_wall 310 | wall 22252\n",
            "2020-08-04 03:07:10 | INFO | fairseq_cli.train | begin training epoch 70\n",
            "2020-08-04 03:07:25 | INFO | train_inner | epoch 071:     50 / 1015 loss=3.856, nll_loss=2.183, ppl=4.54, wps=9256.3, ups=2.61, wpb=3544.2, bsz=156.6, num_updates=71100, lr=0.000237189, gnorm=1.457, train_wall=30, wall=22267\n",
            "2020-08-04 03:07:56 | INFO | train_inner | epoch 071:    150 / 1015 loss=3.855, nll_loss=2.178, ppl=4.53, wps=11518.9, ups=3.29, wpb=3497.6, bsz=143.2, num_updates=71200, lr=0.000237023, gnorm=1.523, train_wall=30, wall=22298\n",
            "2020-08-04 03:08:26 | INFO | train_inner | epoch 071:    250 / 1015 loss=3.825, nll_loss=2.146, ppl=4.42, wps=11532.7, ups=3.25, wpb=3549.6, bsz=154.6, num_updates=71300, lr=0.000236856, gnorm=1.454, train_wall=31, wall=22329\n",
            "2020-08-04 03:08:57 | INFO | train_inner | epoch 071:    350 / 1015 loss=3.833, nll_loss=2.155, ppl=4.45, wps=11647.4, ups=3.28, wpb=3551.1, bsz=161.7, num_updates=71400, lr=0.000236691, gnorm=1.496, train_wall=30, wall=22359\n",
            "2020-08-04 03:09:27 | INFO | train_inner | epoch 071:    450 / 1015 loss=3.886, nll_loss=2.216, ppl=4.64, wps=11945.6, ups=3.28, wpb=3646.5, bsz=150.2, num_updates=71500, lr=0.000236525, gnorm=1.435, train_wall=30, wall=22390\n",
            "2020-08-04 03:09:58 | INFO | train_inner | epoch 071:    550 / 1015 loss=3.888, nll_loss=2.219, ppl=4.65, wps=11727.3, ups=3.29, wpb=3568, bsz=157.5, num_updates=71600, lr=0.00023636, gnorm=1.509, train_wall=30, wall=22420\n",
            "2020-08-04 03:10:28 | INFO | train_inner | epoch 071:    650 / 1015 loss=3.888, nll_loss=2.218, ppl=4.65, wps=11522, ups=3.29, wpb=3501.7, bsz=159.6, num_updates=71700, lr=0.000236195, gnorm=1.492, train_wall=30, wall=22450\n",
            "2020-08-04 03:10:58 | INFO | train_inner | epoch 071:    750 / 1015 loss=3.877, nll_loss=2.207, ppl=4.62, wps=11582.5, ups=3.31, wpb=3501.3, bsz=177.2, num_updates=71800, lr=0.00023603, gnorm=1.514, train_wall=30, wall=22481\n",
            "2020-08-04 03:11:29 | INFO | train_inner | epoch 071:    850 / 1015 loss=3.903, nll_loss=2.236, ppl=4.71, wps=11581, ups=3.29, wpb=3519.9, bsz=157.6, num_updates=71900, lr=0.000235866, gnorm=1.521, train_wall=30, wall=22511\n",
            "2020-08-04 03:11:59 | INFO | train_inner | epoch 071:    950 / 1015 loss=3.929, nll_loss=2.266, ppl=4.81, wps=11731.3, ups=3.28, wpb=3574.1, bsz=157.8, num_updates=72000, lr=0.000235702, gnorm=1.518, train_wall=30, wall=22541\n",
            "2020-08-04 03:12:19 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:12:25 | INFO | valid | epoch 071 | valid on 'valid' subset | loss 4.247 | nll_loss 2.551 | ppl 5.86 | wps 27100.1 | wpb 2866.6 | bsz 127.8 | num_updates 72065 | best_loss 4.247\n",
            "2020-08-04 03:12:25 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:12:28 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint71.pt (epoch 71 @ 72065 updates, score 4.247) (writing took 2.527211772998271 seconds)\n",
            "2020-08-04 03:12:28 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:12:28 | INFO | fairseq_cli.train | end of epoch 71 (average epoch stats below)\n",
            "2020-08-04 03:12:28 | INFO | train | epoch 071 | loss 3.876 | nll_loss 2.205 | ppl 4.61 | wps 11336.1 | ups 3.19 | wpb 3550.1 | bsz 157.9 | num_updates 72065 | lr 0.000235596 | gnorm 1.491 | train_wall 308 | wall 22570\n",
            "2020-08-04 03:12:28 | INFO | fairseq_cli.train | begin training epoch 71\n",
            "2020-08-04 03:12:39 | INFO | train_inner | epoch 072:     35 / 1015 loss=3.843, nll_loss=2.169, ppl=4.5, wps=9163.3, ups=2.55, wpb=3600.4, bsz=176.2, num_updates=72100, lr=0.000235539, gnorm=1.444, train_wall=30, wall=22581\n",
            "2020-08-04 03:13:09 | INFO | train_inner | epoch 072:    135 / 1015 loss=3.823, nll_loss=2.143, ppl=4.42, wps=11757.8, ups=3.27, wpb=3598.1, bsz=144.5, num_updates=72200, lr=0.000235376, gnorm=1.443, train_wall=31, wall=22611\n",
            "2020-08-04 03:13:39 | INFO | train_inner | epoch 072:    235 / 1015 loss=3.828, nll_loss=2.15, ppl=4.44, wps=11535.9, ups=3.33, wpb=3464.8, bsz=170.2, num_updates=72300, lr=0.000235213, gnorm=1.53, train_wall=30, wall=22641\n",
            "2020-08-04 03:14:10 | INFO | train_inner | epoch 072:    335 / 1015 loss=3.889, nll_loss=2.218, ppl=4.65, wps=11733.9, ups=3.28, wpb=3572.3, bsz=142.5, num_updates=72400, lr=0.00023505, gnorm=1.496, train_wall=30, wall=22672\n",
            "2020-08-04 03:14:40 | INFO | train_inner | epoch 072:    435 / 1015 loss=3.842, nll_loss=2.166, ppl=4.49, wps=11630.3, ups=3.28, wpb=3547.5, bsz=175.8, num_updates=72500, lr=0.000234888, gnorm=1.48, train_wall=30, wall=22702\n",
            "2020-08-04 03:15:10 | INFO | train_inner | epoch 072:    535 / 1015 loss=3.91, nll_loss=2.243, ppl=4.73, wps=11530.7, ups=3.3, wpb=3489.2, bsz=146.2, num_updates=72600, lr=0.000234726, gnorm=1.559, train_wall=30, wall=22733\n",
            "2020-08-04 03:15:41 | INFO | train_inner | epoch 072:    635 / 1015 loss=3.836, nll_loss=2.16, ppl=4.47, wps=11708.9, ups=3.29, wpb=3556.2, bsz=174.7, num_updates=72700, lr=0.000234565, gnorm=1.462, train_wall=30, wall=22763\n",
            "2020-08-04 03:16:11 | INFO | train_inner | epoch 072:    735 / 1015 loss=3.883, nll_loss=2.213, ppl=4.64, wps=11587.4, ups=3.29, wpb=3523.2, bsz=159.4, num_updates=72800, lr=0.000234404, gnorm=1.531, train_wall=30, wall=22793\n",
            "2020-08-04 03:16:42 | INFO | train_inner | epoch 072:    835 / 1015 loss=3.905, nll_loss=2.237, ppl=4.71, wps=11542.5, ups=3.28, wpb=3518.8, bsz=149.6, num_updates=72900, lr=0.000234243, gnorm=1.522, train_wall=30, wall=22824\n",
            "2020-08-04 03:17:12 | INFO | train_inner | epoch 072:    935 / 1015 loss=3.92, nll_loss=2.256, ppl=4.78, wps=11789.3, ups=3.28, wpb=3592.7, bsz=156.5, num_updates=73000, lr=0.000234082, gnorm=1.509, train_wall=30, wall=22854\n",
            "2020-08-04 03:17:37 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:17:43 | INFO | valid | epoch 072 | valid on 'valid' subset | loss 4.245 | nll_loss 2.551 | ppl 5.86 | wps 27027.8 | wpb 2866.6 | bsz 127.8 | num_updates 73080 | best_loss 4.245\n",
            "2020-08-04 03:17:43 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:17:45 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint72.pt (epoch 72 @ 73080 updates, score 4.245) (writing took 2.5298659299987776 seconds)\n",
            "2020-08-04 03:17:45 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:17:45 | INFO | fairseq_cli.train | end of epoch 72 (average epoch stats below)\n",
            "2020-08-04 03:17:45 | INFO | train | epoch 072 | loss 3.871 | nll_loss 2.199 | ppl 4.59 | wps 11350.6 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 73080 | lr 0.000233954 | gnorm 1.498 | train_wall 308 | wall 22887\n",
            "2020-08-04 03:17:45 | INFO | fairseq_cli.train | begin training epoch 72\n",
            "2020-08-04 03:17:51 | INFO | train_inner | epoch 073:     20 / 1015 loss=3.903, nll_loss=2.235, ppl=4.71, wps=9185.9, ups=2.55, wpb=3606.7, bsz=154.3, num_updates=73100, lr=0.000233922, gnorm=1.49, train_wall=30, wall=22894\n",
            "2020-08-04 03:18:22 | INFO | train_inner | epoch 073:    120 / 1015 loss=3.803, nll_loss=2.122, ppl=4.35, wps=11641.2, ups=3.28, wpb=3551.6, bsz=156.6, num_updates=73200, lr=0.000233762, gnorm=1.481, train_wall=30, wall=22924\n",
            "2020-08-04 03:18:52 | INFO | train_inner | epoch 073:    220 / 1015 loss=3.861, nll_loss=2.187, ppl=4.55, wps=11576.1, ups=3.3, wpb=3507.1, bsz=155.8, num_updates=73300, lr=0.000233603, gnorm=1.545, train_wall=30, wall=22954\n",
            "2020-08-04 03:19:23 | INFO | train_inner | epoch 073:    320 / 1015 loss=3.828, nll_loss=2.15, ppl=4.44, wps=11631.2, ups=3.3, wpb=3528.2, bsz=159.6, num_updates=73400, lr=0.000233444, gnorm=1.513, train_wall=30, wall=22985\n",
            "2020-08-04 03:19:53 | INFO | train_inner | epoch 073:    420 / 1015 loss=3.858, nll_loss=2.182, ppl=4.54, wps=11677.4, ups=3.28, wpb=3556.2, bsz=153.4, num_updates=73500, lr=0.000233285, gnorm=1.508, train_wall=30, wall=23015\n",
            "2020-08-04 03:20:24 | INFO | train_inner | epoch 073:    520 / 1015 loss=3.849, nll_loss=2.174, ppl=4.51, wps=11804.8, ups=3.27, wpb=3605.9, bsz=159.2, num_updates=73600, lr=0.000233126, gnorm=1.486, train_wall=30, wall=23046\n",
            "2020-08-04 03:20:54 | INFO | train_inner | epoch 073:    620 / 1015 loss=3.891, nll_loss=2.222, ppl=4.67, wps=11855.9, ups=3.27, wpb=3620.8, bsz=158.1, num_updates=73700, lr=0.000232968, gnorm=1.493, train_wall=30, wall=23076\n",
            "2020-08-04 03:21:25 | INFO | train_inner | epoch 073:    720 / 1015 loss=3.867, nll_loss=2.196, ppl=4.58, wps=11642, ups=3.28, wpb=3547.3, bsz=164.9, num_updates=73800, lr=0.00023281, gnorm=1.499, train_wall=30, wall=23107\n",
            "2020-08-04 03:21:55 | INFO | train_inner | epoch 073:    820 / 1015 loss=3.902, nll_loss=2.234, ppl=4.71, wps=11690, ups=3.28, wpb=3567.5, bsz=158.5, num_updates=73900, lr=0.000232653, gnorm=1.499, train_wall=30, wall=23137\n",
            "2020-08-04 03:22:25 | INFO | train_inner | epoch 073:    920 / 1015 loss=3.92, nll_loss=2.256, ppl=4.78, wps=11734.4, ups=3.32, wpb=3530.1, bsz=161.9, num_updates=74000, lr=0.000232495, gnorm=1.568, train_wall=30, wall=23167\n",
            "2020-08-04 03:22:54 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:23:00 | INFO | valid | epoch 073 | valid on 'valid' subset | loss 4.246 | nll_loss 2.55 | ppl 5.86 | wps 27128.9 | wpb 2866.6 | bsz 127.8 | num_updates 74095 | best_loss 4.245\n",
            "2020-08-04 03:23:00 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:23:02 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint73.pt (epoch 73 @ 74095 updates, score 4.246) (writing took 1.5802511809997668 seconds)\n",
            "2020-08-04 03:23:02 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:23:02 | INFO | fairseq_cli.train | end of epoch 73 (average epoch stats below)\n",
            "2020-08-04 03:23:02 | INFO | train | epoch 073 | loss 3.869 | nll_loss 2.196 | ppl 4.58 | wps 11387.9 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 74095 | lr 0.000232346 | gnorm 1.514 | train_wall 308 | wall 23204\n",
            "2020-08-04 03:23:02 | INFO | fairseq_cli.train | begin training epoch 73\n",
            "2020-08-04 03:23:03 | INFO | train_inner | epoch 074:      5 / 1015 loss=3.91, nll_loss=2.243, ppl=4.73, wps=9194.5, ups=2.62, wpb=3507.6, bsz=151.8, num_updates=74100, lr=0.000232338, gnorm=1.534, train_wall=30, wall=23205\n",
            "2020-08-04 03:23:34 | INFO | train_inner | epoch 074:    105 / 1015 loss=3.822, nll_loss=2.141, ppl=4.41, wps=11546.6, ups=3.3, wpb=3501.4, bsz=153.4, num_updates=74200, lr=0.000232182, gnorm=1.511, train_wall=30, wall=23236\n",
            "2020-08-04 03:24:04 | INFO | train_inner | epoch 074:    205 / 1015 loss=3.815, nll_loss=2.135, ppl=4.39, wps=11749.5, ups=3.29, wpb=3575.8, bsz=166.7, num_updates=74300, lr=0.000232025, gnorm=1.49, train_wall=30, wall=23266\n",
            "2020-08-04 03:24:35 | INFO | train_inner | epoch 074:    305 / 1015 loss=3.833, nll_loss=2.154, ppl=4.45, wps=11653.1, ups=3.29, wpb=3545, bsz=157.8, num_updates=74400, lr=0.000231869, gnorm=1.542, train_wall=30, wall=23297\n",
            "2020-08-04 03:25:05 | INFO | train_inner | epoch 074:    405 / 1015 loss=3.857, nll_loss=2.183, ppl=4.54, wps=11775.6, ups=3.31, wpb=3560.9, bsz=164.7, num_updates=74500, lr=0.000231714, gnorm=1.498, train_wall=30, wall=23327\n",
            "2020-08-04 03:25:35 | INFO | train_inner | epoch 074:    505 / 1015 loss=3.888, nll_loss=2.218, ppl=4.65, wps=11659.7, ups=3.29, wpb=3542.3, bsz=153.3, num_updates=74600, lr=0.000231558, gnorm=1.537, train_wall=30, wall=23357\n",
            "2020-08-04 03:26:06 | INFO | train_inner | epoch 074:    605 / 1015 loss=3.859, nll_loss=2.185, ppl=4.55, wps=11528.3, ups=3.28, wpb=3514.5, bsz=162.6, num_updates=74700, lr=0.000231403, gnorm=1.532, train_wall=30, wall=23388\n",
            "2020-08-04 03:26:36 | INFO | train_inner | epoch 074:    705 / 1015 loss=3.859, nll_loss=2.187, ppl=4.55, wps=11815, ups=3.3, wpb=3582.7, bsz=161.8, num_updates=74800, lr=0.000231249, gnorm=1.524, train_wall=30, wall=23418\n",
            "2020-08-04 03:27:07 | INFO | train_inner | epoch 074:    805 / 1015 loss=3.875, nll_loss=2.202, ppl=4.6, wps=11616.7, ups=3.27, wpb=3556.2, bsz=151.6, num_updates=74900, lr=0.000231094, gnorm=1.524, train_wall=31, wall=23449\n",
            "2020-08-04 03:27:37 | INFO | train_inner | epoch 074:    905 / 1015 loss=3.946, nll_loss=2.283, ppl=4.87, wps=11643.2, ups=3.31, wpb=3518.9, bsz=141.8, num_updates=75000, lr=0.00023094, gnorm=1.567, train_wall=30, wall=23479\n",
            "2020-08-04 03:28:07 | INFO | train_inner | epoch 074:   1005 / 1015 loss=3.903, nll_loss=2.237, ppl=4.71, wps=11883, ups=3.28, wpb=3618, bsz=160.2, num_updates=75100, lr=0.000230786, gnorm=1.495, train_wall=30, wall=23509\n",
            "2020-08-04 03:28:10 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:28:16 | INFO | valid | epoch 074 | valid on 'valid' subset | loss 4.258 | nll_loss 2.561 | ppl 5.9 | wps 27044.4 | wpb 2866.6 | bsz 127.8 | num_updates 75110 | best_loss 4.245\n",
            "2020-08-04 03:28:16 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:28:18 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint74.pt (epoch 74 @ 75110 updates, score 4.258) (writing took 1.6433282340003643 seconds)\n",
            "2020-08-04 03:28:18 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:28:18 | INFO | fairseq_cli.train | end of epoch 74 (average epoch stats below)\n",
            "2020-08-04 03:28:18 | INFO | train | epoch 074 | loss 3.865 | nll_loss 2.192 | ppl 4.57 | wps 11391.7 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 75110 | lr 0.000230771 | gnorm 1.521 | train_wall 307 | wall 23520\n",
            "2020-08-04 03:28:18 | INFO | fairseq_cli.train | begin training epoch 74\n",
            "2020-08-04 03:28:45 | INFO | train_inner | epoch 075:     90 / 1015 loss=3.826, nll_loss=2.148, ppl=4.43, wps=9320.9, ups=2.62, wpb=3556.2, bsz=166.3, num_updates=75200, lr=0.000230633, gnorm=1.493, train_wall=30, wall=23548\n",
            "2020-08-04 03:29:16 | INFO | train_inner | epoch 075:    190 / 1015 loss=3.838, nll_loss=2.16, ppl=4.47, wps=11680.2, ups=3.29, wpb=3555.5, bsz=155.5, num_updates=75300, lr=0.00023048, gnorm=1.517, train_wall=30, wall=23578\n",
            "2020-08-04 03:29:46 | INFO | train_inner | epoch 075:    290 / 1015 loss=3.831, nll_loss=2.152, ppl=4.45, wps=11616.1, ups=3.27, wpb=3549.4, bsz=159.8, num_updates=75400, lr=0.000230327, gnorm=1.51, train_wall=30, wall=23609\n",
            "2020-08-04 03:30:17 | INFO | train_inner | epoch 075:    390 / 1015 loss=3.841, nll_loss=2.164, ppl=4.48, wps=11578.7, ups=3.29, wpb=3520.5, bsz=154.1, num_updates=75500, lr=0.000230174, gnorm=1.537, train_wall=30, wall=23639\n",
            "2020-08-04 03:30:47 | INFO | train_inner | epoch 075:    490 / 1015 loss=3.8, nll_loss=2.119, ppl=4.34, wps=11860.5, ups=3.27, wpb=3622.7, bsz=170.1, num_updates=75600, lr=0.000230022, gnorm=1.466, train_wall=30, wall=23669\n",
            "2020-08-04 03:31:18 | INFO | train_inner | epoch 075:    590 / 1015 loss=3.842, nll_loss=2.167, ppl=4.49, wps=11531.1, ups=3.28, wpb=3517.7, bsz=171.8, num_updates=75700, lr=0.00022987, gnorm=1.52, train_wall=30, wall=23700\n",
            "2020-08-04 03:31:48 | INFO | train_inner | epoch 075:    690 / 1015 loss=3.897, nll_loss=2.228, ppl=4.68, wps=11537.6, ups=3.31, wpb=3481, bsz=152.2, num_updates=75800, lr=0.000229718, gnorm=1.573, train_wall=30, wall=23730\n",
            "2020-08-04 03:32:18 | INFO | train_inner | epoch 075:    790 / 1015 loss=3.899, nll_loss=2.23, ppl=4.69, wps=11737.6, ups=3.29, wpb=3566.9, bsz=147.7, num_updates=75900, lr=0.000229567, gnorm=1.523, train_wall=30, wall=23761\n",
            "2020-08-04 03:32:49 | INFO | train_inner | epoch 075:    890 / 1015 loss=3.894, nll_loss=2.226, ppl=4.68, wps=11679.3, ups=3.3, wpb=3540.4, bsz=160, num_updates=76000, lr=0.000229416, gnorm=1.558, train_wall=30, wall=23791\n",
            "2020-08-04 03:33:19 | INFO | train_inner | epoch 075:    990 / 1015 loss=3.934, nll_loss=2.271, ppl=4.83, wps=11829.9, ups=3.31, wpb=3569.3, bsz=144.1, num_updates=76100, lr=0.000229265, gnorm=1.527, train_wall=30, wall=23821\n",
            "2020-08-04 03:33:26 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:33:33 | INFO | valid | epoch 075 | valid on 'valid' subset | loss 4.262 | nll_loss 2.566 | ppl 5.92 | wps 27131.8 | wpb 2866.6 | bsz 127.8 | num_updates 76125 | best_loss 4.245\n",
            "2020-08-04 03:33:33 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:33:34 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint75.pt (epoch 75 @ 76125 updates, score 4.262) (writing took 1.4803851389988267 seconds)\n",
            "2020-08-04 03:33:34 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:33:34 | INFO | fairseq_cli.train | end of epoch 75 (average epoch stats below)\n",
            "2020-08-04 03:33:34 | INFO | train | epoch 075 | loss 3.86 | nll_loss 2.187 | ppl 4.55 | wps 11401.3 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 76125 | lr 0.000229227 | gnorm 1.522 | train_wall 307 | wall 23836\n",
            "2020-08-04 03:33:34 | INFO | fairseq_cli.train | begin training epoch 75\n",
            "2020-08-04 03:33:57 | INFO | train_inner | epoch 076:     75 / 1015 loss=3.782, nll_loss=2.099, ppl=4.29, wps=9495.6, ups=2.61, wpb=3631.6, bsz=176.1, num_updates=76200, lr=0.000229114, gnorm=1.446, train_wall=31, wall=23859\n",
            "2020-08-04 03:34:28 | INFO | train_inner | epoch 076:    175 / 1015 loss=3.812, nll_loss=2.132, ppl=4.38, wps=11756.1, ups=3.29, wpb=3576.9, bsz=154.9, num_updates=76300, lr=0.000228964, gnorm=1.487, train_wall=30, wall=23890\n",
            "2020-08-04 03:34:58 | INFO | train_inner | epoch 076:    275 / 1015 loss=3.806, nll_loss=2.124, ppl=4.36, wps=11552.6, ups=3.28, wpb=3526.1, bsz=157, num_updates=76400, lr=0.000228814, gnorm=1.534, train_wall=30, wall=23920\n",
            "2020-08-04 03:35:28 | INFO | train_inner | epoch 076:    375 / 1015 loss=3.857, nll_loss=2.181, ppl=4.54, wps=11689.9, ups=3.3, wpb=3539.7, bsz=151.5, num_updates=76500, lr=0.000228665, gnorm=1.554, train_wall=30, wall=23950\n",
            "2020-08-04 03:35:59 | INFO | train_inner | epoch 076:    475 / 1015 loss=3.859, nll_loss=2.186, ppl=4.55, wps=11652.4, ups=3.3, wpb=3533.5, bsz=167, num_updates=76600, lr=0.000228515, gnorm=1.561, train_wall=30, wall=23981\n",
            "2020-08-04 03:36:29 | INFO | train_inner | epoch 076:    575 / 1015 loss=3.866, nll_loss=2.192, ppl=4.57, wps=11711.9, ups=3.28, wpb=3574.4, bsz=148.8, num_updates=76700, lr=0.000228366, gnorm=1.55, train_wall=30, wall=24011\n",
            "2020-08-04 03:36:59 | INFO | train_inner | epoch 076:    675 / 1015 loss=3.887, nll_loss=2.216, ppl=4.65, wps=11534.9, ups=3.33, wpb=3463.9, bsz=149.3, num_updates=76800, lr=0.000228218, gnorm=1.554, train_wall=30, wall=24041\n",
            "2020-08-04 03:37:29 | INFO | train_inner | epoch 076:    775 / 1015 loss=3.907, nll_loss=2.24, ppl=4.72, wps=11652.6, ups=3.32, wpb=3514.3, bsz=152.8, num_updates=76900, lr=0.000228069, gnorm=1.557, train_wall=30, wall=24072\n",
            "2020-08-04 03:38:00 | INFO | train_inner | epoch 076:    875 / 1015 loss=3.876, nll_loss=2.205, ppl=4.61, wps=11614.4, ups=3.29, wpb=3531.1, bsz=166.1, num_updates=77000, lr=0.000227921, gnorm=1.554, train_wall=30, wall=24102\n",
            "2020-08-04 03:38:30 | INFO | train_inner | epoch 076:    975 / 1015 loss=3.887, nll_loss=2.219, ppl=4.66, wps=11702.5, ups=3.28, wpb=3568.6, bsz=164.2, num_updates=77100, lr=0.000227773, gnorm=1.535, train_wall=30, wall=24132\n",
            "2020-08-04 03:38:42 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:38:49 | INFO | valid | epoch 076 | valid on 'valid' subset | loss 4.255 | nll_loss 2.553 | ppl 5.87 | wps 27031.2 | wpb 2866.6 | bsz 127.8 | num_updates 77140 | best_loss 4.245\n",
            "2020-08-04 03:38:49 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:38:50 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint76.pt (epoch 76 @ 77140 updates, score 4.255) (writing took 1.657102819001011 seconds)\n",
            "2020-08-04 03:38:50 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:38:50 | INFO | fairseq_cli.train | end of epoch 76 (average epoch stats below)\n",
            "2020-08-04 03:38:50 | INFO | train | epoch 076 | loss 3.857 | nll_loss 2.183 | ppl 4.54 | wps 11393.7 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 77140 | lr 0.000227714 | gnorm 1.534 | train_wall 307 | wall 24152\n",
            "2020-08-04 03:38:50 | INFO | fairseq_cli.train | begin training epoch 76\n",
            "2020-08-04 03:39:09 | INFO | train_inner | epoch 077:     60 / 1015 loss=3.856, nll_loss=2.181, ppl=4.53, wps=9504.9, ups=2.6, wpb=3650.9, bsz=151.5, num_updates=77200, lr=0.000227626, gnorm=1.504, train_wall=30, wall=24171\n",
            "2020-08-04 03:39:39 | INFO | train_inner | epoch 077:    160 / 1015 loss=3.806, nll_loss=2.126, ppl=4.37, wps=11838.9, ups=3.29, wpb=3594.2, bsz=167, num_updates=77300, lr=0.000227478, gnorm=1.481, train_wall=30, wall=24201\n",
            "2020-08-04 03:40:09 | INFO | train_inner | epoch 077:    260 / 1015 loss=3.812, nll_loss=2.13, ppl=4.38, wps=11570.4, ups=3.3, wpb=3505.4, bsz=164.4, num_updates=77400, lr=0.000227331, gnorm=1.533, train_wall=30, wall=24231\n",
            "2020-08-04 03:40:40 | INFO | train_inner | epoch 077:    360 / 1015 loss=3.869, nll_loss=2.194, ppl=4.58, wps=11523.8, ups=3.31, wpb=3477.1, bsz=137.6, num_updates=77500, lr=0.000227185, gnorm=1.586, train_wall=30, wall=24262\n",
            "2020-08-04 03:41:10 | INFO | train_inner | epoch 077:    460 / 1015 loss=3.868, nll_loss=2.195, ppl=4.58, wps=11755.1, ups=3.3, wpb=3558.9, bsz=155.9, num_updates=77600, lr=0.000227038, gnorm=1.539, train_wall=30, wall=24292\n",
            "2020-08-04 03:41:40 | INFO | train_inner | epoch 077:    560 / 1015 loss=3.888, nll_loss=2.217, ppl=4.65, wps=11385, ups=3.3, wpb=3445.7, bsz=145.4, num_updates=77700, lr=0.000226892, gnorm=1.618, train_wall=30, wall=24322\n",
            "2020-08-04 03:42:11 | INFO | train_inner | epoch 077:    660 / 1015 loss=3.866, nll_loss=2.194, ppl=4.58, wps=11813.7, ups=3.27, wpb=3615.2, bsz=164.3, num_updates=77800, lr=0.000226746, gnorm=1.502, train_wall=31, wall=24353\n",
            "2020-08-04 03:42:41 | INFO | train_inner | epoch 077:    760 / 1015 loss=3.848, nll_loss=2.173, ppl=4.51, wps=11620.2, ups=3.27, wpb=3555.2, bsz=166, num_updates=77900, lr=0.000226601, gnorm=1.555, train_wall=30, wall=24383\n",
            "2020-08-04 03:43:12 | INFO | train_inner | epoch 077:    860 / 1015 loss=3.889, nll_loss=2.219, ppl=4.66, wps=11710.4, ups=3.27, wpb=3576.2, bsz=152.2, num_updates=78000, lr=0.000226455, gnorm=1.557, train_wall=30, wall=24414\n",
            "2020-08-04 03:43:42 | INFO | train_inner | epoch 077:    960 / 1015 loss=3.92, nll_loss=2.254, ppl=4.77, wps=11664.8, ups=3.3, wpb=3534.7, bsz=142.8, num_updates=78100, lr=0.00022631, gnorm=1.558, train_wall=30, wall=24444\n",
            "2020-08-04 03:43:59 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:44:05 | INFO | valid | epoch 077 | valid on 'valid' subset | loss 4.27 | nll_loss 2.57 | ppl 5.94 | wps 27071.4 | wpb 2866.6 | bsz 127.8 | num_updates 78155 | best_loss 4.245\n",
            "2020-08-04 03:44:05 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:44:07 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint77.pt (epoch 77 @ 78155 updates, score 4.27) (writing took 1.708558341000753 seconds)\n",
            "2020-08-04 03:44:07 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:44:07 | INFO | fairseq_cli.train | end of epoch 77 (average epoch stats below)\n",
            "2020-08-04 03:44:07 | INFO | train | epoch 077 | loss 3.854 | nll_loss 2.179 | ppl 4.53 | wps 11385.2 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 78155 | lr 0.000226231 | gnorm 1.54 | train_wall 308 | wall 24469\n",
            "2020-08-04 03:44:07 | INFO | fairseq_cli.train | begin training epoch 77\n",
            "2020-08-04 03:44:21 | INFO | train_inner | epoch 078:     45 / 1015 loss=3.771, nll_loss=2.087, ppl=4.25, wps=9371.9, ups=2.6, wpb=3611.5, bsz=181, num_updates=78200, lr=0.000226166, gnorm=1.496, train_wall=31, wall=24483\n",
            "2020-08-04 03:44:51 | INFO | train_inner | epoch 078:    145 / 1015 loss=3.834, nll_loss=2.155, ppl=4.45, wps=11622.9, ups=3.31, wpb=3515.8, bsz=145.5, num_updates=78300, lr=0.000226021, gnorm=1.574, train_wall=30, wall=24513\n",
            "2020-08-04 03:45:21 | INFO | train_inner | epoch 078:    245 / 1015 loss=3.806, nll_loss=2.125, ppl=4.36, wps=11633.3, ups=3.29, wpb=3537.8, bsz=158.7, num_updates=78400, lr=0.000225877, gnorm=1.532, train_wall=30, wall=24543\n",
            "2020-08-04 03:45:52 | INFO | train_inner | epoch 078:    345 / 1015 loss=3.842, nll_loss=2.166, ppl=4.49, wps=11910, ups=3.29, wpb=3623.7, bsz=160.7, num_updates=78500, lr=0.000225733, gnorm=1.516, train_wall=30, wall=24574\n",
            "2020-08-04 03:46:22 | INFO | train_inner | epoch 078:    445 / 1015 loss=3.854, nll_loss=2.178, ppl=4.52, wps=11618.3, ups=3.3, wpb=3522.6, bsz=147, num_updates=78600, lr=0.000225589, gnorm=1.548, train_wall=30, wall=24604\n",
            "2020-08-04 03:46:53 | INFO | train_inner | epoch 078:    545 / 1015 loss=3.815, nll_loss=2.137, ppl=4.4, wps=11673.4, ups=3.28, wpb=3562.6, bsz=171.9, num_updates=78700, lr=0.000225446, gnorm=1.513, train_wall=30, wall=24635\n",
            "2020-08-04 03:47:23 | INFO | train_inner | epoch 078:    645 / 1015 loss=3.87, nll_loss=2.197, ppl=4.58, wps=11514.8, ups=3.29, wpb=3499.1, bsz=158.9, num_updates=78800, lr=0.000225303, gnorm=1.627, train_wall=30, wall=24665\n",
            "2020-08-04 03:47:53 | INFO | train_inner | epoch 078:    745 / 1015 loss=3.896, nll_loss=2.227, ppl=4.68, wps=11640.3, ups=3.29, wpb=3535.2, bsz=147.2, num_updates=78900, lr=0.00022516, gnorm=1.543, train_wall=30, wall=24695\n",
            "2020-08-04 03:48:24 | INFO | train_inner | epoch 078:    845 / 1015 loss=3.868, nll_loss=2.197, ppl=4.59, wps=11696.3, ups=3.3, wpb=3547.4, bsz=160.2, num_updates=79000, lr=0.000225018, gnorm=1.563, train_wall=30, wall=24726\n",
            "2020-08-04 03:48:54 | INFO | train_inner | epoch 078:    945 / 1015 loss=3.895, nll_loss=2.228, ppl=4.68, wps=11799.6, ups=3.3, wpb=3573.5, bsz=163.9, num_updates=79100, lr=0.000224875, gnorm=1.571, train_wall=30, wall=24756\n",
            "2020-08-04 03:49:15 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:49:21 | INFO | valid | epoch 078 | valid on 'valid' subset | loss 4.26 | nll_loss 2.566 | ppl 5.92 | wps 27185 | wpb 2866.6 | bsz 127.8 | num_updates 79170 | best_loss 4.245\n",
            "2020-08-04 03:49:21 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:49:23 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint78.pt (epoch 78 @ 79170 updates, score 4.26) (writing took 1.599316882002313 seconds)\n",
            "2020-08-04 03:49:23 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:49:23 | INFO | fairseq_cli.train | end of epoch 78 (average epoch stats below)\n",
            "2020-08-04 03:49:23 | INFO | train | epoch 078 | loss 3.851 | nll_loss 2.176 | ppl 4.52 | wps 11401.7 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 79170 | lr 0.000224776 | gnorm 1.552 | train_wall 307 | wall 24785\n",
            "2020-08-04 03:49:23 | INFO | fairseq_cli.train | begin training epoch 78\n",
            "2020-08-04 03:49:32 | INFO | train_inner | epoch 079:     30 / 1015 loss=3.871, nll_loss=2.199, ppl=4.59, wps=9336.2, ups=2.63, wpb=3547.8, bsz=159, num_updates=79200, lr=0.000224733, gnorm=1.555, train_wall=30, wall=24794\n",
            "2020-08-04 03:50:02 | INFO | train_inner | epoch 079:    130 / 1015 loss=3.781, nll_loss=2.095, ppl=4.27, wps=11617.9, ups=3.29, wpb=3533.6, bsz=153.5, num_updates=79300, lr=0.000224592, gnorm=1.559, train_wall=30, wall=24824\n",
            "2020-08-04 03:50:33 | INFO | train_inner | epoch 079:    230 / 1015 loss=3.812, nll_loss=2.131, ppl=4.38, wps=11628.8, ups=3.3, wpb=3527.8, bsz=158.9, num_updates=79400, lr=0.00022445, gnorm=1.548, train_wall=30, wall=24855\n",
            "2020-08-04 03:51:03 | INFO | train_inner | epoch 079:    330 / 1015 loss=3.807, nll_loss=2.127, ppl=4.37, wps=11836.7, ups=3.29, wpb=3597.1, bsz=165.2, num_updates=79500, lr=0.000224309, gnorm=1.5, train_wall=30, wall=24885\n",
            "2020-08-04 03:51:34 | INFO | train_inner | epoch 079:    430 / 1015 loss=3.813, nll_loss=2.133, ppl=4.39, wps=11615.9, ups=3.28, wpb=3542.6, bsz=167.3, num_updates=79600, lr=0.000224168, gnorm=1.555, train_wall=30, wall=24916\n",
            "2020-08-04 03:52:04 | INFO | train_inner | epoch 079:    530 / 1015 loss=3.844, nll_loss=2.167, ppl=4.49, wps=11757.8, ups=3.3, wpb=3564.6, bsz=155.2, num_updates=79700, lr=0.000224027, gnorm=1.547, train_wall=30, wall=24946\n",
            "2020-08-04 03:52:34 | INFO | train_inner | epoch 079:    630 / 1015 loss=3.869, nll_loss=2.197, ppl=4.58, wps=11664.8, ups=3.28, wpb=3554.2, bsz=154.6, num_updates=79800, lr=0.000223887, gnorm=1.578, train_wall=30, wall=24977\n",
            "2020-08-04 03:53:05 | INFO | train_inner | epoch 079:    730 / 1015 loss=3.904, nll_loss=2.236, ppl=4.71, wps=11687, ups=3.3, wpb=3538.1, bsz=143.8, num_updates=79900, lr=0.000223747, gnorm=1.605, train_wall=30, wall=25007\n",
            "2020-08-04 03:53:35 | INFO | train_inner | epoch 079:    830 / 1015 loss=3.853, nll_loss=2.18, ppl=4.53, wps=11813.2, ups=3.28, wpb=3606.4, bsz=169.7, num_updates=80000, lr=0.000223607, gnorm=1.525, train_wall=30, wall=25037\n",
            "2020-08-04 03:54:05 | INFO | train_inner | epoch 079:    930 / 1015 loss=3.882, nll_loss=2.212, ppl=4.63, wps=11667.5, ups=3.31, wpb=3526.1, bsz=164.5, num_updates=80100, lr=0.000223467, gnorm=1.594, train_wall=30, wall=25068\n",
            "2020-08-04 03:54:31 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:54:37 | INFO | valid | epoch 079 | valid on 'valid' subset | loss 4.251 | nll_loss 2.558 | ppl 5.89 | wps 26991.5 | wpb 2866.6 | bsz 127.8 | num_updates 80185 | best_loss 4.245\n",
            "2020-08-04 03:54:37 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:54:39 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint79.pt (epoch 79 @ 80185 updates, score 4.251) (writing took 1.519302914999571 seconds)\n",
            "2020-08-04 03:54:39 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:54:39 | INFO | fairseq_cli.train | end of epoch 79 (average epoch stats below)\n",
            "2020-08-04 03:54:39 | INFO | train | epoch 079 | loss 3.848 | nll_loss 2.172 | ppl 4.51 | wps 11400.7 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 80185 | lr 0.000223349 | gnorm 1.56 | train_wall 307 | wall 25101\n",
            "2020-08-04 03:54:39 | INFO | fairseq_cli.train | begin training epoch 79\n",
            "2020-08-04 03:54:44 | INFO | train_inner | epoch 080:     15 / 1015 loss=3.899, nll_loss=2.232, ppl=4.7, wps=9225, ups=2.62, wpb=3516, bsz=152.3, num_updates=80200, lr=0.000223328, gnorm=1.572, train_wall=30, wall=25106\n",
            "2020-08-04 03:55:14 | INFO | train_inner | epoch 080:    115 / 1015 loss=3.789, nll_loss=2.104, ppl=4.3, wps=11746.8, ups=3.3, wpb=3558.7, bsz=147.1, num_updates=80300, lr=0.000223189, gnorm=1.539, train_wall=30, wall=25136\n",
            "2020-08-04 03:55:44 | INFO | train_inner | epoch 080:    215 / 1015 loss=3.751, nll_loss=2.065, ppl=4.18, wps=11743.7, ups=3.31, wpb=3547.9, bsz=187.8, num_updates=80400, lr=0.00022305, gnorm=1.517, train_wall=30, wall=25166\n",
            "2020-08-04 03:56:14 | INFO | train_inner | epoch 080:    315 / 1015 loss=3.758, nll_loss=2.073, ppl=4.21, wps=11785.3, ups=3.28, wpb=3589.8, bsz=186.6, num_updates=80500, lr=0.000222911, gnorm=1.502, train_wall=30, wall=25197\n",
            "2020-08-04 03:56:45 | INFO | train_inner | epoch 080:    415 / 1015 loss=3.846, nll_loss=2.17, ppl=4.5, wps=11486.3, ups=3.3, wpb=3480.4, bsz=155.7, num_updates=80600, lr=0.000222773, gnorm=1.591, train_wall=30, wall=25227\n",
            "2020-08-04 03:57:15 | INFO | train_inner | epoch 080:    515 / 1015 loss=3.883, nll_loss=2.212, ppl=4.63, wps=11872.2, ups=3.28, wpb=3616.2, bsz=142, num_updates=80700, lr=0.000222635, gnorm=1.534, train_wall=30, wall=25257\n",
            "2020-08-04 03:57:46 | INFO | train_inner | epoch 080:    615 / 1015 loss=3.851, nll_loss=2.175, ppl=4.52, wps=11717.8, ups=3.29, wpb=3562.3, bsz=142.7, num_updates=80800, lr=0.000222497, gnorm=1.549, train_wall=30, wall=25288\n",
            "2020-08-04 03:58:16 | INFO | train_inner | epoch 080:    715 / 1015 loss=3.876, nll_loss=2.204, ppl=4.61, wps=11755.5, ups=3.3, wpb=3566.3, bsz=153, num_updates=80900, lr=0.00022236, gnorm=1.571, train_wall=30, wall=25318\n",
            "2020-08-04 03:58:47 | INFO | train_inner | epoch 080:    815 / 1015 loss=3.883, nll_loss=2.213, ppl=4.64, wps=11764.2, ups=3.26, wpb=3605.6, bsz=159.2, num_updates=81000, lr=0.000222222, gnorm=1.551, train_wall=31, wall=25349\n",
            "2020-08-04 03:59:17 | INFO | train_inner | epoch 080:    915 / 1015 loss=3.867, nll_loss=2.195, ppl=4.58, wps=11492.2, ups=3.3, wpb=3477.8, bsz=168.5, num_updates=81100, lr=0.000222085, gnorm=1.622, train_wall=30, wall=25379\n",
            "2020-08-04 03:59:47 | INFO | train_inner | epoch 080:   1015 / 1015 loss=3.941, nll_loss=2.278, ppl=4.85, wps=11574.3, ups=3.31, wpb=3495.2, bsz=136.9, num_updates=81200, lr=0.000221948, gnorm=1.652, train_wall=30, wall=25409\n",
            "2020-08-04 03:59:47 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 03:59:53 | INFO | valid | epoch 080 | valid on 'valid' subset | loss 4.25 | nll_loss 2.551 | ppl 5.86 | wps 27139.1 | wpb 2866.6 | bsz 127.8 | num_updates 81200 | best_loss 4.245\n",
            "2020-08-04 03:59:53 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 03:59:55 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint80.pt (epoch 80 @ 81200 updates, score 4.25) (writing took 1.4633277130014903 seconds)\n",
            "2020-08-04 03:59:55 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 03:59:55 | INFO | fairseq_cli.train | end of epoch 80 (average epoch stats below)\n",
            "2020-08-04 03:59:55 | INFO | train | epoch 080 | loss 3.844 | nll_loss 2.168 | ppl 4.49 | wps 11411.8 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 81200 | lr 0.000221948 | gnorm 1.562 | train_wall 307 | wall 25417\n",
            "2020-08-04 03:59:55 | INFO | fairseq_cli.train | begin training epoch 80\n",
            "2020-08-04 04:00:25 | INFO | train_inner | epoch 081:    100 / 1015 loss=3.747, nll_loss=2.057, ppl=4.16, wps=9253.5, ups=2.62, wpb=3534.5, bsz=161.6, num_updates=81300, lr=0.000221812, gnorm=1.543, train_wall=30, wall=25447\n",
            "2020-08-04 04:00:56 | INFO | train_inner | epoch 081:    200 / 1015 loss=3.814, nll_loss=2.132, ppl=4.38, wps=11671.9, ups=3.3, wpb=3539.3, bsz=152.1, num_updates=81400, lr=0.000221676, gnorm=1.581, train_wall=30, wall=25478\n",
            "2020-08-04 04:01:26 | INFO | train_inner | epoch 081:    300 / 1015 loss=3.836, nll_loss=2.157, ppl=4.46, wps=11503.9, ups=3.31, wpb=3478.4, bsz=145.9, num_updates=81500, lr=0.00022154, gnorm=1.598, train_wall=30, wall=25508\n",
            "2020-08-04 04:01:56 | INFO | train_inner | epoch 081:    400 / 1015 loss=3.82, nll_loss=2.141, ppl=4.41, wps=11821, ups=3.27, wpb=3620, bsz=158.1, num_updates=81600, lr=0.000221404, gnorm=1.509, train_wall=31, wall=25539\n",
            "2020-08-04 04:02:27 | INFO | train_inner | epoch 081:    500 / 1015 loss=3.781, nll_loss=2.099, ppl=4.28, wps=11647.2, ups=3.29, wpb=3541, bsz=176.5, num_updates=81700, lr=0.000221268, gnorm=1.548, train_wall=30, wall=25569\n",
            "2020-08-04 04:02:57 | INFO | train_inner | epoch 081:    600 / 1015 loss=3.852, nll_loss=2.178, ppl=4.53, wps=11852.9, ups=3.29, wpb=3603.5, bsz=160.9, num_updates=81800, lr=0.000221133, gnorm=1.529, train_wall=30, wall=25599\n",
            "2020-08-04 04:03:28 | INFO | train_inner | epoch 081:    700 / 1015 loss=3.839, nll_loss=2.164, ppl=4.48, wps=11677.1, ups=3.3, wpb=3541.1, bsz=165, num_updates=81900, lr=0.000220998, gnorm=1.562, train_wall=30, wall=25630\n",
            "2020-08-04 04:03:58 | INFO | train_inner | epoch 081:    800 / 1015 loss=3.903, nll_loss=2.235, ppl=4.71, wps=11696.8, ups=3.3, wpb=3539.4, bsz=146.6, num_updates=82000, lr=0.000220863, gnorm=1.606, train_wall=30, wall=25660\n",
            "2020-08-04 04:04:28 | INFO | train_inner | epoch 081:    900 / 1015 loss=3.929, nll_loss=2.265, ppl=4.81, wps=11821.8, ups=3.28, wpb=3605.5, bsz=145.3, num_updates=82100, lr=0.000220729, gnorm=1.574, train_wall=30, wall=25690\n",
            "2020-08-04 04:04:59 | INFO | train_inner | epoch 081:   1000 / 1015 loss=3.873, nll_loss=2.203, ppl=4.6, wps=11695.1, ups=3.27, wpb=3577.7, bsz=171.8, num_updates=82200, lr=0.000220594, gnorm=1.566, train_wall=30, wall=25721\n",
            "2020-08-04 04:05:03 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 04:05:09 | INFO | valid | epoch 081 | valid on 'valid' subset | loss 4.271 | nll_loss 2.569 | ppl 5.93 | wps 27176.6 | wpb 2866.6 | bsz 127.8 | num_updates 82215 | best_loss 4.245\n",
            "2020-08-04 04:05:09 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 04:05:11 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint81.pt (epoch 81 @ 82215 updates, score 4.271) (writing took 1.6239064520013926 seconds)\n",
            "2020-08-04 04:05:11 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 04:05:11 | INFO | fairseq_cli.train | end of epoch 81 (average epoch stats below)\n",
            "2020-08-04 04:05:11 | INFO | train | epoch 081 | loss 3.84 | nll_loss 2.164 | ppl 4.48 | wps 11392.3 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 82215 | lr 0.000220574 | gnorm 1.567 | train_wall 307 | wall 25733\n",
            "2020-08-04 04:05:11 | INFO | fairseq_cli.train | begin training epoch 81\n",
            "2020-08-04 04:05:37 | INFO | train_inner | epoch 082:     85 / 1015 loss=3.804, nll_loss=2.121, ppl=4.35, wps=9111.8, ups=2.65, wpb=3432.1, bsz=148, num_updates=82300, lr=0.00022046, gnorm=1.678, train_wall=30, wall=25759\n",
            "2020-08-04 04:06:07 | INFO | train_inner | epoch 082:    185 / 1015 loss=3.82, nll_loss=2.138, ppl=4.4, wps=11487.1, ups=3.3, wpb=3485.4, bsz=146.5, num_updates=82400, lr=0.000220326, gnorm=1.593, train_wall=30, wall=25789\n",
            "2020-08-04 04:06:37 | INFO | train_inner | epoch 082:    285 / 1015 loss=3.806, nll_loss=2.124, ppl=4.36, wps=11619, ups=3.28, wpb=3545, bsz=150.3, num_updates=82500, lr=0.000220193, gnorm=1.582, train_wall=30, wall=25820\n",
            "2020-08-04 04:07:08 | INFO | train_inner | epoch 082:    385 / 1015 loss=3.863, nll_loss=2.19, ppl=4.56, wps=11788.4, ups=3.3, wpb=3573.1, bsz=159.4, num_updates=82600, lr=0.000220059, gnorm=1.592, train_wall=30, wall=25850\n",
            "2020-08-04 04:07:39 | INFO | train_inner | epoch 082:    485 / 1015 loss=3.773, nll_loss=2.089, ppl=4.25, wps=11818.2, ups=3.25, wpb=3638.9, bsz=176.3, num_updates=82700, lr=0.000219926, gnorm=1.486, train_wall=31, wall=25881\n",
            "2020-08-04 04:08:09 | INFO | train_inner | epoch 082:    585 / 1015 loss=3.867, nll_loss=2.195, ppl=4.58, wps=11686.4, ups=3.3, wpb=3540.1, bsz=161.3, num_updates=82800, lr=0.000219793, gnorm=1.624, train_wall=30, wall=25911\n",
            "2020-08-04 04:08:39 | INFO | train_inner | epoch 082:    685 / 1015 loss=3.814, nll_loss=2.135, ppl=4.39, wps=11584, ups=3.28, wpb=3529.8, bsz=165.2, num_updates=82900, lr=0.000219661, gnorm=1.556, train_wall=30, wall=25941\n",
            "2020-08-04 04:09:10 | INFO | train_inner | epoch 082:    785 / 1015 loss=3.845, nll_loss=2.171, ppl=4.5, wps=11665.6, ups=3.32, wpb=3518, bsz=168.6, num_updates=83000, lr=0.000219529, gnorm=1.585, train_wall=30, wall=25972\n",
            "2020-08-04 04:09:40 | INFO | train_inner | epoch 082:    885 / 1015 loss=3.87, nll_loss=2.199, ppl=4.59, wps=11858.7, ups=3.28, wpb=3613.9, bsz=151.7, num_updates=83100, lr=0.000219396, gnorm=1.533, train_wall=30, wall=26002\n",
            "2020-08-04 04:10:10 | INFO | train_inner | epoch 082:    985 / 1015 loss=3.92, nll_loss=2.255, ppl=4.77, wps=11758.9, ups=3.31, wpb=3554.2, bsz=146.6, num_updates=83200, lr=0.000219265, gnorm=1.616, train_wall=30, wall=26032\n",
            "2020-08-04 04:10:19 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 04:10:25 | INFO | valid | epoch 082 | valid on 'valid' subset | loss 4.245 | nll_loss 2.546 | ppl 5.84 | wps 26991.4 | wpb 2866.6 | bsz 127.8 | num_updates 83230 | best_loss 4.245\n",
            "2020-08-04 04:10:25 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 04:10:28 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint82.pt (epoch 82 @ 83230 updates, score 4.245) (writing took 2.5154523639976105 seconds)\n",
            "2020-08-04 04:10:28 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 04:10:28 | INFO | fairseq_cli.train | end of epoch 82 (average epoch stats below)\n",
            "2020-08-04 04:10:28 | INFO | train | epoch 082 | loss 3.839 | nll_loss 2.163 | ppl 4.48 | wps 11367.2 | ups 3.2 | wpb 3550.1 | bsz 157.9 | num_updates 83230 | lr 0.000219225 | gnorm 1.579 | train_wall 307 | wall 26050\n",
            "2020-08-04 04:10:28 | INFO | fairseq_cli.train | begin training epoch 82\n",
            "2020-08-04 04:10:49 | INFO | train_inner | epoch 083:     70 / 1015 loss=3.808, nll_loss=2.128, ppl=4.37, wps=9023.5, ups=2.56, wpb=3526.6, bsz=162.5, num_updates=83300, lr=0.000219133, gnorm=1.561, train_wall=30, wall=26071\n",
            "2020-08-04 04:11:20 | INFO | train_inner | epoch 083:    170 / 1015 loss=3.787, nll_loss=2.102, ppl=4.29, wps=11593, ups=3.27, wpb=3543.8, bsz=155.2, num_updates=83400, lr=0.000219001, gnorm=1.578, train_wall=30, wall=26102\n",
            "2020-08-04 04:11:50 | INFO | train_inner | epoch 083:    270 / 1015 loss=3.83, nll_loss=2.15, ppl=4.44, wps=11724.2, ups=3.33, wpb=3525, bsz=146.2, num_updates=83500, lr=0.00021887, gnorm=1.613, train_wall=30, wall=26132\n",
            "2020-08-04 04:12:20 | INFO | train_inner | epoch 083:    370 / 1015 loss=3.815, nll_loss=2.135, ppl=4.39, wps=11750, ups=3.3, wpb=3559, bsz=156.1, num_updates=83600, lr=0.000218739, gnorm=1.601, train_wall=30, wall=26162\n",
            "2020-08-04 04:12:51 | INFO | train_inner | epoch 083:    470 / 1015 loss=3.863, nll_loss=2.188, ppl=4.56, wps=11614.8, ups=3.29, wpb=3526.9, bsz=147.4, num_updates=83700, lr=0.000218609, gnorm=1.609, train_wall=30, wall=26193\n",
            "2020-08-04 04:13:21 | INFO | train_inner | epoch 083:    570 / 1015 loss=3.776, nll_loss=2.093, ppl=4.27, wps=11793.2, ups=3.28, wpb=3597.2, bsz=184.6, num_updates=83800, lr=0.000218478, gnorm=1.523, train_wall=30, wall=26223\n",
            "2020-08-04 04:13:52 | INFO | train_inner | epoch 083:    670 / 1015 loss=3.843, nll_loss=2.169, ppl=4.5, wps=11801.7, ups=3.28, wpb=3601.2, bsz=171.1, num_updates=83900, lr=0.000218348, gnorm=1.54, train_wall=30, wall=26254\n",
            "2020-08-04 04:14:22 | INFO | train_inner | epoch 083:    770 / 1015 loss=3.875, nll_loss=2.204, ppl=4.61, wps=11644.4, ups=3.28, wpb=3548.9, bsz=155.8, num_updates=84000, lr=0.000218218, gnorm=1.603, train_wall=30, wall=26284\n",
            "2020-08-04 04:14:52 | INFO | train_inner | epoch 083:    870 / 1015 loss=3.877, nll_loss=2.207, ppl=4.62, wps=11735.4, ups=3.3, wpb=3553.6, bsz=151.9, num_updates=84100, lr=0.000218088, gnorm=1.607, train_wall=30, wall=26314\n",
            "2020-08-04 04:15:23 | INFO | train_inner | epoch 083:    970 / 1015 loss=3.889, nll_loss=2.219, ppl=4.65, wps=11419.9, ups=3.27, wpb=3487.7, bsz=151.5, num_updates=84200, lr=0.000217959, gnorm=1.69, train_wall=30, wall=26345\n",
            "2020-08-04 04:15:37 | INFO | fairseq_cli.train | begin validation on \"valid\" subset\n",
            "2020-08-04 04:15:43 | INFO | valid | epoch 083 | valid on 'valid' subset | loss 4.246 | nll_loss 2.548 | ppl 5.85 | wps 27023.4 | wpb 2866.6 | bsz 127.8 | num_updates 84245 | best_loss 4.245\n",
            "2020-08-04 04:15:43 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 04:15:44 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint83.pt (epoch 83 @ 84245 updates, score 4.246) (writing took 1.5836112949982635 seconds)\n",
            "2020-08-04 04:15:44 | INFO | fairseq.data.iterators | Data loading buffer is empty or nearly empty. This may indicate a data loading bottleneck, and increasing the number of workers (--num-workers) may help.\n",
            "2020-08-04 04:15:44 | INFO | fairseq_cli.train | end of epoch 83 (average epoch stats below)\n",
            "2020-08-04 04:15:44 | INFO | train | epoch 083 | loss 3.837 | nll_loss 2.16 | ppl 4.47 | wps 11389.9 | ups 3.21 | wpb 3550.1 | bsz 157.9 | num_updates 84245 | lr 0.0002179 | gnorm 1.592 | train_wall 308 | wall 26366\n",
            "2020-08-04 04:15:44 | INFO | fairseq_cli.train | begin training epoch 83\n",
            "2020-08-04 04:16:01 | INFO | train_inner | epoch 084:     55 / 1015 loss=3.804, nll_loss=2.123, ppl=4.36, wps=9253.2, ups=2.62, wpb=3532.2, bsz=166, num_updates=84300, lr=0.000217829, gnorm=1.583, train_wall=30, wall=26383\n",
            "2020-08-04 04:16:31 | INFO | train_inner | epoch 084:    155 / 1015 loss=3.786, nll_loss=2.101, ppl=4.29, wps=11539.5, ups=3.31, wpb=3491.3, bsz=148.5, num_updates=84400, lr=0.0002177, gnorm=1.608, train_wall=30, wall=26413\n",
            "2020-08-04 04:17:02 | INFO | train_inner | epoch 084:    255 / 1015 loss=3.813, nll_loss=2.132, ppl=4.38, wps=11583.2, ups=3.3, wpb=3504.8, bsz=147.4, num_updates=84500, lr=0.000217571, gnorm=1.613, train_wall=30, wall=26444\n",
            "2020-08-04 04:17:32 | INFO | train_inner | epoch 084:    355 / 1015 loss=3.787, nll_loss=2.104, ppl=4.3, wps=11684.7, ups=3.28, wpb=3560, bsz=167.8, num_updates=84600, lr=0.000217443, gnorm=1.566, train_wall=30, wall=26474\n",
            "2020-08-04 04:18:03 | INFO | train_inner | epoch 084:    455 / 1015 loss=3.805, nll_loss=2.124, ppl=4.36, wps=11724.8, ups=3.28, wpb=3577.5, bsz=166.2, num_updates=84700, lr=0.000217314, gnorm=1.578, train_wall=30, wall=26505\n",
            "2020-08-04 04:18:33 | INFO | train_inner | epoch 084:    555 / 1015 loss=3.792, nll_loss=2.111, ppl=4.32, wps=11815.4, ups=3.28, wpb=3607.4, bsz=166.9, num_updates=84800, lr=0.000217186, gnorm=1.542, train_wall=30, wall=26535\n",
            "2020-08-04 04:19:03 | INFO | train_inner | epoch 084:    655 / 1015 loss=3.913, nll_loss=2.245, ppl=4.74, wps=11622, ups=3.33, wpb=3493.3, bsz=143.3, num_updates=84900, lr=0.000217058, gnorm=1.693, train_wall=30, wall=26565\n",
            "2020-08-04 04:19:34 | INFO | train_inner | epoch 084:    755 / 1015 loss=3.831, nll_loss=2.155, ppl=4.45, wps=11780.2, ups=3.28, wpb=3594.1, bsz=164.6, num_updates=85000, lr=0.00021693, gnorm=1.559, train_wall=30, wall=26596\n",
            "2020-08-04 04:19:34 | INFO | fairseq_cli.train | begin save checkpoint\n",
            "2020-08-04 04:19:35 | INFO | fairseq.checkpoint_utils | saved checkpoint /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_last.pt (epoch 84 @ 85000 updates, score None) (writing took 0.8750277500002994 seconds)\n",
            "2020-08-04 04:19:35 | INFO | fairseq_cli.train | end of epoch 84 (average epoch stats below)\n",
            "2020-08-04 04:19:35 | INFO | train | epoch 084 | loss 3.812 | nll_loss 2.132 | ppl 4.38 | wps 11615.5 | ups 3.28 | wpb 3541.7 | bsz 159.4 | num_updates 85000 | lr 0.00021693 | gnorm 1.594 | train_wall 229 | wall 26597\n",
            "2020-08-04 04:19:35 | INFO | fairseq_cli.train | done training in 26595.4 seconds\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uKmtrRC3Q562",
        "colab_type": "text"
      },
      "source": [
        "###**3.7.   Average the 10 Last Trained Models**\n",
        "\n",
        "Because Stochastic Gradient Descent is used for model optimization, namely applying a mini-batch instead of the entire training set to update the model at each step, the last trained model might overfit to the last used mini-batch.\n",
        "\n",
        "Checkpoint averaging is a method used in machine translation to improve the translation performance of the model by achieving more robust parameters.\n",
        "\n",
        "The `average_checkpoints.py` file from fairseq allows to average the trainable parameters of a user-defined number of pretrained models, in this case the last 10 saved checkpoints."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hVeGhmO-Ppyq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "6da4e558-02ee-4181-a476-304d94a1ac4c"
      },
      "source": [
        "!python /content/fairseq/scripts/average_checkpoints.py --inputs $SAVE \\\n",
        "    --num-epoch-checkpoints 10 --output {SAVE}\"/checkpoint_last10_avg.pt\""
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Namespace(checkpoint_upper_bound=None, inputs=['/content/checkpoints/local_joint_attention_iwslt_de_en'], num_epoch_checkpoints=10, num_update_checkpoints=None, output='/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_last10_avg.pt')\n",
            "averaging checkpoints:  ['/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint83.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint82.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint81.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint80.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint79.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint78.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint77.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint76.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint75.pt', '/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint74.pt']\n",
            "Finished writing averaged checkpoint to /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_last10_avg.pt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8E5FeGOLQWjG",
        "colab_type": "text"
      },
      "source": [
        "###**3.8.   Evaluation of the Trained Model on the [BLEU Benchmark](https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213)** \n",
        "\n",
        "\n",
        ">BLEU stands for **Bilingual Evaluation Understudy**.\n",
        "It is the most commonly used metric amongst the various existing ones for evaluating models that solve sequence-to-sequence problems, i.e., the output is not merely a classification but a sequence of words that may be of different length than the input sequence.  \n",
        "The resulting score is a value between **0 and 100**, where the value 100 means the translation is identical with the human reference translation, while a score of 0 indicates that the machine translation has no matches with the human one.  \n",
        ">>The score is calculated based on **an average of unigram, bigram, trigram and 4-gram precision**, where n-grams are a sequence of n words that occur next to each other in a given text. In the computation of the score, the length of the sentence translated by the model is also penalized if it is shorter compared with the length of the reference translation, and this is known as the **brevity penalty**.  \n",
        "\n",
        "Included in the fairseq toolkit is also the `fairseq-generate` script for evaluating the BLEU score of the trained model on the test set. \n",
        "\n",
        "The above trained model should result in a BLEU score similar to the **35.7** BLEU score reported in the paper, as can be seen on the right side of the [table below](https://arxiv.org/abs/1905.06596).\n",
        "\n",
        "Even though these scores may sound low, one must not forget that even humans cannot always come up with the perfect translation. The scores of **state-of-the-art models** are usually **between 20 and 40**, which makes the results of this model very impressive and they are one of the highest scores achieved in German-to-English translation.\n",
        "\n",
        "**Checkpoint averaging** should as well show slightly better results in the BLEU score compared with the results of the best-performing model (on the validation set).\n",
        "\n",
        "<center><img src=https://d3i71xaburhd42.cloudfront.net/b0f0a5a21619d70748a4dc007983cc111f1b301e/4-Table1-1.png width=\"450\"></center>\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5PnqhDOYQQxc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 581
        },
        "outputId": "b1422d66-ec58-4cff-ff62-2a051f061238"
      },
      "source": [
        "!echo 'Evaluating the best-performing trained model'\n",
        "!fairseq-generate /content/data-bin/iwslt14.joined-dictionary.31K.de-en --user-dir /content/models/ \\\n",
        "    --path {SAVE}\"/checkpoint_best.pt\" \\\n",
        "    --batch-size 32 --beam 5 --remove-bpe --lenpen 1.7 --gen-subset test --quiet\n",
        "\n",
        "!echo 'Evaluating the averaged model'\n",
        "!fairseq-generate /content/data-bin/iwslt14.joined-dictionary.31K.de-en --user-dir /content/models \\\n",
        "    --path {SAVE}\"/checkpoint_last10_avg.pt\" \\\n",
        "    --batch-size 32 --beam 5 --remove-bpe --lenpen 1.7 --gen-subset test --quiet"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Evaluating the best-performing trained model\n",
            "2020-08-04 04:20:06 | INFO | fairseq_cli.generate | Namespace(all_gather_list_size=16384, beam=5, bf16=False, bpe=None, broadcast_buffers=False, bucket_cap_mb=25, checkpoint_suffix='', cpu=False, criterion='cross_entropy', data='/content/data-bin/iwslt14.joined-dictionary.31K.de-en', data_buffer_size=10, dataset_impl=None, ddp_backend='c10d', decoding_format=None, device_id=0, distributed_backend='nccl', distributed_init_method=None, distributed_no_spawn=False, distributed_port=-1, distributed_rank=0, distributed_world_size=1, distributed_wrapper='DDP', diverse_beam_groups=-1, diverse_beam_strength=0.5, diversity_rate=-1.0, empty_cache_freq=0, eval_bleu=False, eval_bleu_args=None, eval_bleu_detok='space', eval_bleu_detok_args=None, eval_bleu_print_samples=False, eval_bleu_remove_bpe=None, eval_tokenized_bleu=False, fast_stat_sync=False, find_unused_parameters=False, fix_batches_to_gpus=False, force_anneal=None, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, gen_subset='test', iter_decode_eos_penalty=0.0, iter_decode_force_max_iter=False, iter_decode_max_iter=10, iter_decode_with_beam=1, iter_decode_with_external_reranker=False, left_pad_source='True', left_pad_target='False', lenpen=1.7, load_alignments=False, localsgd_frequency=3, log_format=None, log_interval=100, lr_scheduler='fixed', lr_shrink=0.1, match_source_len=False, max_len_a=0, max_len_b=200, max_sentences=32, max_source_positions=1024, max_target_positions=1024, max_tokens=None, memory_efficient_bf16=False, memory_efficient_fp16=False, min_len=1, min_loss_scale=0.0001, model_overrides='{}', model_parallel_size=1, momentum=0.99, nbest=1, no_beamable_mm=False, no_early_stop=False, no_progress_bar=False, no_repeat_ngram_size=0, no_seed_provided=True, nprocs_per_node=1, num_batch_buckets=0, num_shards=1, num_workers=1, optimizer='nag', path='/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_best.pt', prefix_size=0, print_alignment=False, print_step=False, profile=False, quantization_config_path=None, quiet=True, remove_bpe='@@ ', replace_unk=None, required_batch_size_multiple=8, results_path=None, retain_dropout=False, retain_dropout_modules=None, retain_iter_history=False, sacrebleu=False, sampling=False, sampling_topk=-1, sampling_topp=-1.0, score_reference=False, seed=1, shard_id=0, skip_invalid_size_inputs_valid_test=False, slowmo_algorithm='LocalSGD', slowmo_momentum=None, source_lang=None, target_lang=None, task='translation', temperature=1.0, tensorboard_logdir='', threshold_loss_scale=None, tokenizer=None, tpu=False, truncate_source=False, unkpen=0, unnormalized=False, upsample_primary=1, user_dir='/content/models/', warmup_updates=0, weight_decay=0.0)\n",
            "2020-08-04 04:20:06 | INFO | fairseq.tasks.translation | [de] dictionary: 30760 types\n",
            "2020-08-04 04:20:06 | INFO | fairseq.tasks.translation | [en] dictionary: 30760 types\n",
            "2020-08-04 04:20:06 | INFO | fairseq.data.data_utils | loaded 6750 examples from: /content/data-bin/iwslt14.joined-dictionary.31K.de-en/test.de-en.de\n",
            "2020-08-04 04:20:06 | INFO | fairseq.data.data_utils | loaded 6750 examples from: /content/data-bin/iwslt14.joined-dictionary.31K.de-en/test.de-en.en\n",
            "2020-08-04 04:20:06 | INFO | fairseq.tasks.translation | /content/data-bin/iwslt14.joined-dictionary.31K.de-en test de-en 6750 examples\n",
            "2020-08-04 04:20:06 | INFO | fairseq_cli.generate | loading model(s) from /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_best.pt\n",
            "  0% 0/211 [00:00<?, ?it/s]/content/fairseq/fairseq/sequence_generator.py:373: UserWarning: This overload of nonzero is deprecated:\n",
            "\tnonzero()\n",
            "Consider using one of the following signatures instead:\n",
            "\tnonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)\n",
            "  batch_idxs = batch_mask.nonzero().squeeze(-1)\n",
            "2020-08-04 04:22:34 | INFO | fairseq_cli.generate | NOTE: hypothesis and token scores are output in base 2\n",
            "2020-08-04 04:22:34 | INFO | fairseq_cli.generate | Translated 6750 sentences (141080 tokens) in 117.8s (57.31 sentences/s, 1197.90 tokens/s)\n",
            "2020-08-04 04:22:34 | INFO | fairseq_cli.generate | Generate test with beam=5: BLEU4 = 35.10, 67.8/42.5/28.5/19.6 (BP=0.986, ratio=0.986, syslen=129291, reflen=131162)\n",
            "Evaluating the averaged model\n",
            "2020-08-04 04:22:37 | INFO | fairseq_cli.generate | Namespace(all_gather_list_size=16384, beam=5, bf16=False, bpe=None, broadcast_buffers=False, bucket_cap_mb=25, checkpoint_suffix='', cpu=False, criterion='cross_entropy', data='/content/data-bin/iwslt14.joined-dictionary.31K.de-en', data_buffer_size=10, dataset_impl=None, ddp_backend='c10d', decoding_format=None, device_id=0, distributed_backend='nccl', distributed_init_method=None, distributed_no_spawn=False, distributed_port=-1, distributed_rank=0, distributed_world_size=1, distributed_wrapper='DDP', diverse_beam_groups=-1, diverse_beam_strength=0.5, diversity_rate=-1.0, empty_cache_freq=0, eval_bleu=False, eval_bleu_args=None, eval_bleu_detok='space', eval_bleu_detok_args=None, eval_bleu_print_samples=False, eval_bleu_remove_bpe=None, eval_tokenized_bleu=False, fast_stat_sync=False, find_unused_parameters=False, fix_batches_to_gpus=False, force_anneal=None, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, gen_subset='test', iter_decode_eos_penalty=0.0, iter_decode_force_max_iter=False, iter_decode_max_iter=10, iter_decode_with_beam=1, iter_decode_with_external_reranker=False, left_pad_source='True', left_pad_target='False', lenpen=1.7, load_alignments=False, localsgd_frequency=3, log_format=None, log_interval=100, lr_scheduler='fixed', lr_shrink=0.1, match_source_len=False, max_len_a=0, max_len_b=200, max_sentences=32, max_source_positions=1024, max_target_positions=1024, max_tokens=None, memory_efficient_bf16=False, memory_efficient_fp16=False, min_len=1, min_loss_scale=0.0001, model_overrides='{}', model_parallel_size=1, momentum=0.99, nbest=1, no_beamable_mm=False, no_early_stop=False, no_progress_bar=False, no_repeat_ngram_size=0, no_seed_provided=True, nprocs_per_node=1, num_batch_buckets=0, num_shards=1, num_workers=1, optimizer='nag', path='/content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_last10_avg.pt', prefix_size=0, print_alignment=False, print_step=False, profile=False, quantization_config_path=None, quiet=True, remove_bpe='@@ ', replace_unk=None, required_batch_size_multiple=8, results_path=None, retain_dropout=False, retain_dropout_modules=None, retain_iter_history=False, sacrebleu=False, sampling=False, sampling_topk=-1, sampling_topp=-1.0, score_reference=False, seed=1, shard_id=0, skip_invalid_size_inputs_valid_test=False, slowmo_algorithm='LocalSGD', slowmo_momentum=None, source_lang=None, target_lang=None, task='translation', temperature=1.0, tensorboard_logdir='', threshold_loss_scale=None, tokenizer=None, tpu=False, truncate_source=False, unkpen=0, unnormalized=False, upsample_primary=1, user_dir='/content/models', warmup_updates=0, weight_decay=0.0)\n",
            "2020-08-04 04:22:37 | INFO | fairseq.tasks.translation | [de] dictionary: 30760 types\n",
            "2020-08-04 04:22:37 | INFO | fairseq.tasks.translation | [en] dictionary: 30760 types\n",
            "2020-08-04 04:22:37 | INFO | fairseq.data.data_utils | loaded 6750 examples from: /content/data-bin/iwslt14.joined-dictionary.31K.de-en/test.de-en.de\n",
            "2020-08-04 04:22:37 | INFO | fairseq.data.data_utils | loaded 6750 examples from: /content/data-bin/iwslt14.joined-dictionary.31K.de-en/test.de-en.en\n",
            "2020-08-04 04:22:37 | INFO | fairseq.tasks.translation | /content/data-bin/iwslt14.joined-dictionary.31K.de-en test de-en 6750 examples\n",
            "2020-08-04 04:22:37 | INFO | fairseq_cli.generate | loading model(s) from /content/checkpoints/local_joint_attention_iwslt_de_en/checkpoint_last10_avg.pt\n",
            "  0% 0/211 [00:00<?, ?it/s]/content/fairseq/fairseq/sequence_generator.py:373: UserWarning: This overload of nonzero is deprecated:\n",
            "\tnonzero()\n",
            "Consider using one of the following signatures instead:\n",
            "\tnonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)\n",
            "  batch_idxs = batch_mask.nonzero().squeeze(-1)\n",
            "2020-08-04 04:24:58 | INFO | fairseq_cli.generate | NOTE: hypothesis and token scores are output in base 2\n",
            "2020-08-04 04:24:58 | INFO | fairseq_cli.generate | Translated 6750 sentences (138858 tokens) in 113.0s (59.73 sentences/s, 1228.71 tokens/s)\n",
            "2020-08-04 04:24:58 | INFO | fairseq_cli.generate | Generate test with beam=5: BLEU4 = 35.48, 68.9/43.5/29.4/20.3 (BP=0.969, ratio=0.970, syslen=127220, reflen=131162)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b2z3PtQlasdR",
        "colab_type": "text"
      },
      "source": [
        "###**3.9.   Interactive Translation**\n",
        "\n",
        "Using the checkpoints file of the averaged model, it is now possible to input a German sentence in the trained model and generate its English translation in an interactive way."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OldcRxzKad1Q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 360
        },
        "outputId": "80897460-d80b-4603-c1a9-403e2b539ad7"
      },
      "source": [
        "# first install some libraries\n",
        "!pip install sacremoses\n",
        "!pip install subword-nmt"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\r\u001b[K     |▍                               | 10kB 28.3MB/s eta 0:00:01\r\u001b[K     |▊                               | 20kB 3.4MB/s eta 0:00:01\r\u001b[K     |█▏                              | 30kB 4.4MB/s eta 0:00:01\r\u001b[K     |█▌                              | 40kB 4.8MB/s eta 0:00:01\r\u001b[K     |█▉                              | 51kB 3.9MB/s eta 0:00:01\r\u001b[K     |██▎                             | 61kB 4.3MB/s eta 0:00:01\r\u001b[K     |██▋                             | 71kB 4.8MB/s eta 0:00:01\r\u001b[K     |███                             | 81kB 5.2MB/s eta 0:00:01\r\u001b[K     |███▍                            | 92kB 5.4MB/s eta 0:00:01\r\u001b[K     |███▊                            | 102kB 5.2MB/s eta 0:00:01\r\u001b[K     |████                            | 112kB 5.2MB/s eta 0:00:01\r\u001b[K     |████▌                           | 122kB 5.2MB/s eta 0:00:01\r\u001b[K     |████▉                           | 133kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████▏                          | 143kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████▋                          | 153kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████                          | 163kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████▎                         | 174kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████▊                         | 184kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████                         | 194kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████▍                        | 204kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████▉                        | 215kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████▏                       | 225kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████▌                       | 235kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████                       | 245kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████▎                      | 256kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████▋                      | 266kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████                      | 276kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████▍                     | 286kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 296kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████▏                    | 307kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████▌                    | 317kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 327kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████▎                   | 337kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████▋                   | 348kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 358kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████▍                  | 368kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████▊                  | 378kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████                  | 389kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████▌                 | 399kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████▉                 | 409kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████▏                | 419kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████▋                | 430kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████                | 440kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████▎               | 450kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████▊               | 460kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████               | 471kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████▍              | 481kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████▉              | 491kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████▏             | 501kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████▌             | 512kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 522kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████▎            | 532kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████▋            | 542kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████            | 552kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████▍           | 563kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████▊           | 573kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████▏          | 583kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████▌          | 593kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████▉          | 604kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████▎         | 614kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████▋         | 624kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 634kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████▍        | 645kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████▊        | 655kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 665kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████▌       | 675kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████▉       | 686kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▏      | 696kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▋      | 706kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 716kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▎     | 727kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▊     | 737kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 747kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▍    | 757kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▉    | 768kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▏   | 778kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▌   | 788kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████   | 798kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▎  | 808kB 5.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▋  | 819kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 829kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▍ | 839kB 5.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▊ | 849kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▏| 860kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▌| 870kB 5.2MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▉| 880kB 5.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 890kB 5.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from sacremoses) (2019.12.20)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses) (0.16.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from sacremoses) (4.41.1)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893260 sha256=6cfb7b8cfe091369799744578856a94eb67238b4b01592faf23b40615ae3c0eb\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sacremoses\n",
            "Successfully installed sacremoses-0.0.43\n",
            "Collecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Installing collected packages: subword-nmt\n",
            "Successfully installed subword-nmt-0.3.7\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kizVRkDF-qEj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "9d259909-a3d7-4f1f-a6ca-03ebd7c09651"
      },
      "source": [
        "# We need to use the bpe tokens code file in 'bpe_codes'\n",
        "# Bpe_codes takes a file that must be in the same directory as specified in the first argument\n",
        "# The first directory indicates where the checkpoints were saved\n",
        "%cd $SAVE\n",
        "!wget https://raw.githubusercontent.com/wejdene14/NMT-Joint/master/code\n",
        "\n",
        "# Define the model and restore the averaged checkpoint\n",
        "de2en = JointAttentionModel.from_pretrained(\n",
        "  SAVE,\n",
        "  checkpoint_file='checkpoint_last10_avg.pt',\n",
        "  data_name_or_path='/content/data-bin/iwslt14.joined-dictionary.31K.de-en/',\n",
        "  tokenizer='moses',\n",
        "  source_lang='de',\n",
        "  target_lang='en',\n",
        "  bpe='subword_nmt',\n",
        "  bpe_codes='code'\n",
        ")\n",
        "\n",
        "# If dropout was used during training, disable dropout before testing\n",
        "de2en.eval()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/checkpoints/local_joint_attention_iwslt_de_en\n",
            "--2020-08-04 04:25:08--  https://raw.githubusercontent.com/wejdene14/NMT-Joint/master/code\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 353021 (345K) [text/plain]\n",
            "Saving to: ‘code’\n",
            "\n",
            "code                100%[===================>] 344.75K  --.-KB/s    in 0.03s   \n",
            "\n",
            "2020-08-04 04:25:08 (9.84 MB/s) - ‘code’ saved [353021/353021]\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GeneratorHubInterface(\n",
              "  (models): ModuleList(\n",
              "    (0): JointAttentionModel(\n",
              "      (encoder): JointAttentionEncoder(\n",
              "        (embed_tokens): Embedding(30760, 256, padding_idx=1)\n",
              "        (embed_positions): SinusoidalPositionalEmbedding()\n",
              "      )\n",
              "      (decoder): JointAttentionDecoder(\n",
              "        (embed_tokens): Embedding(30760, 256, padding_idx=1)\n",
              "        (embed_positions): SinusoidalPositionalEmbedding()\n",
              "        (layers): ModuleList(\n",
              "          (0): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (1): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (2): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (3): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (4): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (5): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (6): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (7): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (8): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (9): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (10): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (11): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (12): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "          (13): ProtectedTransformerDecoderLayer(\n",
              "            (self_attn): ProtectedMultiheadAttention(\n",
              "              (out_proj): Linear(in_features=256, out_features=256, bias=True)\n",
              "            )\n",
              "            (self_attn_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "            (fc1): Linear(in_features=256, out_features=1024, bias=True)\n",
              "            (fc2): Linear(in_features=1024, out_features=256, bias=True)\n",
              "            (final_layer_norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gMDx6Cw0_ZAe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f1f6a14c-5175-4101-f986-3f764b835e27"
      },
      "source": [
        "# Translate a German sentence\n",
        "de = 'Hallo Welt!'\n",
        "en = de2en.translate(de.lower())\n",
        "print(en)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hello world!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rnztyCvk3M_e",
        "colab_type": "text"
      },
      "source": [
        "##**4. Next Step**\n",
        "\n",
        "This new architecture produces state-of-the-art results in the German-English translation and outperforms other models by at least 0.5 in the BLEU score.\n",
        "\n",
        "With this provided Notebook, it is possible to experiment by training on a different and/or larger dataset, such as the [WMT16 English-German](https://drive.google.com/uc?export=download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8) dataset, as well as playing around with the hyperparameters of the model."
      ]
    }
  ]
}