NMT_Bleu_Score.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "200361138_LAB7.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nZfqzG-psHbd"
      },
      "source": [
        "#**Lab 7: Neural Machine Translation (Extra Guide)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r9GBwu1PsSR3"
      },
      "source": [
        "This week, we'll be build a neural machine translation model based on the sequence-to-sequence (seq2seq) models proposed by Sutskever et al., 2014 and Cho et al., 2014. The seq2seq model is widely used in machine translation systems such as Google’s neural machine translation system (GNMT) (Wu et al., 2016).\n",
        "\n",
        "A folder, **nmt_lab_files** has been provided for you. This folder contains 3 files:\n",
        "1. **data.30.vi** - a file. each line of the file contains a Vietnamese sentence to be translated (i.e. the source sentences)\n",
        "2. **data.30.en** - a file. each line of the file contains an English sentence corresponding to the Vietnamese sentence in the same line position. (i.e. the target sentences)\n",
        "3. **nmt_model_keras.py** - the incomplete code for this lab.\n",
        "\n",
        "The doc file provided contains an explanation of the code file and a guide on how to complete the code (by doing 3 tasks). Read the doc file and if you can, complete the code as instructed. When the code is completed, skip to section xx of this notebook. \n",
        "\n",
        "This notebook (prior to section section xx) merely contains further explanation on sections of the code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KyDvxbvTt70n"
      },
      "source": [
        "##**LanguageDict**\n",
        "\n",
        "LanguageDict is a class for creating language dict objects."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AtHvI1pGvMBG"
      },
      "source": [
        "##**The <load_dataset()> Method**\n",
        "\n",
        "This helper method reads from the source and target files to \n",
        "- load max_num_examples sentences, \n",
        "- split the sentences them into train, development and testing, and\n",
        "- return relevant data.\n",
        "The code for this is fully commented. \n",
        "\n",
        "<br>\n",
        "\n",
        "As an example to the kind of ouput returned by this model, let's assume we are translating the sentence 'I like dogs' from English to English (this of course is never the case), such that the tokenized and case normalized source sentence list and target sentence list are as follows:\n",
        "\n",
        "\n",
        "```\n",
        "# In our case this would actually be [['tôi', 'thích', 'thỏ']], i.e the Vietnamese equivalent of the English sentence. \n",
        "# We've used English to English here so we can follow along with the code.\n",
        "source_words = [['i', 'like', 'rabbits']] \n",
        "target_words = [['i', 'like', 'rabbits']]\n",
        "```\n",
        "The word2ids for the source and target language dictionaries would look something like:\n",
        "```\n",
        "source_dict.word2ids = {'<PAD>': 0, '<UNK>': 1, 'i': 2, 'like': 3, 'rabbits':4}\n",
        "\n",
        "# end and start tokens are added for the target words\n",
        "target_dict.word2ids = {'<PAD>': 0, '<UNK>': 1, '<start>': 2, 'i': 3, 'like': 4, 'rabbits':5, '<end>':6}\n",
        "\n",
        "```\n",
        "Let's also assume that we are training and testing on this same dataset of one sentence.\n",
        "The **source words** for train/dev/test will be given as\n",
        "```\n",
        "# a batch_size X max_sent_length array.\n",
        "source_words_train = [[2,3,4]] # corresponding to ['i', 'like', 'rabbits']\n",
        "source_words_dev = [[2,3,4]]  # corresponding to ['i', 'like', 'rabbits']\n",
        "source_words_test = [[2,3,4]] # corresponding to ['i', 'like', 'rabbits']\n",
        "```\n",
        "\n",
        "The **target words** for train data will be given as follows (dev/test don't need target words as the model will provide this):\n",
        "```\n",
        "target_words_train = [[2,3,4,5]] # corresponding to ['<start>', 'i', 'like', 'rabbits']\n",
        "```\n",
        "\n",
        "The **target words labels** for each word will be the word after it. The target word labels for train/dev/test data will be given as follows\n",
        "```\n",
        "target_words_train_labels = [[3,4,5,6]] # corresponding to ['i', 'like', 'rabbits', '<end>']\n",
        "target_words_dev_labels = [[3,4,5,6]] # corresponding to ['i', 'like', 'rabbits', '<end>']\n",
        "target_words_test_labels = [[3,4,5,6]] # corresponding to ['i', 'like', 'rabbits', '<end>']\n",
        "```\n",
        "The dimensions for train target words labels would be expanded to this:\n",
        "`[[3], [4], [5], [6]]`\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cc8KZaJn3uBK"
      },
      "source": [
        "##**The Neural Translation Model (NMT)**\n",
        "\n",
        "For the NMT the network (a system of connected layers/models) used for training differs slightly from the network used for inference. Both use the the seq-to-seq encoder-decoder architecture. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sgtz6XAl4E8D"
      },
      "source": [
        "###**The training mode**\n",
        "\n",
        "**Encoder**\n",
        "\n",
        "Given:\n",
        "- `source_words`: a `batch_size(num_sents) x max_sentence_length` array representing the source words. In our mini example, this would be the Vietnamese equivalent of `['i', 'like', 'rabbits']`; `[['tôi', 'thích', 'thỏ']]`\n",
        "\n",
        "The following steps comprise the encoding network:\n",
        "\n",
        "1. transform `source_words` into `source_words_embeddings` using a randomly initialized embedding lookup. source_words_embeddings is thus a `batch_size(num_sents) x max_sentence_length x embedding_dim` array.\n",
        "2. Apply embedding dropout of `embedding_dropout_rate`.\n",
        "3. Use a single `LSTM` with `hidden_size` units to learn a representation for the source words i.e. to encode the input. \n",
        "\n",
        "    (a.) The hidden and cell states for this `LSTM` are initialized to zeros (i.e. we leave the `initial_states = None` default as is).\n",
        "\n",
        "    (b.) We save the `encoder_output` (the sequence not just the last state); and the encoder (hidden and cell) states. \n",
        "\n",
        "This way, the model encodes a representation for the source words. Task 1 guides you to complete the encoder part of the training model.\n",
        "\n",
        "<br>\n",
        "\n",
        "**Decoder (No Attention)**\n",
        "\n",
        "Given:\n",
        "- `target_words`: a `batch_size(i.e.num_sents in batch) x max_sentence_length+1` array representing the target words. This is a time shifted translation of the source words with an added (prepended) `<START>` token `['<start>', 'i', 'like', 'rabbits']`.\n",
        "\n",
        "The decoding is in the following steps:\n",
        "\n",
        "1. transform `target_words` into `target_words_embeddings` using a randomly initialized embedding lookup. target_words_embeddings is thus a `batch_size x max_sentence_length+1 x embedding_dim` array.\n",
        "\n",
        "2.  Apply embedding dropout of `embedding_dropout_rate`.\n",
        "\n",
        "3. Use a single `LSTM` with `hidden_size` units to learn a representation for the target words. Some context is given to this model by using the encoder states to initialize the decoder lstm. This way the encoder state for `'tôi'` for example is used to learn to the representation (and next word prediction, see number 4.) for the `'<start>'` token, and so on.\n",
        "\n",
        "4. For each token representation, use a dense layer to predict a `target_vocab_size` vector which is the probability that any given word in the target vocabulary is the next word following the represented token. The output `decoder_outputs_train` is thus a `batch_size x max_sent_length x target_vocab_size` array.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-4tsMpCYJUk1"
      },
      "source": [
        "###**The Inference Mode**\n",
        "\n",
        "**Encoder**\n",
        "\n",
        "The inference time encoding follows the same steps as training time encoding.\n",
        "\n",
        "<br>\n",
        "\n",
        "**Decoder (No attention)**\n",
        "\n",
        "During training time, we passed a `batch_size(num_sents) x max_sentence_length` array representing the target words into the decoder lstm. The decoder_lstm learns how to represent a given target sentence using the context from the encoder lstm (that learns to represent a source sentence).  \n",
        "\n",
        "At test time, several things are different:\n",
        "\n",
        "1. We no longer have access to a complete translation of the source sentence (recall that no target_words array exists for dev and test sets). Rather, we initialize the target_words_array as thus:\n",
        "\n",
        "    Each expected sentence contains only a single token index, the index of the `'<start>'` token. So, the target_word_dev/test is a `batch_size x 1` array. (see the nmt.eval() function for this)\n",
        "\n",
        "2. This `batch_size x 1` array is fed to the trained decoder_lstm and the predicted array is a `batch_size x 1 x target_vocab_size` such that taking the argmax of this array accross the dimension 2 will give the most probable next word. \n",
        "\n",
        "    For example, at time_step `0`, the first time step, where the `step_target_words` given is the `batch_size x 1` array containing the `'<start>'` token, the next word prediction of the decoder is for each sentence (in the batch) the initial word in the sentence. \n",
        "\n",
        "3. At the first time step, the decoder_lstm still uses the encoder_states as it's initial states. At subsequent time steps, it uses it's own states from the previous time steps. This is also what the decoder_lstm does at training time but it is made more explicit here as we loop over time steps using a for loop.\n",
        "(see nmt.eval())\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFIKOF5sed68"
      },
      "source": [
        "##**Training Without Attention**\n",
        "\n",
        "If you've completed Tasks 1 and 2, you are ready to train the NMT model without attention.\n",
        "\n",
        "Run the following cells to train the model for 10 epochs. It also shows the model summary of the each model you encapsulated.\n",
        "\n",
        "If you're using a GPU, training will no more than 10 minutes and you will get a BLEU score between 4 and 5. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8V5YoFP3yQO-",
        "outputId": "4de8a5dc-87f1-454e-9905-8b384011c03d"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive/')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "upXa_erqyzKm"
      },
      "source": [
        "# change this to the path to your folder. Remember to start from the home directory\n",
        "PATH = 'MyDrive/nmt/' "
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ffbGcgrRy7p6"
      },
      "source": [
        "PATH_TO_FOLDER = \"/content/drive/\" + PATH"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XfSgKakK0QgV"
      },
      "source": [
        "import sys\n",
        "sys.path.append(PATH_TO_FOLDER)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NmSgv44y0TN9"
      },
      "source": [
        "SOURCE_PATH = PATH_TO_FOLDER + 'data.30.vi'\n",
        "TARGET_PATH = PATH_TO_FOLDER + 'data.30.en'"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OOTMtc_l0_ut"
      },
      "source": [
        "import nmt_model_keras as nmt "
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IsO7wW6U1w2m",
        "outputId": "c4ccb205-629d-4abd-cc1a-993b5a9bc637"
      },
      "source": [
        "nmt.main(SOURCE_PATH, TARGET_PATH, use_attention=False)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loading dictionaries\n",
            "read 24000/3000/3000 train/dev/test batches\n",
            "number of tokens in source: 2034, number of tokens in target:2506\n",
            "Task 1(a): Creating the embedding lookups...\n",
            "\n",
            "Task 1(b): Looking up source and target words...\n",
            "\n",
            "Task 1(c): Creating an encoder\n",
            "WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
            "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
            "\t\t\t\t\t\t Train Model Summary.\n",
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_2 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_1 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_target (Embedding)    (None, None, 100)    250600      input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "embedding_source (Embedding)    (None, None, 100)    203400      input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dropout (Dropout)               (None, None, 100)    0           embedding_source[0][0]           \n",
            "                                                                 embedding_target[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "lstm (LSTM)                     [(None, None, 200),  240800      dropout[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lstm_1 (LSTM)                   [(None, None, 200),  240800      dropout[1][0]                    \n",
            "                                                                 lstm[0][1]                       \n",
            "                                                                 lstm[0][2]                       \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, None, 2506)   503706      lstm_1[0][0]                     \n",
            "==================================================================================================\n",
            "Total params: 1,439,306\n",
            "Trainable params: 1,439,306\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "\t\t\t\t\t\t Inference Time Encoder Model Summary.\n",
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, None)]            0         \n",
            "_________________________________________________________________\n",
            "embedding_source (Embedding) (None, None, 100)         203400    \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, None, 100)         0         \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  [(None, None, 200), (None 240800    \n",
            "=================================================================\n",
            "Total params: 444,200\n",
            "Trainable params: 444,200\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "\n",
            " Putting together the decoder states\n",
            "\t\t\t\t\t\t Decoder Inference Model summary\n",
            "Model: \"model_2\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_2 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_target (Embedding)    (None, None, 100)    250600      input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dropout (Dropout)               (None, None, 100)    0           embedding_target[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "input_3 (InputLayer)            [(None, 200)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_4 (InputLayer)            [(None, 200)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "lstm_1 (LSTM)                   [(None, None, 200),  240800      dropout[1][0]                    \n",
            "                                                                 input_3[0][0]                    \n",
            "                                                                 input_4[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "input_5 (InputLayer)            [(None, None, 200)]  0                                            \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, None, 2506)   503706      lstm_1[1][0]                     \n",
            "==================================================================================================\n",
            "Total params: 995,106\n",
            "Trainable params: 995,106\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "None\n",
            "Starting training epoch 1/10\n",
            "240/240 [==============================] - 29s 97ms/step - loss: 2.3729 - accuracy: 0.1975\n",
            "Time used for epoch 1: 0 m 28 s\n",
            "Evaluating on dev set after epoch 1/10:\n",
            "Model BLEU score: 2.13\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: and i &apos;m going to <unk> the <unk> of the <unk> , and i &apos;m going to <unk> the <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: and i &apos;m going to be <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: i &apos;m going to <unk> the <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and i &apos;m going to <unk> the <unk> of the <unk> of the <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: and i &apos;m going to be <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 2/10\n",
            "240/240 [==============================] - 22s 92ms/step - loss: 1.8261 - accuracy: 0.3029\n",
            "Time used for epoch 2: 0 m 22 s\n",
            "Evaluating on dev set after epoch 2/10:\n",
            "Model BLEU score: 2.17\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: i &apos;m going to <unk> the <unk> of the <unk> of the <unk> of the <unk> of the <unk> of the <unk> of the <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: and then we have to be <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and it &apos;s a <unk> , and we have to be <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 3/10\n",
            "240/240 [==============================] - 23s 95ms/step - loss: 1.7173 - accuracy: 0.3292\n",
            "Time used for epoch 3: 0 m 22 s\n",
            "Evaluating on dev set after epoch 3/10:\n",
            "Model BLEU score: 3.01\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: we have to be <unk> , and we <unk> the <unk> of the <unk> <unk> <unk> <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but the <unk> is <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and the <unk> <unk> , the <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: and it &apos;s <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 4/10\n",
            "240/240 [==============================] - 23s 96ms/step - loss: 1.6395 - accuracy: 0.3476\n",
            "Time used for epoch 4: 0 m 23 s\n",
            "Evaluating on dev set after epoch 4/10:\n",
            "Model BLEU score: 3.14\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: we &apos;re <unk> , and they &apos;re <unk> , and i was <unk> , and i was <unk> to <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but it &apos;s not a <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and there &apos;s a <unk> <unk> , you know , it &apos;s a <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 5/10\n",
            "240/240 [==============================] - 23s 96ms/step - loss: 1.5854 - accuracy: 0.3593\n",
            "Time used for epoch 5: 0 m 23 s\n",
            "Evaluating on dev set after epoch 5/10:\n",
            "Model BLEU score: 3.88\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: and i was <unk> to <unk> <unk> , and i <unk> , and i <unk> it up to the <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but it &apos;s not <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see , <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: it &apos;s <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 6/10\n",
            "240/240 [==============================] - 23s 96ms/step - loss: 1.5451 - accuracy: 0.3662\n",
            "Time used for epoch 6: 0 m 23 s\n",
            "Evaluating on dev set after epoch 6/10:\n",
            "Model BLEU score: 3.90\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: and the <unk> <unk> was <unk> , <unk> , <unk> , <unk> , <unk> , <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but here &apos;s the <unk> of the <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and there &apos;s a <unk> <unk> in the <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 7/10\n",
            "240/240 [==============================] - 23s 96ms/step - loss: 1.5106 - accuracy: 0.3731\n",
            "Time used for epoch 7: 0 m 23 s\n",
            "Evaluating on dev set after epoch 7/10:\n",
            "Model BLEU score: 4.39\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s a <unk> <unk> , and you &apos;re <unk> , and it &apos;s a <unk> <unk> , and the <unk> <unk> <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but that &apos;s what happens .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and there &apos;s a <unk> <unk> <unk> , and you see it .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 8/10\n",
            "240/240 [==============================] - 23s 96ms/step - loss: 1.4822 - accuracy: 0.3793\n",
            "Time used for epoch 8: 0 m 23 s\n",
            "Evaluating on dev set after epoch 8/10:\n",
            "Model BLEU score: 4.55\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s a <unk> <unk> , and it &apos;s a <unk> <unk> , and it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but that &apos;s what it is .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see , you know , it &apos;s a <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: it &apos;s a <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 9/10\n",
            "240/240 [==============================] - 23s 97ms/step - loss: 1.4596 - accuracy: 0.3828\n",
            "Time used for epoch 9: 0 m 23 s\n",
            "Evaluating on dev set after epoch 9/10:\n",
            "Model BLEU score: 4.60\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s a <unk> <unk> , <unk> , <unk> , <unk> , <unk> , <unk> , <unk> , <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but that &apos;s the <unk> of the <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: the <unk> <unk> <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see the <unk> <unk> of the <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is a <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 4 s\n",
            "Starting training epoch 10/10\n",
            "240/240 [==============================] - 22s 93ms/step - loss: 1.4395 - accuracy: 0.3861\n",
            "Time used for epoch 10: 0 m 22 s\n",
            "Evaluating on dev set after epoch 10/10:\n",
            "Model BLEU score: 4.53\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s a <unk> <unk> , and i &apos;m going to <unk> the <unk> of the <unk> , and i &apos;m going to <unk> the <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but this is the <unk> of the <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: <unk> <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see , the <unk> <unk> of the <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Training finished!\n",
            "Time used for training: 4 m 44 s\n",
            "Evaluating on test set:\n",
            "Model BLEU score: 4.76\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: and the <unk> is <unk> <unk> <unk> , and you &apos;re <unk> .\n",
            "Actual Sentence: the second quote is from the head of the u.k. financial services <unk> .\n",
            "2 Sample\n",
            "Predicted Sentence: so it &apos;s a <unk> .\n",
            "Actual Sentence: it gets worse .\n",
            "3 Sample\n",
            "Predicted Sentence: what &apos;s the <unk> of the <unk> ?\n",
            "Actual Sentence: what &apos;s happening here ? how can this be possible ?\n",
            "4 Sample\n",
            "Predicted Sentence: no , no , it &apos;s not a <unk> .\n",
            "Actual Sentence: unfortunately , the answer is yes .\n",
            "5 Sample\n",
            "Predicted Sentence: but of course , you &apos;re going to be a <unk> <unk> of the <unk> <unk> of the <unk> .\n",
            "Actual Sentence: but there &apos;s an <unk> solution which is coming from what is known as the science of <unk> .\n",
            "========================\n",
            "Time used for evaluate on test set: 0 m 4 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yLelP7iTJMvr"
      },
      "source": [
        "Here we are getting the BLEU score of 4.76 for the model without attention."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5d6HmlCMR9jw"
      },
      "source": [
        "Here I have printed out the actual sentence and predicted sentence for each epoch without the attention model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZkHu5GTJW1j"
      },
      "source": [
        "The predicted and the sample outputs for the model without attention are as follows:\n",
        "\n",
        "1 Sample\n",
        "Predicted Sentence: and the <unk> is <unk> <unk> <unk> , and you &apos;re <unk> .\n",
        "\n",
        "Actual Sentence: the second quote is from the head of the u.k. financial services <unk> .\n",
        "\n",
        "2 Sample\n",
        "Predicted Sentence: so it &apos;s a <unk> .\n",
        "\n",
        "Actual Sentence: it gets worse .\n",
        "\n",
        "3 Sample\n",
        "Predicted Sentence: what &apos;s the <unk> of the <unk> ?\n",
        "\n",
        "Actual Sentence: what &apos;s happening here ? how can this be possible ?\n",
        "\n",
        "4 Sample\n",
        "Predicted Sentence: no , no , it &apos;s not a <unk> .\n",
        "\n",
        "Actual Sentence: unfortunately , the answer is yes .\n",
        "\n",
        "5 Sample\n",
        "Predicted Sentence: but of course , you &apos;re going to be a <unk> <unk> of the <unk> <unk> of the <unk> .\n",
        "\n",
        "Actual Sentence: but there &apos;s an <unk> solution which is coming from what is known as the science of <unk> ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlMDC3DJi12c"
      },
      "source": [
        "##**Decoding with Attention**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4cQKwvFqurVY"
      },
      "source": [
        "The inputs to the attention layer are encoder and decoder outputs. The attention mechanism:\n",
        "1. Computes a score (a luong score) for each source word\n",
        "2. Weights the words by their luong scores.\n",
        "3. Concatenates the wieghted encoder representation with the decoder_ouput.\n",
        "This new decoder output will now be the input to the decoder_dense layer. \n",
        "\n",
        "Task 3 description in the doc file outlines the steps for this in detail. Once you have completed this Task, you are now ready to train with attention. Training time will be no more than 10 minutes using a GPU and you should get a bleu score of about 15."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "23t6wfpLkb2F",
        "outputId": "59ce163c-90ba-41fc-f2c2-462ccb553285"
      },
      "source": [
        "nmt.main(SOURCE_PATH, TARGET_PATH, use_attention=True)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loading dictionaries\n",
            "read 24000/3000/3000 train/dev/test batches\n",
            "number of tokens in source: 2034, number of tokens in target:2506\n",
            "Task 1(a): Creating the embedding lookups...\n",
            "\n",
            "Task 1(b): Looking up source and target words...\n",
            "\n",
            "Task 1(c): Creating an encoder\n",
            "WARNING:tensorflow:Layer lstm_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
            "WARNING:tensorflow:Layer lstm_3 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
            "\t\t\t\t\t\t Train Model Summary.\n",
            "Model: \"model_3\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_6 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_source (Embedding)    (None, None, 100)    203400      input_6[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "input_7 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "dropout_1 (Dropout)             (None, None, 100)    0           embedding_source[0][0]           \n",
            "                                                                 embedding_target[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "embedding_target (Embedding)    (None, None, 100)    250600      input_7[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lstm_2 (LSTM)                   [(None, None, 200),  240800      dropout_1[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "lstm_3 (LSTM)                   [(None, None, 200),  240800      dropout_1[1][0]                  \n",
            "                                                                 lstm_2[0][1]                     \n",
            "                                                                 lstm_2[0][2]                     \n",
            "__________________________________________________________________________________________________\n",
            "attention_layer (AttentionLayer (None, None, 400)    0           lstm_2[0][0]                     \n",
            "                                                                 lstm_3[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "dense_1 (Dense)                 (None, None, 2506)   1004906     attention_layer[0][0]            \n",
            "==================================================================================================\n",
            "Total params: 1,940,506\n",
            "Trainable params: 1,940,506\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "\t\t\t\t\t\t Inference Time Encoder Model Summary.\n",
            "Model: \"model_4\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_6 (InputLayer)         [(None, None)]            0         \n",
            "_________________________________________________________________\n",
            "embedding_source (Embedding) (None, None, 100)         203400    \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, None, 100)         0         \n",
            "_________________________________________________________________\n",
            "lstm_2 (LSTM)                [(None, None, 200), (None 240800    \n",
            "=================================================================\n",
            "Total params: 444,200\n",
            "Trainable params: 444,200\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "\n",
            " Putting together the decoder states\n",
            "\t\t\t\t\t\t Decoder Inference Model summary\n",
            "Model: \"model_5\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_7 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_target (Embedding)    (None, None, 100)    250600      input_7[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dropout_1 (Dropout)             (None, None, 100)    0           embedding_target[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "input_8 (InputLayer)            [(None, 200)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_9 (InputLayer)            [(None, 200)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_10 (InputLayer)           [(None, None, 200)]  0                                            \n",
            "__________________________________________________________________________________________________\n",
            "lstm_3 (LSTM)                   [(None, None, 200),  240800      dropout_1[1][0]                  \n",
            "                                                                 input_8[0][0]                    \n",
            "                                                                 input_9[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "attention_layer (AttentionLayer (None, None, 400)    0           input_10[0][0]                   \n",
            "                                                                 lstm_3[1][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "dense_1 (Dense)                 (None, None, 2506)   1004906     attention_layer[1][0]            \n",
            "==================================================================================================\n",
            "Total params: 1,496,306\n",
            "Trainable params: 1,496,306\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "None\n",
            "Starting training epoch 1/10\n",
            "240/240 [==============================] - 28s 103ms/step - loss: 2.3450 - accuracy: 0.2062\n",
            "Time used for epoch 1: 0 m 27 s\n",
            "Evaluating on dev set after epoch 1/10:\n",
            "Model BLEU score: 4.68\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: it &apos;s <unk> , it &apos;s <unk> , it &apos;s <unk> , it &apos;s <unk> , it &apos;s <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but it &apos;s not a <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: this is the <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and the <unk> <unk> is <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: the <unk> is <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 2/10\n",
            "240/240 [==============================] - 25s 103ms/step - loss: 1.5857 - accuracy: 0.3945\n",
            "Time used for epoch 2: 0 m 24 s\n",
            "Evaluating on dev set after epoch 2/10:\n",
            "Model BLEU score: 10.09\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s a <unk> , and every <unk> , <unk> , and it was <unk> , and it was <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but here &apos;s just just the <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: and the <unk> <unk> are <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see this <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 3/10\n",
            "240/240 [==============================] - 25s 105ms/step - loss: 1.3339 - accuracy: 0.4518\n",
            "Time used for epoch 3: 0 m 25 s\n",
            "Evaluating on dev set after epoch 3/10:\n",
            "Model BLEU score: 12.73\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s four <unk> , to each of the <unk> , and then it &apos;s a <unk> , it &apos;s a <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but here &apos;s just the <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: the <unk> of the <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here is the <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 4/10\n",
            "240/240 [==============================] - 25s 106ms/step - loss: 1.1890 - accuracy: 0.4821\n",
            "Time used for epoch 4: 0 m 25 s\n",
            "Evaluating on dev set after epoch 4/10:\n",
            "Model BLEU score: 13.79\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there was four <unk> , to each one of the time , and it &apos;s going through the <unk> , it &apos;s <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but here &apos;s just the beginning .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s the way to <unk> the <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here is the <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this body can be <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 5/10\n",
            "240/240 [==============================] - 25s 103ms/step - loss: 1.1006 - accuracy: 0.5025\n",
            "Time used for epoch 5: 0 m 24 s\n",
            "Evaluating on dev set after epoch 5/10:\n",
            "Model BLEU score: 14.62\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s four <unk> , to every single person to see it through the <unk> , it &apos;s <unk> <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but this is really just beginning .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: the <unk> is <unk> around the axis .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here is the <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 6/10\n",
            "240/240 [==============================] - 24s 102ms/step - loss: 1.0389 - accuracy: 0.5175\n",
            "Time used for epoch 6: 0 m 24 s\n",
            "Evaluating on dev set after epoch 6/10:\n",
            "Model BLEU score: 14.59\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s four <unk> , to every time , and it &apos;s when it moves off the photo , it &apos;s about the <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but this is actually the beginning of the process .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: the <unk> is by the <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here &apos;s <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this can be <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 7/10\n",
            "240/240 [==============================] - 25s 104ms/step - loss: 0.9911 - accuracy: 0.5294\n",
            "Time used for epoch 7: 0 m 24 s\n",
            "Evaluating on dev set after epoch 7/10:\n",
            "Model BLEU score: 14.51\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s four <unk> , to every <unk> , to every walk when it &apos;s gone back to the <unk> , it &apos;s gone .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but this is really really really really been really <unk> .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: and the way to <unk> the <unk> around the morning .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here is the <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 8/10\n",
            "240/240 [==============================] - 24s 100ms/step - loss: 0.9553 - accuracy: 0.5398\n",
            "Time used for epoch 8: 0 m 23 s\n",
            "Evaluating on dev set after epoch 8/10:\n",
            "Model BLEU score: 15.14\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s four genetic <unk> , to each of the <unk> , it &apos;s <unk> to the <unk> , it starts to the <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but this is really just the beginning .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: and the <unk> is <unk> <unk> around the <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here &apos;s the <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 9/10\n",
            "240/240 [==============================] - 25s 102ms/step - loss: 0.9275 - accuracy: 0.5475\n",
            "Time used for epoch 9: 0 m 24 s\n",
            "Evaluating on dev set after epoch 9/10:\n",
            "Model BLEU score: 14.98\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there were four genetic <unk> , to each of the time , it &apos;s about the <unk> , it works in the <unk> , it &apos;s <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but this is really just the beginning of the beginning .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: and the <unk> is by <unk> <unk> .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see this <unk> <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the present <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Starting training epoch 10/10\n",
            "240/240 [==============================] - 25s 104ms/step - loss: 0.9032 - accuracy: 0.5540\n",
            "Time used for epoch 10: 0 m 25 s\n",
            "Evaluating on dev set after epoch 10/10:\n",
            "Model BLEU score: 14.79\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: there &apos;s four genetic <unk> , and every time it falls on this <unk> , it &apos;s <unk> .\n",
            "Actual Sentence: there are four <unk> <unk> that , each time this ring <unk> it , as it <unk> the <unk> of the display , it <unk> up a position signal .\n",
            "2 Sample\n",
            "Predicted Sentence: but it &apos;s really just the beginning of the beginning .\n",
            "Actual Sentence: but this is really just the beginning .\n",
            "3 Sample\n",
            "Predicted Sentence: it &apos;s been formed <unk> around the axis .\n",
            "Actual Sentence: it <unk> this by <unk> <unk> about two <unk> .\n",
            "4 Sample\n",
            "Predicted Sentence: and you can see here is the <unk> <unk> .\n",
            "Actual Sentence: so as you can see here , this is a , <unk> <unk> <unk> board .\n",
            "5 Sample\n",
            "Predicted Sentence: this is the <unk> .\n",
            "Actual Sentence: these blocks <unk> <unk> .\n",
            "========================\n",
            "Time used for evaluate on dev set: 0 m 5 s\n",
            "Training finished!\n",
            "Time used for training: 5 m 6 s\n",
            "Evaluating on test set:\n",
            "Model BLEU score: 15.40\n",
            "===== PRINTING OUTPUT====\n",
            "1 Sample\n",
            "Predicted Sentence: the second path of the <unk> <unk> people from the <unk> <unk> <unk> <unk> .\n",
            "Actual Sentence: the second quote is from the head of the u.k. financial services <unk> .\n",
            "2 Sample\n",
            "Predicted Sentence: so it &apos;s worse than .\n",
            "Actual Sentence: it gets worse .\n",
            "3 Sample\n",
            "Predicted Sentence: what &apos;s happening here ? what is this possible ?\n",
            "Actual Sentence: what &apos;s happening here ? how can this be possible ?\n",
            "4 Sample\n",
            "Predicted Sentence: it &apos;s not <unk> , right ?\n",
            "Actual Sentence: unfortunately , the answer is yes .\n",
            "5 Sample\n",
            "Predicted Sentence: but , there &apos;s a very exciting <unk> to know that is a <unk> scientist .\n",
            "Actual Sentence: but there &apos;s an <unk> solution which is coming from what is known as the science of <unk> .\n",
            "========================\n",
            "Time used for evaluate on test set: 0 m 5 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AJThB_jAKJWK"
      },
      "source": [
        "In this model with attention we are getting the BLEU score of 15.40."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AaQ-5vnKRtpH"
      },
      "source": [
        "Here i have printed out the actual sentence and predicted sentence for each epoch for with attention model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYdgBmLrKZxW"
      },
      "source": [
        "The predicted and actual sentence for attention model are as follows:\n",
        "\n",
        "1 Sample\n",
        "Predicted Sentence: the second path of the <unk> <unk> people from the <unk> <unk> <unk> <unk> .\n",
        "\n",
        "Actual Sentence: the second quote is from the head of the u.k. financial services <unk> .\n",
        "\n",
        "2 Sample\n",
        "Predicted Sentence: so it &apos;s worse than .\n",
        "\n",
        "Actual Sentence: it gets worse .\n",
        "\n",
        "3 Sample\n",
        "Predicted Sentence: what &apos;s happening here ? what is this possible ?\n",
        "\n",
        "Actual Sentence: what &apos;s happening here ? how can this be possible ?\n",
        "\n",
        "4 Sample\n",
        "Predicted Sentence: it &apos;s not <unk> , right ?\n",
        "\n",
        "Actual Sentence: unfortunately , the answer is yes .\n",
        "\n",
        "5 Sample\n",
        "Predicted Sentence: but , there &apos;s a very exciting <unk> to know that is a <unk> scientist .\n",
        "\n",
        "Actual Sentence: but there &apos;s an <unk> solution which is coming from what is known as the science of <unk> .\n",
        "\n"
      ]
    }
  ]
}