global_enc_dec_with_(dot_Alignment_model)LuongAttention.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mejJonsu2o7r"
      },
      "source": [
        "#### Taken from [tensorflow-tutorial ](https://www.tensorflow.org/tutorials/text/nmt_with_attention ) and added a *translate_batch()* function to translate a batch and dump outputs into a file"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# Neural machine translation with attention"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RNxZQoi6_gyF",
        "outputId": "1d57b02b-aa9d-4137-ce5e-2924c40262d5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tnxXKDjq3jEL"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import unicodedata\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import io\n",
        "import time\n",
        "from spanish_dataset import NMTDataset\n",
        "from tensorflow.python.ops import math_ops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YsrDDMlKA2nd",
        "outputId": "8bb73b61-9bb6-495c-90cd-99f75665c45e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ls: cannot access 'utils': No such file or directory\n"
          ]
        }
      ],
      "source": [
        "!ls utils\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GOi42V79Ydlr"
      },
      "source": [
        "We'll use the same dataset we worked on notebook-1 (text-processing). For our convenience we've created a utils/dataset.py file which returns train and validation tf.data.Dataset objects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sy9mcFmxBP4J"
      },
      "outputs": [],
      "source": [
        "BUFFER_SIZE = 90000\n",
        "BATCH_SIZE = 32\n",
        "num_examples = 80000\n",
        "\n",
        "dataset_creator = NMTDataset('en-spa')\n",
        "train_dataset, val_dataset, inp_lang, targ_lang = dataset_creator.call(num_examples, BUFFER_SIZE, BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wVtSz89FBk0D",
        "outputId": "21b55126-3575-494a-9786-f33129031729"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Inpute Vocabulary Size: 17592\n",
            "Target Vocabulary Size: 9219\n"
          ]
        }
      ],
      "source": [
        "\n",
        "print(\"Inpute Vocabulary Size: {}\".format(len(inp_lang.word_index)))\n",
        "print(\"Target Vocabulary Size: {}\".format(len(targ_lang.word_index)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qc6-NK1GtWQt",
        "outputId": "2f6b898a-7d5f-4345-cc9c-7783f23d91b2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(TensorShape([32, 20]), TensorShape([32, 14]))"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "example_input_batch, example_target_batch = next(iter(train_dataset))\n",
        "example_input_batch.shape, example_target_batch.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNfHIF71ulLu"
      },
      "source": [
        "## Write the encoder and decoder model\n",
        "\n",
        "Implement an encoder-decoder model with attention which you can read about in the TensorFlow [Neural Machine Translation (seq2seq) tutorial](https://github.com/tensorflow/nmt). This example uses a more recent set of APIs. This notebook implements the [attention equations](https://github.com/tensorflow/nmt#background-on-the-attention-mechanism) from the seq2seq tutorial. The following diagram shows that each input words is assigned a weight by the attention mechanism which is then used by the decoder to predict the next word in the sentence. The below picture and formulas are an example of attention mechanism from [Luong's paper](https://arxiv.org/abs/1508.04025v5).\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_mechanism.jpg\" width=\"500\" alt=\"attention mechanism\">\n",
        "\n",
        "The input is put through an encoder model which gives us the encoder output of shape *(batch_size, max_length, hidden_size)* and the encoder hidden state of shape *(batch_size, hidden_size)*.\n",
        "\n",
        "Here are the equations that are implemented:\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_0.jpg\" alt=\"attention equation 0\" width=\"800\">\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_1.jpg\" alt=\"attention equation 1\" width=\"800\">\n",
        "\n",
        "This tutorial uses [Luong attention](https://arxiv.org/abs/1508.04025) for the encoder. Let's decide on notation before writing the simplified form:\n",
        "\n",
        "* FC = Fully connected (dense) layer\n",
        "* EO = Encoder output\n",
        "* H = hidden state\n",
        "* X = input to the decoder\n",
        "\n",
        "And the pseudo-code:\n",
        "\n",
        "* `score = MatMul(Transpose(H)xE0)`\n",
        "* `attention weights = softmax(score, axis = 1)`. Softmax by default is applied on the last axis but here we want to apply it on the *1st axis*, since the shape of score is *(batch_size, max_length, hidden_size)*. `Max_length` is the length of our input. Since we are trying to assign a weight to each input, softmax should be applied on that axis.\n",
        "* `context vector = sum(attention weights * EO, axis = 1)`. Same reason as above for choosing axis as 1.\n",
        "* `embedding output` = The input to the decoder X is passed through an embedding layer.\n",
        "* `merged vector = concat(embedding output, context vector)`\n",
        "* This merged vector is then given to the GRU\n",
        "\n",
        "The shapes of all the vectors at each step have been specified in the comments in the code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dgVsBjbVDBwX"
      },
      "outputs": [],
      "source": [
        "# Define some useful parameters for further use\n",
        "\n",
        "vocab_inp_size = len(inp_lang.word_index)+1\n",
        "vocab_tar_size = len(targ_lang.word_index)+1\n",
        "max_length_input = example_input_batch.shape[1]\n",
        "max_length_output = example_target_batch.shape[1]\n",
        "\n",
        "embedding_dim = 250\n",
        "units = 512\n",
        "steps_per_epoch = num_examples//BATCH_SIZE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nZ2rI24i3jFg"
      },
      "outputs": [],
      "source": [
        "# Encoder is composed of embedding layer and then one GRU layer. It produces outputs and last hidden states.\n",
        "# Encoder Outputs shape = (BATCH_SIZE, max_length_input, units)\n",
        "# Last Hidden State Shape = (BATCH_SIZE, units)\n",
        "\n",
        "class Encoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):\n",
        "    super(Encoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.enc_units = enc_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.enc_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "\n",
        "  def call(self, x, hidden):\n",
        "    x = self.embedding(x)\n",
        "    output, state = self.gru(x, initial_state = hidden)\n",
        "    return output, state\n",
        "\n",
        "  def initialize_hidden_state(self):\n",
        "    return tf.zeros((self.batch_sz, self.enc_units))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "60gSVh05Jl6l",
        "outputId": "dfd5a059-7f9a-4c57-a6ee-323e6aebecb5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Encoder output shape: (batch size, sequence length, units) (32, 20, 512)\n",
            "Encoder Hidden state shape: (batch size, units) (32, 512)\n"
          ]
        }
      ],
      "source": [
        "encoder = Encoder(vocab_inp_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "# sample input\n",
        "sample_hidden = encoder.initialize_hidden_state()\n",
        "sample_output, sample_hidden = encoder(example_input_batch, sample_hidden)\n",
        "print ('Encoder output shape: (batch size, sequence length, units) {}'.format(sample_output.shape))\n",
        "print ('Encoder Hidden state shape: (batch size, units) {}'.format(sample_hidden.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "umohpBN2OM94"
      },
      "outputs": [],
      "source": [
        "def LuongAttention(query, values):\n",
        "  # query hidden state shape == (batch_size, hidden size)\n",
        "  # query_with_time_axis shape == (batch_size, 1, hidden size)\n",
        "  # values shape == (batch_size, max_len, hidden size)\n",
        "  # we are doing this to broadcast addition along the time axis to calculate the score\n",
        "\n",
        "  query_with_time_axis = tf.expand_dims(query, 1)\n",
        "  # score shape == (batch_size, max_length, 1)\n",
        "  # we get 1 at the last axis because we are applying score to self.V\n",
        "  # the shape of the tensor before applying self.V is (batch_size, max_length, units)\n",
        "\n",
        "  #-------- COMPUTING EQUATION (4: Luong's Attention)  (Dot Alignment Model)  ---------#\n",
        "  # score shape == (batch_size,1,maxLength)\n",
        "  score = tf.matmul(query_with_time_axis, values, transpose_b=True)\n",
        "  # print(f'score is{score.shape}')\n",
        "\n",
        "  #-------- COMPUTING EQUATION (1) ------------#\n",
        "  # attention_weights shape == (batch_size,1,maxLength)\n",
        "  attention_weights = tf.nn.softmax(score, axis=1)\n",
        "  # print(f'attention_weights is{attention_weights.shape}')\n",
        "\n",
        "\n",
        "  # Reshape attention_weights to shape = (batch_size, max_length, 1)\n",
        "  attention_weights = tf.reshape(attention_weights, shape=(-1, attention_weights.shape[2], 1))\n",
        "\n",
        "  #---------- COMPUTING EQUATION (2) -----------#\n",
        "  # context_vector shape after sum == (batch_size, hidden_size)\n",
        "  context_vector = attention_weights * values\n",
        "  # Context vector is passed on to the curren time step's decoder cell\n",
        "  context_vector = tf.reduce_sum(context_vector, axis=1)\n",
        "\n",
        "  return context_vector, attention_weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k534zTHiDjQU",
        "outputId": "59e59862-b1df-4937-a88f-b872b547da9c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "context vector shape: (batch size, units) (32, 512)\n",
            "Attention weights shape: (batch_size, sequence_length, 1) (32, 20, 1)\n"
          ]
        }
      ],
      "source": [
        "attention_result, attention_weights = LuongAttention(sample_hidden, sample_output)\n",
        "\n",
        "print(\"context vector shape: (batch size, units) {}\".format(attention_result.shape))\n",
        "print(\"Attention weights shape: (batch_size, sequence_length, 1) {}\".format(attention_weights.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZzBum9KecI_"
      },
      "source": [
        "Decoder based on Luong Attention architecture"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yJ_B3mhW3jFk"
      },
      "outputs": [],
      "source": [
        "class Decoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):\n",
        "    super(Decoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.dec_units = dec_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.dec_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "  def call(self, x, hidden, enc_output):\n",
        "\n",
        "    x = self.embedding(x) # (m,1,emb_dim)\n",
        "\n",
        "    output,state = self.gru(x) # output (m,1,n_s) state (m,n_s)\n",
        "    context_vector, attention_weights = LuongAttention(state, enc_output)  # context_vector (m,n_a) attention_weigths (m,maxlen,1)\n",
        "    output = tf.concat([tf.expand_dims(context_vector, 1), output], axis=-1) # output  (m,1,n_a+n_s)\n",
        "\n",
        "    output = tf.reshape(output, (-1, output.shape[2])) # output (m,n_a+n_s)\n",
        "\n",
        "    # output shape == (batch_size, vocab)\n",
        "    x = self.fc(output) #\n",
        "\n",
        "    return x, state, attention_weights\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P5UY8wko3jFp",
        "outputId": "e0135bb9-a776-4edb-96b7-6f461b36fe28"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Decoder output shape: (batch_size, vocab size) (32, 9220)\n"
          ]
        }
      ],
      "source": [
        "decoder = Decoder(vocab_tar_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "sample_decoder_output, _, _ = decoder(tf.random.uniform((BATCH_SIZE, 1)),\n",
        "                                      sample_hidden, sample_output)\n",
        "\n",
        "print ('Decoder output shape: (batch_size, vocab size) {}'.format(sample_decoder_output.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ch_71VbIRfK"
      },
      "source": [
        "## Define the optimizer and the loss function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WmTHr5iV3jFr"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMVWzzsfNl4e"
      },
      "source": [
        "## Checkpoints (Object-based saving)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Zj8bXQTgNwrF"
      },
      "outputs": [],
      "source": [
        "checkpoint_dir = './training_checkpoints'\n",
        "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
        "checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
        "                                 encoder=encoder,\n",
        "                                 decoder=decoder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpObfY22IddU"
      },
      "source": [
        "## Training\n",
        "\n",
        "1. Pass the *input* through the *encoder* which return *encoder output* and the *encoder hidden state*.\n",
        "2. The encoder output, encoder hidden state and the decoder input (which is the *start token*) is passed to the decoder.\n",
        "3. The decoder returns the *predictions* and the *decoder hidden state*.\n",
        "4. The decoder hidden state is then passed back into the model and the predictions are used to calculate the loss.\n",
        "5. Use *teacher forcing* to decide the next input to the decoder.\n",
        "6. *Teacher forcing* is the technique where the *target word* is passed as the *next input* to the decoder.\n",
        "7. The final step is to calculate the gradients and apply it to the optimizer and backpropagate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sC9ArXSsVfqn"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_step(inp, targ, enc_hidden):\n",
        "  loss = 0\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "    enc_output, enc_hidden = encoder(inp, enc_hidden)\n",
        "\n",
        "    dec_hidden = enc_hidden\n",
        "\n",
        "    dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "\n",
        "    # Teacher forcing - feeding the target as the next input\n",
        "    for t in range(1, targ.shape[1]):\n",
        "      # passing enc_output to the decoder\n",
        "      predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)\n",
        "\n",
        "      loss += loss_function(targ[:, t], predictions)\n",
        "\n",
        "      # using teacher forcing\n",
        "      dec_input = tf.expand_dims(targ[:, t], 1)\n",
        "\n",
        "  batch_loss = (loss / int(targ.shape[1]))\n",
        "\n",
        "  variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, variables))\n",
        "\n",
        "  return batch_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ddefjBMa3jF0",
        "outputId": "37af33ec-84a7-413d-ec62-3b6731353a18"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1 Batch 0 Loss 0.9396\n",
            "Epoch 1 Batch 100 Loss 1.0041\n",
            "Epoch 1 Batch 200 Loss 0.7799\n",
            "Epoch 1 Batch 300 Loss 0.8605\n",
            "Epoch 1 Batch 400 Loss 0.8701\n",
            "Epoch 1 Batch 500 Loss 0.8332\n",
            "Epoch 1 Batch 600 Loss 0.8801\n",
            "Epoch 1 Batch 700 Loss 0.9999\n",
            "Epoch 1 Batch 800 Loss 0.8139\n",
            "Epoch 1 Batch 900 Loss 0.8113\n",
            "Epoch 1 Batch 1000 Loss 0.9146\n",
            "Epoch 1 Batch 1100 Loss 0.7331\n",
            "Epoch 1 Batch 1200 Loss 0.9367\n",
            "Epoch 1 Batch 1300 Loss 0.8179\n",
            "Epoch 1 Batch 1400 Loss 0.8276\n",
            "Epoch 1 Batch 1500 Loss 0.7064\n",
            "Epoch 1 Batch 1600 Loss 0.8811\n",
            "Epoch 1 Batch 1700 Loss 0.9360\n",
            "Epoch 1 Batch 1800 Loss 0.8159\n",
            "Epoch 1 Batch 1900 Loss 0.8807\n",
            "Epoch 1 Loss 0.6789\n",
            "Time taken for 1 epoch 62.23703646659851 sec\n",
            "\n",
            "Epoch 2 Batch 0 Loss 0.4986\n",
            "Epoch 2 Batch 100 Loss 0.5722\n",
            "Epoch 2 Batch 200 Loss 0.6751\n",
            "Epoch 2 Batch 300 Loss 0.4546\n",
            "Epoch 2 Batch 400 Loss 0.5929\n",
            "Epoch 2 Batch 500 Loss 0.7433\n",
            "Epoch 2 Batch 600 Loss 0.5801\n",
            "Epoch 2 Batch 700 Loss 0.6339\n",
            "Epoch 2 Batch 800 Loss 0.5005\n",
            "Epoch 2 Batch 900 Loss 0.5257\n",
            "Epoch 2 Batch 1000 Loss 0.5703\n",
            "Epoch 2 Batch 1100 Loss 0.5200\n",
            "Epoch 2 Batch 1200 Loss 0.4938\n",
            "Epoch 2 Batch 1300 Loss 0.6134\n",
            "Epoch 2 Batch 1400 Loss 0.4925\n",
            "Epoch 2 Batch 1500 Loss 0.5590\n",
            "Epoch 2 Batch 1600 Loss 0.6172\n",
            "Epoch 2 Batch 1700 Loss 0.5520\n",
            "Epoch 2 Batch 1800 Loss 0.7702\n",
            "Epoch 2 Batch 1900 Loss 0.5171\n",
            "Epoch 2 Loss 0.4728\n",
            "Time taken for 1 epoch 62.04894709587097 sec\n",
            "\n",
            "Epoch 3 Batch 0 Loss 0.3972\n",
            "Epoch 3 Batch 100 Loss 0.5037\n",
            "Epoch 3 Batch 200 Loss 0.4661\n",
            "Epoch 3 Batch 300 Loss 0.4638\n",
            "Epoch 3 Batch 400 Loss 0.4203\n",
            "Epoch 3 Batch 500 Loss 0.3677\n",
            "Epoch 3 Batch 600 Loss 0.5347\n",
            "Epoch 3 Batch 700 Loss 0.4777\n",
            "Epoch 3 Batch 800 Loss 0.3498\n",
            "Epoch 3 Batch 900 Loss 0.4253\n",
            "Epoch 3 Batch 1000 Loss 0.3584\n",
            "Epoch 3 Batch 1100 Loss 0.3964\n",
            "Epoch 3 Batch 1200 Loss 0.5583\n",
            "Epoch 3 Batch 1300 Loss 0.6251\n",
            "Epoch 3 Batch 1400 Loss 0.4684\n",
            "Epoch 3 Batch 1500 Loss 0.5801\n",
            "Epoch 3 Batch 1600 Loss 0.4516\n",
            "Epoch 3 Batch 1700 Loss 0.4918\n",
            "Epoch 3 Batch 1800 Loss 0.5195\n",
            "Epoch 3 Batch 1900 Loss 0.5406\n",
            "Epoch 3 Loss 0.3758\n",
            "Time taken for 1 epoch 60.90569567680359 sec\n",
            "\n",
            "Epoch 4 Batch 0 Loss 0.5269\n",
            "Epoch 4 Batch 100 Loss 0.4392\n",
            "Epoch 4 Batch 200 Loss 0.3844\n",
            "Epoch 4 Batch 300 Loss 0.4213\n",
            "Epoch 4 Batch 400 Loss 0.4611\n",
            "Epoch 4 Batch 500 Loss 0.3996\n",
            "Epoch 4 Batch 600 Loss 0.4571\n",
            "Epoch 4 Batch 700 Loss 0.4296\n",
            "Epoch 4 Batch 800 Loss 0.4211\n",
            "Epoch 4 Batch 900 Loss 0.4351\n",
            "Epoch 4 Batch 1000 Loss 0.3984\n",
            "Epoch 4 Batch 1100 Loss 0.3065\n",
            "Epoch 4 Batch 1200 Loss 0.3369\n",
            "Epoch 4 Batch 1300 Loss 0.4308\n",
            "Epoch 4 Batch 1400 Loss 0.4267\n",
            "Epoch 4 Batch 1500 Loss 0.4565\n",
            "Epoch 4 Batch 1600 Loss 0.4262\n",
            "Epoch 4 Batch 1700 Loss 0.3338\n",
            "Epoch 4 Batch 1800 Loss 0.3667\n",
            "Epoch 4 Batch 1900 Loss 0.4812\n",
            "Epoch 4 Loss 0.3283\n",
            "Time taken for 1 epoch 61.29142785072327 sec\n",
            "\n",
            "Epoch 5 Batch 0 Loss 0.4104\n",
            "Epoch 5 Batch 100 Loss 0.3782\n",
            "Epoch 5 Batch 200 Loss 0.4570\n",
            "Epoch 5 Batch 300 Loss 0.2519\n",
            "Epoch 5 Batch 400 Loss 0.3247\n",
            "Epoch 5 Batch 500 Loss 0.2717\n",
            "Epoch 5 Batch 600 Loss 0.2396\n",
            "Epoch 5 Batch 700 Loss 0.3818\n",
            "Epoch 5 Batch 800 Loss 0.3417\n",
            "Epoch 5 Batch 900 Loss 0.3894\n",
            "Epoch 5 Batch 1000 Loss 0.3328\n",
            "Epoch 5 Batch 1100 Loss 0.4942\n",
            "Epoch 5 Batch 1200 Loss 0.3304\n",
            "Epoch 5 Batch 1300 Loss 0.3955\n",
            "Epoch 5 Batch 1400 Loss 0.3750\n",
            "Epoch 5 Batch 1500 Loss 0.4465\n",
            "Epoch 5 Batch 1600 Loss 0.4804\n",
            "Epoch 5 Batch 1700 Loss 0.3671\n",
            "Epoch 5 Batch 1800 Loss 0.2866\n",
            "Epoch 5 Batch 1900 Loss 0.3077\n",
            "Epoch 5 Loss 0.2944\n",
            "Time taken for 1 epoch 60.43047618865967 sec\n",
            "\n",
            "Epoch 6 Batch 0 Loss 0.3833\n",
            "Epoch 6 Batch 100 Loss 0.3933\n",
            "Epoch 6 Batch 200 Loss 0.2440\n",
            "Epoch 6 Batch 300 Loss 0.3349\n",
            "Epoch 6 Batch 400 Loss 0.2219\n",
            "Epoch 6 Batch 500 Loss 0.4382\n",
            "Epoch 6 Batch 600 Loss 0.2565\n",
            "Epoch 6 Batch 700 Loss 0.2702\n",
            "Epoch 6 Batch 800 Loss 0.4502\n",
            "Epoch 6 Batch 900 Loss 0.3075\n",
            "Epoch 6 Batch 1000 Loss 0.3826\n",
            "Epoch 6 Batch 1100 Loss 0.3261\n",
            "Epoch 6 Batch 1200 Loss 0.4236\n",
            "Epoch 6 Batch 1300 Loss 0.4192\n",
            "Epoch 6 Batch 1400 Loss 0.4799\n",
            "Epoch 6 Batch 1500 Loss 0.3596\n",
            "Epoch 6 Batch 1600 Loss 0.3900\n",
            "Epoch 6 Batch 1700 Loss 0.3130\n",
            "Epoch 6 Batch 1800 Loss 0.4302\n",
            "Epoch 6 Batch 1900 Loss 0.3087\n",
            "Epoch 6 Loss 0.2690\n",
            "Time taken for 1 epoch 61.81023836135864 sec\n",
            "\n",
            "Epoch 7 Batch 0 Loss 0.3190\n",
            "Epoch 7 Batch 100 Loss 0.2847\n",
            "Epoch 7 Batch 200 Loss 0.3458\n",
            "Epoch 7 Batch 300 Loss 0.3582\n",
            "Epoch 7 Batch 400 Loss 0.2816\n",
            "Epoch 7 Batch 500 Loss 0.2863\n",
            "Epoch 7 Batch 600 Loss 0.1789\n",
            "Epoch 7 Batch 700 Loss 0.3077\n",
            "Epoch 7 Batch 800 Loss 0.3490\n",
            "Epoch 7 Batch 900 Loss 0.3644\n",
            "Epoch 7 Batch 1000 Loss 0.3689\n",
            "Epoch 7 Batch 1100 Loss 0.3001\n",
            "Epoch 7 Batch 1200 Loss 0.3401\n",
            "Epoch 7 Batch 1300 Loss 0.2556\n",
            "Epoch 7 Batch 1400 Loss 0.2913\n",
            "Epoch 7 Batch 1500 Loss 0.2544\n",
            "Epoch 7 Batch 1600 Loss 0.3035\n",
            "Epoch 7 Batch 1700 Loss 0.3170\n",
            "Epoch 7 Batch 1800 Loss 0.3908\n",
            "Epoch 7 Batch 1900 Loss 0.3204\n",
            "Epoch 7 Loss 0.2496\n",
            "Time taken for 1 epoch 61.02233266830444 sec\n",
            "\n",
            "Epoch 8 Batch 0 Loss 0.2634\n",
            "Epoch 8 Batch 100 Loss 0.3477\n",
            "Epoch 8 Batch 200 Loss 0.2539\n",
            "Epoch 8 Batch 300 Loss 0.3423\n",
            "Epoch 8 Batch 400 Loss 0.3386\n",
            "Epoch 8 Batch 500 Loss 0.2206\n",
            "Epoch 8 Batch 600 Loss 0.3387\n",
            "Epoch 8 Batch 700 Loss 0.3699\n",
            "Epoch 8 Batch 800 Loss 0.2410\n",
            "Epoch 8 Batch 900 Loss 0.3695\n",
            "Epoch 8 Batch 1000 Loss 0.3668\n",
            "Epoch 8 Batch 1100 Loss 0.1905\n",
            "Epoch 8 Batch 1200 Loss 0.2657\n",
            "Epoch 8 Batch 1300 Loss 0.3370\n",
            "Epoch 8 Batch 1400 Loss 0.2619\n",
            "Epoch 8 Batch 1500 Loss 0.3350\n",
            "Epoch 8 Batch 1600 Loss 0.2488\n",
            "Epoch 8 Batch 1700 Loss 0.3273\n",
            "Epoch 8 Batch 1800 Loss 0.3169\n",
            "Epoch 8 Batch 1900 Loss 0.3559\n",
            "Epoch 8 Loss 0.2340\n",
            "Time taken for 1 epoch 60.71896266937256 sec\n",
            "\n",
            "Epoch 9 Batch 0 Loss 0.2346\n",
            "Epoch 9 Batch 100 Loss 0.2278\n",
            "Epoch 9 Batch 200 Loss 0.1999\n",
            "Epoch 9 Batch 300 Loss 0.3314\n",
            "Epoch 9 Batch 400 Loss 0.2058\n",
            "Epoch 9 Batch 500 Loss 0.2849\n",
            "Epoch 9 Batch 600 Loss 0.2798\n",
            "Epoch 9 Batch 700 Loss 0.2690\n",
            "Epoch 9 Batch 800 Loss 0.2693\n",
            "Epoch 9 Batch 900 Loss 0.2640\n",
            "Epoch 9 Batch 1000 Loss 0.2741\n",
            "Epoch 9 Batch 1100 Loss 0.3470\n",
            "Epoch 9 Batch 1200 Loss 0.1867\n",
            "Epoch 9 Batch 1300 Loss 0.2501\n",
            "Epoch 9 Batch 1400 Loss 0.1963\n",
            "Epoch 9 Batch 1500 Loss 0.2423\n",
            "Epoch 9 Batch 1600 Loss 0.2773\n",
            "Epoch 9 Batch 1700 Loss 0.3263\n",
            "Epoch 9 Batch 1800 Loss 0.4144\n",
            "Epoch 9 Batch 1900 Loss 0.3243\n",
            "Epoch 9 Loss 0.2211\n",
            "Time taken for 1 epoch 60.16207528114319 sec\n",
            "\n",
            "Epoch 10 Batch 0 Loss 0.2490\n",
            "Epoch 10 Batch 100 Loss 0.2662\n",
            "Epoch 10 Batch 200 Loss 0.2878\n",
            "Epoch 10 Batch 300 Loss 0.2916\n",
            "Epoch 10 Batch 400 Loss 0.2352\n",
            "Epoch 10 Batch 500 Loss 0.2015\n",
            "Epoch 10 Batch 600 Loss 0.2873\n",
            "Epoch 10 Batch 700 Loss 0.3272\n",
            "Epoch 10 Batch 800 Loss 0.2905\n",
            "Epoch 10 Batch 900 Loss 0.2373\n",
            "Epoch 10 Batch 1000 Loss 0.2774\n",
            "Epoch 10 Batch 1100 Loss 0.1716\n",
            "Epoch 10 Batch 1200 Loss 0.3384\n",
            "Epoch 10 Batch 1300 Loss 0.1918\n",
            "Epoch 10 Batch 1400 Loss 0.2734\n",
            "Epoch 10 Batch 1500 Loss 0.2341\n",
            "Epoch 10 Batch 1600 Loss 0.2635\n",
            "Epoch 10 Batch 1700 Loss 0.3433\n",
            "Epoch 10 Batch 1800 Loss 0.2566\n",
            "Epoch 10 Batch 1900 Loss 0.2901\n",
            "Epoch 10 Loss 0.2108\n",
            "Time taken for 1 epoch 60.7624032497406 sec\n",
            "\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  start = time.time()\n",
        "\n",
        "  enc_hidden = encoder.initialize_hidden_state()\n",
        "  total_loss = 0\n",
        "\n",
        "  for (batch, (inp, targ)) in enumerate(train_dataset.take(steps_per_epoch)):\n",
        "    batch_loss = train_step(inp, targ, enc_hidden)\n",
        "    total_loss += batch_loss\n",
        "\n",
        "    if batch % 100 == 0:\n",
        "      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                                   batch,\n",
        "                                                   batch_loss.numpy()))\n",
        "  # saving (checkpoint) the model every 2 epochs\n",
        "  if (epoch + 1) % 2 == 0:\n",
        "    checkpoint.save(file_prefix = checkpoint_prefix)\n",
        "\n",
        "  print('Epoch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                      total_loss / steps_per_epoch))\n",
        "  print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mU3Ce8M6I3rz"
      },
      "source": [
        "## Translate\n",
        "\n",
        "* The evaluate function is similar to the training loop, except we don't use *teacher forcing* here. The input to the decoder at each time step is its previous predictions along with the hidden state and the encoder output.\n",
        "* Stop predicting when the model predicts the *end token*.\n",
        "* And store the *attention weights for every time step*.\n",
        "\n",
        "Note: The encoder output is calculated only once for one input."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EbQpyYs13jF_"
      },
      "outputs": [],
      "source": [
        "def evaluate(sentence):\n",
        "\n",
        "  sentence = dataset_creator.preprocess_sentence(sentence)\n",
        "\n",
        "  inputs = []\n",
        "  for i in sentence.split(\" \"):\n",
        "    try:\n",
        "      idx = inp_lang.word_index[i]\n",
        "      inputs.append(idx)\n",
        "    except:\n",
        "      # 1 for oov token to handle key erro\n",
        "      inputs.append(1)\n",
        "  inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],\n",
        "                                                         maxlen=max_length_input,\n",
        "                                                         padding='post')\n",
        "  inputs = tf.convert_to_tensor(inputs)\n",
        "\n",
        "  result = ''\n",
        "\n",
        "  hidden = [tf.zeros((1, units))]\n",
        "  enc_out, enc_hidden = encoder(inputs, hidden)\n",
        "\n",
        "  dec_hidden = enc_hidden\n",
        "  dec_input = tf.expand_dims([targ_lang.word_index['<start>']], 0)\n",
        "\n",
        "  for t in range(max_length_output):\n",
        "    predictions, dec_hidden, attention_weights = decoder(dec_input,\n",
        "                                                         dec_hidden,\n",
        "                                                         enc_out)\n",
        "\n",
        "    # storing the attention weights to plot later on\n",
        "\n",
        "    predicted_id = tf.argmax(predictions[0]).numpy()\n",
        "\n",
        "    result += targ_lang.index_word[predicted_id] + ' '\n",
        "\n",
        "    if targ_lang.index_word[predicted_id] == '<end>':\n",
        "      return result, sentence\n",
        "\n",
        "    # the predicted ID is fed back into the model\n",
        "    dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "  return result, sentence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sl9zUHzg3jGI"
      },
      "outputs": [],
      "source": [
        "def translate(sentence):\n",
        "  result, sentence = evaluate(sentence)\n",
        "\n",
        "  print('Input: %s' % (sentence))\n",
        "  print('Predicted translation: {}'.format(result))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n250XbnjOaqP"
      },
      "source": [
        "## Restore the latest checkpoint and test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UJpT9D5_OgP6",
        "outputId": "e47a41f8-01f4-4d29-e98d-45bc86997247"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tensorflow.python.checkpoint.checkpoint.CheckpointLoadStatus at 0x7b3ba031a800>"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# restoring the latest checkpoint in checkpoint_dir\n",
        "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WrAM0FDomq3E",
        "outputId": "f70c86f4-5154-4f0b-fd11-123c4bc8091d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> hace mucho frio aqui . <end>\n",
            "Predicted translation: it s very cold here . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'hace mucho frio aqui.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zSx2iM36EZQZ",
        "outputId": "a684da67-b2b5-4008-9551-cfa5c3d02c26"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> esta es mi vida . <end>\n",
            "Predicted translation: this is my life . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'esta es mi vida.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A3LLCx3ZE0Ls",
        "outputId": "a87a1c17-1ca1-4c28-d0e7-80a8563db563"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> ¿ todavia estan en casa ? <end>\n",
            "Predicted translation: are you still at home ? <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'¿todavia estan en casa?')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DUQVLVqUE1YW",
        "outputId": "3188911f-431f-4616-de18-fcf0ea941269"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> trata de averiguarlo . <end>\n",
            "Predicted translation: try to figure it out . <end> \n"
          ]
        }
      ],
      "source": [
        "# wrong translation\n",
        "translate(u'trata de averiguarlo.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IFTOA_Y7Dfcp",
        "outputId": "7f689582-f037-451f-98a7-2478d783f206"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> la comida se desperdicia <end>\n",
            "Predicted translation: the food . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'la comida se desperdicia')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "92aXUSuvwzOt"
      },
      "outputs": [],
      "source": [
        "def translate_batch(test_dataset):\n",
        "  with open('output_text.txt', 'w') as f:\n",
        "    for (inputs, targets) in test_dataset:\n",
        "      outputs = np.zeros((BATCH_SIZE, max_length_output),dtype=np.int16)\n",
        "      hidden_state = tf.zeros((BATCH_SIZE, units))\n",
        "      enc_output, dec_h = encoder(inputs, hidden_state)\n",
        "      dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "      for t in range(max_length_output):\n",
        "        preds, dec_h, _ = decoder(dec_input, dec_h, enc_output)\n",
        "        predicted_id = tf.argmax(preds, axis=1).numpy()\n",
        "        outputs[:, t] = predicted_id\n",
        "        dec_input = tf.expand_dims(predicted_id, 1)\n",
        "      outputs = targ_lang.sequences_to_texts(outputs)\n",
        "      for t, item in enumerate(outputs):\n",
        "        try:\n",
        "          i = item.index('<end>')\n",
        "          f.write(\"%s\\n\" %item[:i])\n",
        "        except:\n",
        "          f.write(\"%s \\n\" % item) # For those translated sequences which didn't correctly translated and have <end> token.\n",
        "\n",
        "outputs = translate_batch(val_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rbHSSD1dT1D0"
      },
      "outputs": [],
      "source": [
        "!head output_text.txt\n",
        "! wc -l output_text.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t2Ci8zsFbL8q"
      },
      "outputs": [],
      "source": [
        "val_targets = list(val_dataset.take(1))\n",
        "val_targets = np.asarray(val_targets[0][1])\n",
        "print(type(val_targets))\n",
        "targ_lang.sequences_to_texts(val_targets)[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kk7sbSx9c1a9"
      },
      "source": [
        "We can see that the model worked well. Despite being not very accurate the translations, however do make some sense."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "okmWWHmMc8T6"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}