global_enc_dec_with_(concat_Alignment_model)LuongAttention.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mejJonsu2o7r"
      },
      "source": [
        "#### Taken from [tensorflow-tutorial ](https://www.tensorflow.org/tutorials/text/nmt_with_attention ) and added a *translate_batch()* function to translate a batch and dump outputs into a file"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# Neural machine translation with attention"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RNxZQoi6_gyF",
        "outputId": "4d0d98c5-6b1b-4897-8a38-4a911da21a6d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "tnxXKDjq3jEL"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import unicodedata\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import io\n",
        "import time\n",
        "from spanish_english_datapreprocessing import NMTDataset\n",
        "from tensorflow.python.ops import math_ops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YsrDDMlKA2nd",
        "outputId": "13a97c49-947f-485d-f200-3099eb2da7ec"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ls: cannot access 'utils': No such file or directory\n"
          ]
        }
      ],
      "source": [
        "!ls utils\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GOi42V79Ydlr"
      },
      "source": [
        "We'll use the same dataset we worked on notebook-1 (text-processing). For our convenience we've created a utils/dataset.py file which returns train and validation tf.data.Dataset objects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sy9mcFmxBP4J",
        "outputId": "35aacdb3-e750-425f-d831-0aeec39a6fe1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\n",
            "2638744/2638744 [==============================] - 0s 0us/step\n"
          ]
        }
      ],
      "source": [
        "BUFFER_SIZE = 90000\n",
        "BATCH_SIZE = 32\n",
        "num_examples = 80000\n",
        "\n",
        "dataset_creator = NMTDataset('en-spa')\n",
        "train_dataset, val_dataset, inp_lang, targ_lang = dataset_creator.call(num_examples, BUFFER_SIZE, BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wVtSz89FBk0D",
        "outputId": "8ad982e4-c71f-43b2-a19b-464eb19497ea"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Inpute Vocabulary Size: 17592\n",
            "Target Vocabulary Size: 9219\n"
          ]
        }
      ],
      "source": [
        "\n",
        "print(\"Inpute Vocabulary Size: {}\".format(len(inp_lang.word_index)))\n",
        "print(\"Target Vocabulary Size: {}\".format(len(targ_lang.word_index)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qc6-NK1GtWQt",
        "outputId": "643dd385-eb63-4f75-dc93-a6876ce285b8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(TensorShape([32, 20]), TensorShape([32, 14]))"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "example_input_batch, example_target_batch = next(iter(train_dataset))\n",
        "example_input_batch.shape, example_target_batch.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNfHIF71ulLu"
      },
      "source": [
        "## Write the encoder and decoder model\n",
        "\n",
        "Implement an encoder-decoder model with attention which you can read about in the TensorFlow [Neural Machine Translation (seq2seq) tutorial](https://github.com/tensorflow/nmt). This example uses a more recent set of APIs. This notebook implements the [attention equations](https://github.com/tensorflow/nmt#background-on-the-attention-mechanism) from the seq2seq tutorial. The following diagram shows that each input words is assigned a weight by the attention mechanism which is then used by the decoder to predict the next word in the sentence. The below picture and formulas are an example of attention mechanism from [Luong's paper](https://arxiv.org/abs/1508.04025v5).\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_mechanism.jpg\" width=\"500\" alt=\"attention mechanism\">\n",
        "\n",
        "The input is put through an encoder model which gives us the encoder output of shape *(batch_size, max_length, hidden_size)* and the encoder hidden state of shape *(batch_size, hidden_size)*.\n",
        "\n",
        "Here are the equations that are implemented:\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_0.jpg\" alt=\"attention equation 0\" width=\"800\">\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_1.jpg\" alt=\"attention equation 1\" width=\"800\">\n",
        "\n",
        "This tutorial uses [Luong attention](https://arxiv.org/abs/1508.04025) for the encoder. Let's decide on notation before writing the simplified form:\n",
        "\n",
        "* FC = Fully connected (dense) layer\n",
        "* EO = Encoder output\n",
        "* H = hidden state\n",
        "* X = input to the decoder\n",
        "\n",
        "And the pseudo-code:\n",
        "\n",
        "* `score = MatMul(Transpose(H)xE0)`\n",
        "* `attention weights = softmax(score, axis = 1)`. Softmax by default is applied on the last axis but here we want to apply it on the *1st axis*, since the shape of score is *(batch_size, max_length, hidden_size)*. `Max_length` is the length of our input. Since we are trying to assign a weight to each input, softmax should be applied on that axis.\n",
        "* `context vector = sum(attention weights * EO, axis = 1)`. Same reason as above for choosing axis as 1.\n",
        "* `embedding output` = The input to the decoder X is passed through an embedding layer.\n",
        "* `merged vector = concat(embedding output, context vector)`\n",
        "* This merged vector is then given to the GRU\n",
        "\n",
        "The shapes of all the vectors at each step have been specified in the comments in the code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "dgVsBjbVDBwX"
      },
      "outputs": [],
      "source": [
        "# Define some useful parameters for further use\n",
        "\n",
        "vocab_inp_size = len(inp_lang.word_index)+1\n",
        "vocab_tar_size = len(targ_lang.word_index)+1\n",
        "max_length_input = example_input_batch.shape[1]\n",
        "max_length_output = example_target_batch.shape[1]\n",
        "\n",
        "embedding_dim = 250\n",
        "units = 512\n",
        "steps_per_epoch = num_examples//BATCH_SIZE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "nZ2rI24i3jFg"
      },
      "outputs": [],
      "source": [
        "# Encoder is composed of embedding layer and then one GRU layer. It produces outputs and last hidden states.\n",
        "# Encoder Outputs shape = (BATCH_SIZE, max_length_input, units)\n",
        "# Last Hidden State Shape = (BATCH_SIZE, units)\n",
        "\n",
        "class Encoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):\n",
        "    super(Encoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.enc_units = enc_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.enc_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "\n",
        "  def call(self, x, hidden):\n",
        "    x = self.embedding(x)\n",
        "    output, state = self.gru(x, initial_state = hidden)\n",
        "    return output, state\n",
        "\n",
        "  def initialize_hidden_state(self):\n",
        "    return tf.zeros((self.batch_sz, self.enc_units))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "60gSVh05Jl6l",
        "outputId": "b765f7c6-0d20-417e-edea-ffdfa2a2cae6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Encoder output shape: (batch size, sequence length, units) (32, 20, 512)\n",
            "Encoder Hidden state shape: (batch size, units) (32, 512)\n"
          ]
        }
      ],
      "source": [
        "encoder = Encoder(vocab_inp_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "# sample input\n",
        "sample_hidden = encoder.initialize_hidden_state()\n",
        "sample_output, sample_hidden = encoder(example_input_batch, sample_hidden)\n",
        "print ('Encoder output shape: (batch size, sequence length, units) {}'.format(sample_output.shape))\n",
        "print ('Encoder Hidden state shape: (batch size, units) {}'.format(sample_hidden.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "umohpBN2OM94"
      },
      "outputs": [],
      "source": [
        "class LuongAttention(tf.keras.layers.Layer):\n",
        "  def __init__(self, units):\n",
        "    super(LuongAttention, self).__init__()\n",
        "    # To recall, score = V*tanh(W1(encoder_outputs) + W2(Prev Step's Hidden State))\n",
        "    self.W1 = tf.keras.layers.Dense(units)\n",
        "    self.W2 = tf.keras.layers.Dense(units)\n",
        "    self.V = tf.keras.layers.Dense(1)\n",
        "\n",
        "  def call(self, query, values):\n",
        "    # query hidden state shape == (batch_size, hidden size) mine (m,n_s)\n",
        "    # query_with_time_axis shape == (batch_size, 1, hidden size) mine (m,1,n_s)\n",
        "    # values shape == (batch_size, max_len, hidden size) mine (m,maxLen,n_a)\n",
        "    # we are doing this to broadcast addition along the time axis to calculate the score\n",
        "    query_with_time_axis = tf.expand_dims(query, 1) ## (m,1,n_s)\n",
        "\n",
        "    # score shape == (batch_size, max_length, 1) mine (m,maxLen,1)\n",
        "    # we get 1 at the last axis because we are applying score to self.V\n",
        "    # the shape of the tensor before applying self.V is (batch_size, max_length, units)\n",
        "\n",
        "    #-------- COMPUTING EQUATION (4) (Concat Alignment Model)  ---------#\n",
        "    score = self.V(tf.nn.tanh(\n",
        "        self.W1(query_with_time_axis) + self.W2(values))) ## shape is (m,maxLen,1)\n",
        "\n",
        "    #-------- COMPUTING EQUATION (1) ------------#\n",
        "    # attention_weights shape == (batch_size, max_length, 1)\n",
        "    attention_weights = tf.nn.softmax(score, axis=1) ## shape is (m,maxlen,1)\n",
        "\n",
        "    #---------- COMPUTING EQUATION (2) -----------#\n",
        "    # context_vector shape after sum == (batch_size, hidden_size)\n",
        "    context_vector = attention_weights * values ## shape is (m,maxLen,n_a)\n",
        "    # Context vector is passed on to the curren time step's decoder cell\n",
        "    context_vector = tf.reduce_sum(context_vector, axis=1) # (m,n_a)\n",
        "\n",
        "    return context_vector, attention_weights\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k534zTHiDjQU",
        "outputId": "613c6795-e4f3-48c2-86e2-b137c01f4d3c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Attention result shape: (batch size, units) (32, 512)\n",
            "Attention weights shape: (batch_size, sequence_length, 1) (32, 20, 1)\n"
          ]
        }
      ],
      "source": [
        "attention_layer = LuongAttention(50)\n",
        "attention_result, attention_weights = attention_layer(sample_hidden, sample_output)\n",
        "\n",
        "print(\"Attention result shape: (batch size, units) {}\".format(attention_result.shape))\n",
        "print(\"Attention weights shape: (batch_size, sequence_length, 1) {}\".format(attention_weights.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOaVVOaRvkVx"
      },
      "source": [
        "Decoder based on Luong Attention architecture\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "yJ_B3mhW3jFk"
      },
      "outputs": [],
      "source": [
        "class Decoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):\n",
        "    super(Decoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.dec_units = dec_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.dec_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "  def call(self, x, hidden, enc_output):\n",
        "\n",
        "    x = self.embedding(x) # (m,1,emb_dim)\n",
        "\n",
        "    output,state = self.gru(x) # output (m,1,n_s) state (m,n_s)\n",
        "    context_vector, attention_weights = attention_layer(state, enc_output)  # context_vector (m,n_a) attention_weigths (m,maxlen,1)\n",
        "    output = tf.concat([tf.expand_dims(context_vector, 1), output], axis=-1) # output  (m,1,n_a+n_s)\n",
        "\n",
        "    output = tf.reshape(output, (-1, output.shape[2])) # output (m,n_a+n_s)\n",
        "\n",
        "    # output shape == (batch_size, vocab)\n",
        "    x = self.fc(output) #\n",
        "\n",
        "    return x, state, attention_weights\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P5UY8wko3jFp",
        "outputId": "1f13d561-f613-4c30-f494-d057ff849228"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Decoder output shape: (batch_size, vocab size) (32, 9220)\n"
          ]
        }
      ],
      "source": [
        "decoder = Decoder(vocab_tar_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "sample_decoder_output, _, _ = decoder(tf.random.uniform((BATCH_SIZE, 1)),\n",
        "                                      sample_hidden, sample_output)\n",
        "\n",
        "print ('Decoder output shape: (batch_size, vocab size) {}'.format(sample_decoder_output.shape))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ch_71VbIRfK"
      },
      "source": [
        "## Define the optimizer and the loss function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "WmTHr5iV3jFr"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMVWzzsfNl4e"
      },
      "source": [
        "## Checkpoints (Object-based saving)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "Zj8bXQTgNwrF"
      },
      "outputs": [],
      "source": [
        "checkpoint_dir = './training_checkpoints'\n",
        "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
        "checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
        "                                 encoder=encoder,\n",
        "                                 decoder=decoder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpObfY22IddU"
      },
      "source": [
        "## Training\n",
        "\n",
        "1. Pass the *input* through the *encoder* which return *encoder output* and the *encoder hidden state*.\n",
        "2. The encoder output, encoder hidden state and the decoder input (which is the *start token*) is passed to the decoder.\n",
        "3. The decoder returns the *predictions* and the *decoder hidden state*.\n",
        "4. The decoder hidden state is then passed back into the model and the predictions are used to calculate the loss.\n",
        "5. Use *teacher forcing* to decide the next input to the decoder.\n",
        "6. *Teacher forcing* is the technique where the *target word* is passed as the *next input* to the decoder.\n",
        "7. The final step is to calculate the gradients and apply it to the optimizer and backpropagate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "sC9ArXSsVfqn"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_step(inp, targ, enc_hidden):\n",
        "  loss = 0\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "    enc_output, enc_hidden = encoder(inp, enc_hidden)\n",
        "\n",
        "    dec_hidden = enc_hidden\n",
        "\n",
        "    dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "\n",
        "    # Teacher forcing - feeding the target as the next input\n",
        "    for t in range(1, targ.shape[1]):\n",
        "      # passing enc_output to the decoder\n",
        "      predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)\n",
        "\n",
        "      loss += loss_function(targ[:, t], predictions)\n",
        "\n",
        "      # using teacher forcing\n",
        "      dec_input = tf.expand_dims(targ[:, t], 1)\n",
        "\n",
        "  batch_loss = (loss / int(targ.shape[1]))\n",
        "\n",
        "  variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, variables))\n",
        "\n",
        "  return batch_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ddefjBMa3jF0",
        "outputId": "e37fa60a-2758-4528-9ed6-c95a81e19f1e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1 Batch 0 Loss 4.6460\n",
            "Epoch 1 Batch 100 Loss 2.2242\n",
            "Epoch 1 Batch 200 Loss 2.1964\n",
            "Epoch 1 Batch 300 Loss 1.9355\n",
            "Epoch 1 Batch 400 Loss 1.9138\n",
            "Epoch 1 Batch 500 Loss 1.7387\n",
            "Epoch 1 Batch 600 Loss 1.7757\n",
            "Epoch 1 Batch 700 Loss 1.7815\n",
            "Epoch 1 Batch 800 Loss 1.6633\n",
            "Epoch 1 Batch 900 Loss 1.6168\n",
            "Epoch 1 Batch 1000 Loss 1.5075\n",
            "Epoch 1 Batch 1100 Loss 1.7428\n",
            "Epoch 1 Batch 1200 Loss 1.3683\n",
            "Epoch 1 Batch 1300 Loss 1.2969\n",
            "Epoch 1 Batch 1400 Loss 1.3621\n",
            "Epoch 1 Batch 1500 Loss 1.1371\n",
            "Epoch 1 Batch 1600 Loss 1.2556\n",
            "Epoch 1 Batch 1700 Loss 1.1019\n",
            "Epoch 1 Batch 1800 Loss 1.1820\n",
            "Epoch 1 Batch 1900 Loss 1.2625\n",
            "Epoch 1 Loss 1.3106\n",
            "Time taken for 1 epoch 94.94587087631226 sec\n",
            "\n",
            "Epoch 2 Batch 0 Loss 0.8918\n",
            "Epoch 2 Batch 100 Loss 0.9450\n",
            "Epoch 2 Batch 200 Loss 0.8632\n",
            "Epoch 2 Batch 300 Loss 0.7740\n",
            "Epoch 2 Batch 400 Loss 0.9422\n",
            "Epoch 2 Batch 500 Loss 0.8885\n",
            "Epoch 2 Batch 600 Loss 0.8817\n",
            "Epoch 2 Batch 700 Loss 0.9044\n",
            "Epoch 2 Batch 800 Loss 0.6131\n",
            "Epoch 2 Batch 900 Loss 0.8474\n",
            "Epoch 2 Batch 1000 Loss 0.8338\n",
            "Epoch 2 Batch 1100 Loss 0.9354\n",
            "Epoch 2 Batch 1200 Loss 0.7835\n",
            "Epoch 2 Batch 1300 Loss 0.6980\n",
            "Epoch 2 Batch 1400 Loss 0.7218\n",
            "Epoch 2 Batch 1500 Loss 0.7660\n",
            "Epoch 2 Batch 1600 Loss 0.6672\n",
            "Epoch 2 Batch 1700 Loss 0.7120\n",
            "Epoch 2 Batch 1800 Loss 0.9305\n",
            "Epoch 2 Batch 1900 Loss 0.8909\n",
            "Epoch 2 Loss 0.6782\n",
            "Time taken for 1 epoch 64.61592674255371 sec\n",
            "\n",
            "Epoch 3 Batch 0 Loss 0.5383\n",
            "Epoch 3 Batch 100 Loss 0.5491\n",
            "Epoch 3 Batch 200 Loss 0.3892\n",
            "Epoch 3 Batch 300 Loss 0.5870\n",
            "Epoch 3 Batch 400 Loss 0.4495\n",
            "Epoch 3 Batch 500 Loss 0.5990\n",
            "Epoch 3 Batch 600 Loss 0.6237\n",
            "Epoch 3 Batch 700 Loss 0.6487\n",
            "Epoch 3 Batch 800 Loss 0.4733\n",
            "Epoch 3 Batch 900 Loss 0.5698\n",
            "Epoch 3 Batch 1000 Loss 0.5039\n",
            "Epoch 3 Batch 1100 Loss 0.5990\n",
            "Epoch 3 Batch 1200 Loss 0.6959\n",
            "Epoch 3 Batch 1300 Loss 0.6188\n",
            "Epoch 3 Batch 1400 Loss 0.5092\n",
            "Epoch 3 Batch 1500 Loss 0.6627\n",
            "Epoch 3 Batch 1600 Loss 0.4969\n",
            "Epoch 3 Batch 1700 Loss 0.6009\n",
            "Epoch 3 Batch 1800 Loss 0.5714\n",
            "Epoch 3 Batch 1900 Loss 0.5532\n",
            "Epoch 3 Loss 0.4314\n",
            "Time taken for 1 epoch 63.66870594024658 sec\n",
            "\n",
            "Epoch 4 Batch 0 Loss 0.2883\n",
            "Epoch 4 Batch 100 Loss 0.4688\n",
            "Epoch 4 Batch 200 Loss 0.3718\n",
            "Epoch 4 Batch 300 Loss 0.3282\n",
            "Epoch 4 Batch 400 Loss 0.3072\n",
            "Epoch 4 Batch 500 Loss 0.4941\n",
            "Epoch 4 Batch 600 Loss 0.3370\n",
            "Epoch 4 Batch 700 Loss 0.4671\n",
            "Epoch 4 Batch 800 Loss 0.3489\n",
            "Epoch 4 Batch 900 Loss 0.4142\n",
            "Epoch 4 Batch 1000 Loss 0.4354\n",
            "Epoch 4 Batch 1100 Loss 0.4418\n",
            "Epoch 4 Batch 1200 Loss 0.2957\n",
            "Epoch 4 Batch 1300 Loss 0.3280\n",
            "Epoch 4 Batch 1400 Loss 0.4066\n",
            "Epoch 4 Batch 1500 Loss 0.4293\n",
            "Epoch 4 Batch 1600 Loss 0.3978\n",
            "Epoch 4 Batch 1700 Loss 0.5098\n",
            "Epoch 4 Batch 1800 Loss 0.4489\n",
            "Epoch 4 Batch 1900 Loss 0.4905\n",
            "Epoch 4 Loss 0.3057\n",
            "Time taken for 1 epoch 64.55504012107849 sec\n",
            "\n",
            "Epoch 5 Batch 0 Loss 0.3440\n",
            "Epoch 5 Batch 100 Loss 0.3141\n",
            "Epoch 5 Batch 200 Loss 0.2509\n",
            "Epoch 5 Batch 300 Loss 0.1862\n",
            "Epoch 5 Batch 400 Loss 0.2826\n",
            "Epoch 5 Batch 500 Loss 0.2984\n",
            "Epoch 5 Batch 600 Loss 0.3005\n",
            "Epoch 5 Batch 700 Loss 0.3107\n",
            "Epoch 5 Batch 800 Loss 0.2420\n",
            "Epoch 5 Batch 900 Loss 0.2956\n",
            "Epoch 5 Batch 1000 Loss 0.2141\n",
            "Epoch 5 Batch 1100 Loss 0.3091\n",
            "Epoch 5 Batch 1200 Loss 0.1987\n",
            "Epoch 5 Batch 1300 Loss 0.3192\n",
            "Epoch 5 Batch 1400 Loss 0.3137\n",
            "Epoch 5 Batch 1500 Loss 0.2621\n",
            "Epoch 5 Batch 1600 Loss 0.3388\n",
            "Epoch 5 Batch 1700 Loss 0.4140\n",
            "Epoch 5 Batch 1800 Loss 0.2945\n",
            "Epoch 5 Batch 1900 Loss 0.2863\n",
            "Epoch 5 Loss 0.2366\n",
            "Time taken for 1 epoch 63.28073525428772 sec\n",
            "\n",
            "Epoch 6 Batch 0 Loss 0.2644\n",
            "Epoch 6 Batch 100 Loss 0.1852\n",
            "Epoch 6 Batch 200 Loss 0.2642\n",
            "Epoch 6 Batch 300 Loss 0.1726\n",
            "Epoch 6 Batch 400 Loss 0.1788\n",
            "Epoch 6 Batch 500 Loss 0.2014\n",
            "Epoch 6 Batch 600 Loss 0.2325\n",
            "Epoch 6 Batch 700 Loss 0.1825\n",
            "Epoch 6 Batch 800 Loss 0.2871\n",
            "Epoch 6 Batch 900 Loss 0.3575\n",
            "Epoch 6 Batch 1000 Loss 0.2237\n",
            "Epoch 6 Batch 1100 Loss 0.3716\n",
            "Epoch 6 Batch 1200 Loss 0.2261\n",
            "Epoch 6 Batch 1300 Loss 0.2056\n",
            "Epoch 6 Batch 1400 Loss 0.1902\n",
            "Epoch 6 Batch 1500 Loss 0.3022\n",
            "Epoch 6 Batch 1600 Loss 0.2818\n",
            "Epoch 6 Batch 1700 Loss 0.2112\n",
            "Epoch 6 Batch 1800 Loss 0.2908\n",
            "Epoch 6 Batch 1900 Loss 0.3148\n",
            "Epoch 6 Loss 0.1958\n",
            "Time taken for 1 epoch 64.01101851463318 sec\n",
            "\n",
            "Epoch 7 Batch 0 Loss 0.1995\n",
            "Epoch 7 Batch 100 Loss 0.1415\n",
            "Epoch 7 Batch 200 Loss 0.2204\n",
            "Epoch 7 Batch 300 Loss 0.1786\n",
            "Epoch 7 Batch 400 Loss 0.1937\n",
            "Epoch 7 Batch 500 Loss 0.2058\n",
            "Epoch 7 Batch 600 Loss 0.2037\n",
            "Epoch 7 Batch 700 Loss 0.1578\n",
            "Epoch 7 Batch 800 Loss 0.1930\n",
            "Epoch 7 Batch 900 Loss 0.1540\n",
            "Epoch 7 Batch 1000 Loss 0.1975\n",
            "Epoch 7 Batch 1100 Loss 0.1870\n",
            "Epoch 7 Batch 1200 Loss 0.2031\n",
            "Epoch 7 Batch 1300 Loss 0.1452\n",
            "Epoch 7 Batch 1400 Loss 0.1806\n",
            "Epoch 7 Batch 1500 Loss 0.2216\n",
            "Epoch 7 Batch 1600 Loss 0.2965\n",
            "Epoch 7 Batch 1700 Loss 0.2708\n",
            "Epoch 7 Batch 1800 Loss 0.1424\n",
            "Epoch 7 Batch 1900 Loss 0.2462\n",
            "Epoch 7 Loss 0.1674\n",
            "Time taken for 1 epoch 64.07409834861755 sec\n",
            "\n",
            "Epoch 8 Batch 0 Loss 0.1535\n",
            "Epoch 8 Batch 100 Loss 0.1607\n",
            "Epoch 8 Batch 200 Loss 0.1293\n",
            "Epoch 8 Batch 300 Loss 0.1763\n",
            "Epoch 8 Batch 400 Loss 0.1288\n",
            "Epoch 8 Batch 500 Loss 0.2136\n",
            "Epoch 8 Batch 600 Loss 0.2267\n",
            "Epoch 8 Batch 700 Loss 0.1301\n",
            "Epoch 8 Batch 800 Loss 0.2536\n",
            "Epoch 8 Batch 900 Loss 0.1404\n",
            "Epoch 8 Batch 1000 Loss 0.2038\n",
            "Epoch 8 Batch 1100 Loss 0.2278\n",
            "Epoch 8 Batch 1200 Loss 0.1856\n",
            "Epoch 8 Batch 1300 Loss 0.1573\n",
            "Epoch 8 Batch 1400 Loss 0.2030\n",
            "Epoch 8 Batch 1500 Loss 0.1622\n",
            "Epoch 8 Batch 1600 Loss 0.2229\n",
            "Epoch 8 Batch 1700 Loss 0.1384\n",
            "Epoch 8 Batch 1800 Loss 0.2160\n",
            "Epoch 8 Batch 1900 Loss 0.2156\n",
            "Epoch 8 Loss 0.1488\n",
            "Time taken for 1 epoch 64.23983216285706 sec\n",
            "\n",
            "Epoch 9 Batch 0 Loss 0.1446\n",
            "Epoch 9 Batch 100 Loss 0.1774\n",
            "Epoch 9 Batch 200 Loss 0.1184\n",
            "Epoch 9 Batch 300 Loss 0.1834\n",
            "Epoch 9 Batch 400 Loss 0.2029\n",
            "Epoch 9 Batch 500 Loss 0.1733\n",
            "Epoch 9 Batch 600 Loss 0.1982\n",
            "Epoch 9 Batch 700 Loss 0.1841\n",
            "Epoch 9 Batch 800 Loss 0.1895\n",
            "Epoch 9 Batch 900 Loss 0.2007\n",
            "Epoch 9 Batch 1000 Loss 0.2049\n",
            "Epoch 9 Batch 1100 Loss 0.2025\n",
            "Epoch 9 Batch 1200 Loss 0.2051\n",
            "Epoch 9 Batch 1300 Loss 0.1160\n",
            "Epoch 9 Batch 1400 Loss 0.1330\n",
            "Epoch 9 Batch 1500 Loss 0.1530\n",
            "Epoch 9 Batch 1600 Loss 0.2547\n",
            "Epoch 9 Batch 1700 Loss 0.2929\n",
            "Epoch 9 Batch 1800 Loss 0.1626\n",
            "Epoch 9 Batch 1900 Loss 0.2990\n",
            "Epoch 9 Loss 0.1330\n",
            "Time taken for 1 epoch 63.76797294616699 sec\n",
            "\n",
            "Epoch 10 Batch 0 Loss 0.0917\n",
            "Epoch 10 Batch 100 Loss 0.1233\n",
            "Epoch 10 Batch 200 Loss 0.1040\n",
            "Epoch 10 Batch 300 Loss 0.0897\n",
            "Epoch 10 Batch 400 Loss 0.1319\n",
            "Epoch 10 Batch 500 Loss 0.0785\n",
            "Epoch 10 Batch 600 Loss 0.1350\n",
            "Epoch 10 Batch 700 Loss 0.1511\n",
            "Epoch 10 Batch 800 Loss 0.1152\n",
            "Epoch 10 Batch 900 Loss 0.1644\n",
            "Epoch 10 Batch 1000 Loss 0.1234\n",
            "Epoch 10 Batch 1100 Loss 0.1718\n",
            "Epoch 10 Batch 1200 Loss 0.1545\n",
            "Epoch 10 Batch 1300 Loss 0.1038\n",
            "Epoch 10 Batch 1400 Loss 0.1509\n",
            "Epoch 10 Batch 1500 Loss 0.1847\n",
            "Epoch 10 Batch 1600 Loss 0.1306\n",
            "Epoch 10 Batch 1700 Loss 0.1478\n",
            "Epoch 10 Batch 1800 Loss 0.2447\n",
            "Epoch 10 Batch 1900 Loss 0.1724\n",
            "Epoch 10 Loss 0.1218\n",
            "Time taken for 1 epoch 64.6534469127655 sec\n",
            "\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  start = time.time()\n",
        "\n",
        "  enc_hidden = encoder.initialize_hidden_state()\n",
        "  total_loss = 0\n",
        "\n",
        "  for (batch, (inp, targ)) in enumerate(train_dataset.take(steps_per_epoch)):\n",
        "    batch_loss = train_step(inp, targ, enc_hidden)\n",
        "    total_loss += batch_loss\n",
        "\n",
        "    if batch % 100 == 0:\n",
        "      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                                   batch,\n",
        "                                                   batch_loss.numpy()))\n",
        "  # saving (checkpoint) the model every 2 epochs\n",
        "  if (epoch + 1) % 2 == 0:\n",
        "    checkpoint.save(file_prefix = checkpoint_prefix)\n",
        "\n",
        "  print('Epoch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                      total_loss / steps_per_epoch))\n",
        "  print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mU3Ce8M6I3rz"
      },
      "source": [
        "## Translate\n",
        "\n",
        "* The evaluate function is similar to the training loop, except we don't use *teacher forcing* here. The input to the decoder at each time step is its previous predictions along with the hidden state and the encoder output.\n",
        "* Stop predicting when the model predicts the *end token*.\n",
        "* And store the *attention weights for every time step*.\n",
        "\n",
        "Note: The encoder output is calculated only once for one input."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "EbQpyYs13jF_"
      },
      "outputs": [],
      "source": [
        "def evaluate(sentence):\n",
        "\n",
        "  sentence = dataset_creator.preprocess_sentence(sentence)\n",
        "\n",
        "  inputs = []\n",
        "  for i in sentence.split(\" \"):\n",
        "    try:\n",
        "      idx = inp_lang.word_index[i]\n",
        "      inputs.append(idx)\n",
        "    except:\n",
        "      # 1 for oov token to handle key erro\n",
        "      inputs.append(1)\n",
        "  inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],\n",
        "                                                         maxlen=max_length_input,\n",
        "                                                         padding='post')\n",
        "  inputs = tf.convert_to_tensor(inputs)\n",
        "\n",
        "  result = ''\n",
        "\n",
        "  hidden = [tf.zeros((1, units))]\n",
        "  enc_out, enc_hidden = encoder(inputs, hidden)\n",
        "\n",
        "  dec_hidden = enc_hidden\n",
        "  dec_input = tf.expand_dims([targ_lang.word_index['<start>']], 0)\n",
        "\n",
        "  for t in range(max_length_output):\n",
        "    predictions, dec_hidden, attention_weights = decoder(dec_input,\n",
        "                                                         dec_hidden,\n",
        "                                                         enc_out)\n",
        "\n",
        "    # storing the attention weights to plot later on\n",
        "\n",
        "    predicted_id = tf.argmax(predictions[0]).numpy()\n",
        "\n",
        "    result += targ_lang.index_word[predicted_id] + ' '\n",
        "\n",
        "    if targ_lang.index_word[predicted_id] == '<end>':\n",
        "      return result, sentence\n",
        "\n",
        "    # the predicted ID is fed back into the model\n",
        "    dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "  return result, sentence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "sl9zUHzg3jGI"
      },
      "outputs": [],
      "source": [
        "def translate(sentence):\n",
        "  result, sentence = evaluate(sentence)\n",
        "\n",
        "  print('Input: %s' % (sentence))\n",
        "  print('Predicted translation: {}'.format(result))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n250XbnjOaqP"
      },
      "source": [
        "## Restore the latest checkpoint and test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UJpT9D5_OgP6",
        "outputId": "f9341dcc-5e64-4945-d0d2-3bf75e96c70d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<tensorflow.python.checkpoint.checkpoint.CheckpointLoadStatus at 0x7d8ea223c9d0>"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# restoring the latest checkpoint in checkpoint_dir\n",
        "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WrAM0FDomq3E",
        "outputId": "0caa5dd0-cb1c-4f7e-f09f-08e87d0f7a65"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> hace mucho frio aqui . <end>\n",
            "Predicted translation: it s very cold here . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'hace mucho frio aqui.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zSx2iM36EZQZ",
        "outputId": "7b77b06a-17d5-40ef-9c35-605ee660a2fe"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> esta es mi vida . <end>\n",
            "Predicted translation: this is my life . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'esta es mi vida.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A3LLCx3ZE0Ls",
        "outputId": "7d38d076-b2dd-4664-811e-6adbb3fbcebf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> ¿ todavia estan en casa ? <end>\n",
            "Predicted translation: are you still home ? <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'¿todavia estan en casa?')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DUQVLVqUE1YW",
        "outputId": "f865decf-933e-4466-c3fb-64924e9421cc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> trata de averiguarlo . <end>\n",
            "Predicted translation: try to figure it out . <end> \n"
          ]
        }
      ],
      "source": [
        "# wrong translation\n",
        "translate(u'trata de averiguarlo.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IFTOA_Y7Dfcp",
        "outputId": "000969ce-cb04-4597-9ec8-99c2ad08e1e7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> la comida se desperdicia <end>\n",
            "Predicted translation: food prices went out . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'la comida se desperdicia')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WRFpjyP6Yss3",
        "outputId": "ea69d484-2dbb-4cb3-8642-7e1a6be7a454"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> mi casa esta ardiendo <end>\n",
            "Predicted translation: my house is burning . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'mi casa esta ardiendo')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LxOhMeAPY66j",
        "outputId": "206b1588-fb5e-4c39-d53a-d796d3184a60"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> ella se esta riendo de mi pobre ciclo <end>\n",
            "Predicted translation: she is laughing at my being poor to my being poor to my being \n"
          ]
        }
      ],
      "source": [
        "translate(u'ella se esta riendo de mi pobre ciclo')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j0glCDSDZMzp",
        "outputId": "1d692e54-b6cf-4974-d69e-e572bcdc1785"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> el profesor regano a la clase por pobre desempeno . <end>\n",
            "Predicted translation: the teacher for the teacher for the teacher for the teacher for the teacher \n"
          ]
        }
      ],
      "source": [
        "translate(u'El profesor regañó a la clase por pobre desempeño.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2GuafLmjZSAp",
        "outputId": "dc17b53d-0b78-492c-d77d-1b62a614a5d7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> esta lloviendo fuera <end>\n",
            "Predicted translation: it s raining outside . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'Está lloviendo fuera')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mZPToGElZfs6",
        "outputId": "7b886fbe-3e64-42e8-8fc8-5a3862e92d2f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Input: <start> estar tranquilo y relajarse <end>\n",
            "Predicted translation: he d be quiet . <end> \n"
          ]
        }
      ],
      "source": [
        "translate(u'Estar tranquilo y relajarse')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "92aXUSuvwzOt"
      },
      "outputs": [],
      "source": [
        "def translate_batch(test_dataset):\n",
        "  with open('output_text.txt', 'w') as f:\n",
        "    for (inputs, targets) in test_dataset:\n",
        "      outputs = np.zeros((BATCH_SIZE, max_length_output),dtype=np.int16)\n",
        "      hidden_state = tf.zeros((BATCH_SIZE, units))\n",
        "      enc_output, dec_h = encoder(inputs, hidden_state)\n",
        "      dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "      for t in range(max_length_output):\n",
        "        preds, dec_h, _ = decoder(dec_input, dec_h, enc_output)\n",
        "        predicted_id = tf.argmax(preds, axis=1).numpy()\n",
        "        outputs[:, t] = predicted_id\n",
        "        dec_input = tf.expand_dims(predicted_id, 1)\n",
        "      outputs = targ_lang.sequences_to_texts(outputs)\n",
        "      for t, item in enumerate(outputs):\n",
        "        try:\n",
        "          i = item.index('<end>')\n",
        "          f.write(\"%s\\n\" %item[:i])\n",
        "        except:\n",
        "          f.write(\"%s \\n\" % item) # For those translated sequences which didn't correctly translated and have <end> token.\n",
        "\n",
        "outputs = translate_batch(val_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rbHSSD1dT1D0"
      },
      "outputs": [],
      "source": [
        "!head output_text.txt\n",
        "! wc -l output_text.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t2Ci8zsFbL8q"
      },
      "outputs": [],
      "source": [
        "val_targets = list(val_dataset.take(1))\n",
        "val_targets = np.asarray(val_targets[0][1])\n",
        "print(type(val_targets))\n",
        "targ_lang.sequences_to_texts(val_targets)[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kk7sbSx9c1a9"
      },
      "source": [
        "We can see that the model worked well. Despite being not very accurate the translations, however do make some sense."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "okmWWHmMc8T6"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}