Easy_Transformer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Easy Transformer.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LastRemote/Transformer-TF2.0/blob/master/Easy_Transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OgNHSyc9-7BA",
        "colab_type": "text"
      },
      "source": [
        "This tutorial demonstrates building and training a English to Spanish [Transformer Model](https://arxiv.org/abs/1706.03762)  from a few parallel lines. Transformer model is currently a state-of-the-art machine translation system that uses self-attention that can learn both short-range and long-range relations and Feedforward Neural Network that enables model parallelism. Comparing to the [Official TensorFlow Tutorial](https://www.tensorflow.org/alpha/tutorials/text/transformer), this model requires less understanding of TensorFlow specific functions and avoids reshaping Tensors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVcFK64e90tW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# We will be using TensorFlow 2.0 for this tutorial!\n",
        "!pip install tf-nightly-2.0-preview"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mE-aLQkh-GR8",
        "colab_type": "code",
        "outputId": "63828382-0e35-403d-8bdf-526a6ffe384f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "import tensorflow as tf\n",
        "print(\"You have version\", tf.__version__)\n",
        "assert tf.__version__ >= \"2.0\" # TensorFlow ≥ 2.0 required"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "You have version 2.0.0-dev20190517\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KJsNN6j5-d3C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import unicodedata, re\n",
        "\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Lambda, Layer, Embedding, LayerNormalization\n",
        "\n",
        "# We should avoid keras.backend at all cases unless using ONNX support\n",
        "#import tensorflow.keras.backend as backend"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JrMMHJ5TR9lM",
        "colab_type": "text"
      },
      "source": [
        "# Data Preprocessing\n",
        "I used the same data preprocessing process as in [this seq2seq example](https://github.com/random-forests/applied-dl/blob/master/examples/8-seq2seq.ipynb).\n",
        "\n",
        "Let's start by using some parallel text examples in English and Spanish:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6TYs8FSTRyvS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sentences = [\n",
        "  (\"Do you want a cup of coffee?\", \"¿Quieres una taza de café?\"),\n",
        "  (\"I've had coffee already.\", \"Ya tomé café.\"),\n",
        "  (\"Can I get you a coffee?\", \"¿Quieres que te traiga un café?\"),\n",
        "  (\"Please give me some coffee.\", \"Dame algo de café por favor.\"),\n",
        "  (\"Would you like me to make coffee?\", \"¿Quieres que prepare café?\"),\n",
        "  (\"Two coffees, please.\", \"Dos cafés, por favor.\"),\n",
        "  (\"How about a cup of coffee?\", \"¿Qué tal una taza de café?\"),\n",
        "  (\"I drank two cups of coffee.\", \"Me tomé dos tazas de café.\"),\n",
        "  (\"Would you like to have a cup of coffee?\", \"¿Te gustaría tomar una taza de café?\"),\n",
        "  (\"There'll be coffee and cake at five.\", \"A las cinco habrá café y un pastel.\"),\n",
        "  (\"Another coffee, please.\", \"Otro café, por favor.\"),\n",
        "  (\"I made coffee.\", \"Hice café.\"),\n",
        "  (\"I would like to have a cup of coffee.\", \"Quiero beber una taza de café.\"),\n",
        "  (\"Do you want me to make coffee?\", \"¿Quieres que haga café?\"),\n",
        "  (\"It is hard to wake up without a strong cup of coffee.\", \"Es difícil despertarse sin una taza de café fuerte.\"),\n",
        "  (\"All I drank was coffee.\", \"Todo lo que bebí fue café.\"),\n",
        "  (\"I've drunk way too much coffee today.\", \"He bebido demasiado café hoy.\"),\n",
        "  (\"Which do you prefer, tea or coffee?\", \"¿Qué prefieres, té o café?\"),\n",
        "  (\"There are many kinds of coffee.\", \"Hay muchas variedades de café.\"),\n",
        "  (\"I will make some coffee.\",\t\"Prepararé algo de café.\")\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "689Pk6Hz6Mos",
        "colab_type": "text"
      },
      "source": [
        "Regularizing source and target and adding \\<start> and \\<end> token:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7xIAQHyhSHts",
        "colab_type": "code",
        "outputId": "c0aedf70-307d-48fb-d934-283c3b379f1e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "def preprocess(s):\n",
        "  # for details, see https://www.tensorflow.org/alpha/tutorials/sequences/nmt_with_attention\n",
        "  s = ''.join(c for c in unicodedata.normalize('NFD', s) if unicodedata.category(c) != 'Mn')\n",
        "  s = re.sub(r\"([?.!¡,¿])\", r\" \\1 \", s) # Add spaces around punctuations\n",
        "  s = re.sub(r'[\" \"]+', \" \", s) # Remove extra space\n",
        "  s = re.sub(r\"[^a-zA-Z?.!¡,¿áéíóú¡üñ]+\", \" \", s) # Remove other characters\n",
        "  s = s.strip()\n",
        "  s = '<start> ' + s + ' <end>'\n",
        "  return s\n",
        "\n",
        "print(\"Original:\", sentences[0])\n",
        "sentences = [(preprocess(en), preprocess(es)) for (en, es) in sentences]\n",
        "print(\"Preprocessed:\", sentences[0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original: ('Do you want a cup of coffee?', '¿Quieres una taza de café?')\n",
            "Preprocessed: ('<start> Do you want a cup of coffee ? <end>', '<start> ¿ Quieres una taza de cafe ? <end>')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-aq0zFg5tAQ",
        "colab_type": "text"
      },
      "source": [
        "We then tokenize both source and target sentences into lists of integers, and pad zeros at the end of each sequence to the same length."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g3bpAfYoSdTu",
        "colab_type": "code",
        "outputId": "6d091ea9-f964-4a9e-d25c-dc99dcc0ad57",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "source_sentences, target_sentences = list(zip(*sentences))\n",
        "\n",
        "# In this illustration, I choose not to specify num_words and oov_token due to the size of data.\n",
        "# for details, please visit https://keras.io/preprocessing/text/\n",
        "source_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='') \n",
        "source_tokenizer.fit_on_texts(source_sentences)\n",
        "source_data = source_tokenizer.texts_to_sequences(source_sentences)\n",
        "print(\"Sequence:\", source_data[0])\n",
        "source_data = tf.keras.preprocessing.sequence.pad_sequences(source_data, padding='post')\n",
        "print(\"Padded:\", source_data[0])\n",
        "\n",
        "target_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')\n",
        "target_tokenizer.fit_on_texts(target_sentences)\n",
        "target_data = target_tokenizer.texts_to_sequences(target_sentences)\n",
        "target_data = tf.keras.preprocessing.sequence.pad_sequences(target_data, padding='post')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sequence: [1, 12, 8, 19, 9, 10, 6, 3, 7, 2]\n",
            "Padded: [ 1 12  8 19  9 10  6  3  7  2  0  0  0  0  0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tnlm3gEtSsb_",
        "colab_type": "code",
        "outputId": "79103aab-c0e0-4756-d46f-86d8351ecbaa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "# Machine translation models take the entire source sentence and an incomplete sentence in\n",
        "# target language as inputs at once, and predict the next word for the incomplete sentence.\n",
        "# We create labels for the decoder by shifting the target sequence one to the right.\n",
        "target_labels = np.zeros(target_data.shape)\n",
        "target_labels[:,0:target_data.shape[1] -1] = target_data[:,1:]\n",
        "\n",
        "print(\"Target sequence\", target_data[0])\n",
        "print(\"Target label\", target_labels[0])\n",
        "\n",
        "source_vocab_len = len(source_tokenizer.word_index) + 1\n",
        "target_vocab_len = len(target_tokenizer.word_index) + 1\n",
        "\n",
        "print(\"Size of source vocabulary: \", source_vocab_len)\n",
        "print(\"Size of target vocabulary: \", target_vocab_len)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Target sequence [ 1  6 11  9 10  5  3  7  2  0  0  0]\n",
            "Target label [ 6. 11.  9. 10.  5.  3.  7.  2.  0.  0.  0.  0.]\n",
            "Size of source vocabulary:  65\n",
            "Size of target vocabulary:  60\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-oKCaIAWOSX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# For Gradient Tape training\n",
        "dataset = tf.data.Dataset.from_tensor_slices((source_data, target_data, target_labels)).batch(5)\n",
        "# For Keras model.fit()\n",
        "dataset_2 = tf.data.Dataset.from_tensor_slices((source_data, target_data, target_labels))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HrfG2a42TB7F",
        "colab_type": "text"
      },
      "source": [
        "# Transformer Structure\n",
        "Then we build the entire structure for transformer. It is actually not hard at all!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ejOHRGKg-1Oh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Transformer parameters\n",
        "d_model = 64 # 512 in the original paper\n",
        "d_k = 16 # 64 in the original paper\n",
        "d_v = 16 # 64 in the original paper\n",
        "n_heads = 4 # 8 in the original paper\n",
        "n_encoder_layers = 2 # 6 in the original paper\n",
        "n_decoder_layers = 2 # 6 in the original paper\n",
        "\n",
        "max_token_length = 20 # 512 in the original paper"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FyUQ0EU88dDk",
        "colab_type": "text"
      },
      "source": [
        "## Transformer Attention\n",
        "\n",
        "First we will be working on the single head transformer attention mechanism. A single head attention takes 3 inputs as Query (q), Key (k) and Value (v), and it finds an unidirectional connection from each of the query words to each of the key words. In the transformer model key and value inputs are always the same.\n",
        "\n",
        "Each of query, key, value goes through a separate linear transform to a lower dimensionality to make the dimensionality of multi-headed attention to be smaller. Every linear layer in the Transformer model is using Xavier initialization ('glorot_uniform'). The output is then created by a rather simple mathematical equation:![Equation for Attention](https://jalammar.github.io/images/t/self-attention-matrix-calculation-2.png)\n",
        "\n",
        "Image source: Alammar, Jay (2018). The Illustrated Transformer. Retrieved from https://jalammar.github.io/illustrated-transformer/\n",
        "\n",
        "If we are making a decoder self-attention, we have to be a little careful since the full decoding sentence is not available in practice and should be generated step by step. Therefore, we cannot assume future attentions from the query word and a key word that has not been generated. Since the Transformer model is always generating the next word given an incomplete sequence, we should remove the attention from the query word to any word appeared later, which is the strictly upper triangular region except the main diagonals in $ Q \\times K^T $ matrix. That is to set the strict upper triangle of $ Q \\times K^T $ to negative infinity (zero after softmax)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nqXGnAVw_ow3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SingleHeadAttention(Layer):\n",
        "  def __init__(self, input_shape=(3, -1, d_model), dropout=.0, masked=None):\n",
        "    super(SingleHeadAttention, self).__init__()\n",
        "    self.q = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.normalize_q = Lambda(lambda x: x / np.sqrt(d_k))\n",
        "    self.k = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.v = Dense(d_v, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.dropout = dropout\n",
        "    self.masked = masked\n",
        "  \n",
        "  # Inputs: [query, key, value]\n",
        "  def call(self, inputs, training=None):\n",
        "    assert len(inputs) == 3\n",
        "    # We use a lambda layer to divide vector q by sqrt(d_k) according to the equation\n",
        "    q = self.normalize_q(self.q(inputs[0]))\n",
        "    k = self.k(inputs[1])\n",
        "    # The dimensionality of q is (batch_size, query_length, d_k) and that of k is (batch_size, key_length, d_k)\n",
        "    # So we will do a matrix multication by batch after transposing last 2 dimensions of k\n",
        "    # tf.shape(attn_weights) = (batch_size, query_length, key_length)\n",
        "    attn_weights = tf.matmul(q, tf.transpose(k, perm=[0,2,1]))\n",
        "    if self.masked: # Prevent future attentions in decoding self-attention\n",
        "      # Create a matrix where the strict upper triangle (not including main diagonal) is filled with -inf and 0 elsewhere\n",
        "      length = tf.shape(attn_weights)[-1]\n",
        "      #attn_mask = np.triu(tf.fill((length, length), -np.inf), k=1) # We need to use tensorflow functions instead of numpy\n",
        "      attn_mask = tf.fill((length, length), -np.inf)\n",
        "      attn_mask = tf.linalg.band_part(attn_mask, 0, -1) # Get upper triangle\n",
        "      attn_mask = tf.linalg.set_diag(attn_mask, tf.zeros((length))) # Set diagonal to zeros to avoid operations with infinity\n",
        "      # This matrix is added to the attention weights so all future attention will have -inf logits (0 after softmax)\n",
        "      attn_weights += attn_mask\n",
        "    # Softmax along the last dimension\n",
        "    attn_weights = tf.nn.softmax(attn_weights, axis=-1)\n",
        "    if training: # Attention dropout included in the original paper. This is possibly to encourage multihead diversity.\n",
        "      attn_weights = tf.nn.dropout(attn_weights, rate=self.dropout)\n",
        "    v = self.v(inputs[2])\n",
        "    return tf.matmul(attn_weights, v)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t8We5UHPIuZm",
        "colab_type": "text"
      },
      "source": [
        "Now let's use multiple single head attention and a linear layer to build a multihead attention. There is no need to reshape!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DiolVDYwFk00",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MultiHeadAttention(Layer):\n",
        "  def __init__(self, dropout=.0, masked=None):\n",
        "    super(MultiHeadAttention, self).__init__()\n",
        "    self.attn_heads = list()\n",
        "    for i in range(n_heads): \n",
        "      self.attn_heads.append(SingleHeadAttention(dropout=dropout, masked=masked))\n",
        "    self.linear = Dense(d_model, input_shape=(-1, n_heads * d_v), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    \n",
        "  def call(self, x, training=None):\n",
        "    attentions = [self.attn_heads[i](x, training=training) for i in range(n_heads)]\n",
        "    concatenated_attentions = tf.concat(attentions, axis=-1)\n",
        "    return self.linear(concatenated_attentions)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OzCW3dGJKvEJ",
        "colab_type": "text"
      },
      "source": [
        "## Encoder and Decoder\n",
        "\n",
        "This is the flowchart for the whole transformer architecture, where the encoder is the block to the left, and decoder is the block to the right. Note that since the output shape of either encoder or decoder is the same as its corresponding input shape, both the encoder unit and the decoder unit can be stacked.\n",
        "![Transformer Architecture](https://www.tensorflow.org/images/tutorials/transformer/transformer.png)\n",
        "\n",
        "We then present the transformer encoder architecture. Each encoder has a multihead self-attention (encoder-encoder) sublayer and a feedforward sublayer (two dense layers with ReLU activation in between). Each sublayer is followed by a LayerNorm taking the sublayer residually as follows:\n",
        "\n",
        "$$\\Large{\\mathit{LayerNorm}(x + \\mathit{sublayer}(x))} $$\n",
        "\n",
        "Dropout is applied after each sublayer before layer normalization.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gIyEIPm2sq9O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TransformerEncoder(Layer):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerEncoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_attention = MultiHeadAttention(dropout=self.attention_dropout_rate)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerEncoder, self).build(input_shape)\n",
        "  def call(self, x, training=None):\n",
        "    sublayer1 = self.multihead_attention((x, x, x), training=training)\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer1 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    return layernorm2\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DEz1iDH-Y-UG",
        "colab_type": "text"
      },
      "source": [
        "The decoder is constructed in the same fashion, except that there are three sublayers instead of two: a multihead self-attention layer (decoder-decoder), a multihead encoder attention layer (decoder-encoder) and a feedforward layer just like the one in an encoder unit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u6FC8ZLy8_-R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TransformerDecoder(Layer):\n",
        "  def __init__(self, dropout=.0, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerDecoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_self_attention = MultiHeadAttention(dropout=self.attention_dropout_rate, masked=True)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.multihead_encoder_attention = MultiHeadAttention(dropout=self.attention_dropout_rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout3 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization3 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerDecoder, self).build(input_shape)\n",
        "  def call(self, x, hidden, training=None):\n",
        "    sublayer1 = self.multihead_self_attention((x, x, x))\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.multihead_encoder_attention((x, hidden, hidden))\n",
        "    sublayer2 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    \n",
        "    sublayer3 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer3 = self.dropout3(sublayer3, training=training)\n",
        "    layernorm3 = self.layer_normalization2(layernorm2 + sublayer3)\n",
        "    return layernorm3\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EAuiM6DUNT3p",
        "colab_type": "code",
        "outputId": "9cd5d127-d702-4522-8895-997455b41ff2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "# Testing if the dimension matches!\n",
        "x = tf.ones((3, 26, d_model))\n",
        "x1 = tf.ones((3, 18, d_model))\n",
        "single_att = SingleHeadAttention(masked=None)\n",
        "multi_att = MultiHeadAttention()\n",
        "encoder = TransformerEncoder()\n",
        "decoder = TransformerDecoder()\n",
        "y = single_att((x, x, x)) # Self attention\n",
        "y1 = multi_att((x1, x, x)) # Encoder-decoder attention\n",
        "print(tf.shape(y))\n",
        "print(tf.shape(y1))\n",
        "y2 = encoder(x)\n",
        "y3 = decoder(x, y2)\n",
        "\n",
        "print(tf.shape(y2))\n",
        "print(tf.shape(y3))\n",
        "#print(layer.trainable_weights)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor([ 3 26 16], shape=(3,), dtype=int32)\n",
            "tf.Tensor([ 3 18 64], shape=(3,), dtype=int32)\n",
            "tf.Tensor([ 3 26 64], shape=(3,), dtype=int32)\n",
            "tf.Tensor([ 3 26 64], shape=(3,), dtype=int32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5gd9sGHuA22m",
        "colab_type": "text"
      },
      "source": [
        "## Positional Encoding\n",
        "\n",
        "In the original Transformer implementation, the researchers used a sinusoidal function to get positional encoding which will append to encoder and decoder word embeddings. This is to give information about the position of each token. The function looks as follows:\n",
        "\n",
        " $$\\Large{PE_{(pos, 2i)} = \\sin(pos / 10000^{2i / d_{model}})} $$\n",
        "$$\\Large{PE_{(pos, 2i+1)} = \\cos(pos / 10000^{2i / d_{model}})} $$ "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-5ZOk2dowenm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SinusoidalPositionalEncoding(Layer): #Inspired from https://github.com/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer_Torch.ipynb\n",
        "  def __init__(self):\n",
        "    super(SinusoidalPositionalEncoding, self).__init__()\n",
        "    self.sinusoidal_encoding = np.array([self.get_positional_angle(pos) for pos in range(max_token_length)], dtype=np.float32)\n",
        "    self.sinusoidal_encoding[:, 0::2] = np.sin(self.sinusoidal_encoding[:, 0::2])\n",
        "    self.sinusoidal_encoding[:, 1::2] = np.cos(self.sinusoidal_encoding[:, 1::2])\n",
        "    self.sinusoidal_encoding = tf.cast(self.sinusoidal_encoding, dtype=tf.float32) # Casting the array to Tensor for slicing\n",
        "  def call(self, x):\n",
        "    return x + self.sinusoidal_encoding[:tf.shape(x)[1]]\n",
        "    #return x + tf.slice(self.sinusoidal_encoding, [0, 0], [tf.shape(x)[1], d_model])\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "  def get_angle(self, pos, dim):\n",
        "    return pos / np.power(10000, 2 * (dim // 2) / d_model)\n",
        "  def get_positional_angle(self, pos):\n",
        "    return [self.get_angle(pos, dim) for dim in range(d_model)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wlhp-s1G60FQ",
        "colab_type": "text"
      },
      "source": [
        "## Assembling the Full Architecture\n",
        "Now we can build the full architecture of transformer using positional encoding, encoder layers and decoder layers:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OPyXChj3Xjet",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Transformer(Model):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(Transformer, self).__init__(**kwargs)\n",
        "    self.encoding_embedding = Embedding(source_vocab_len, d_model)\n",
        "    self.decoding_embedding = Embedding(target_vocab_len, d_model)\n",
        "    self.pos_encoding = SinusoidalPositionalEncoding()\n",
        "    self.encoder = [TransformerEncoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_encoder_layers)]\n",
        "    self.decoder = [TransformerDecoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_decoder_layers)]\n",
        "    self.decoder_final = Dense(target_vocab_len, input_shape=(None, d_model))\n",
        "  def call(self, inputs, training=None): # Source_sentence and decoder_input\n",
        "    source_sentence, decoder_input = inputs\n",
        "    embedded_source = self.encoding_embedding(source_sentence)\n",
        "    encoder_output = self.pos_encoding(embedded_source)\n",
        "    for encoder_unit in self.encoder:\n",
        "      encoder_output = encoder_unit(encoder_output, training=training)\n",
        "    \n",
        "    embedded_target = self.decoding_embedding(decoder_input)\n",
        "    decoder_output = self.pos_encoding(embedded_target)\n",
        "    for decoder_unit in self.decoder:\n",
        "      decoder_output = decoder_unit(decoder_output, encoder_output, training=training)\n",
        "    if training:\n",
        "      decoder_output = self.decoder_final(decoder_output)\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    else:\n",
        "      decoder_output = self.decoder_final(decoder_output[:, -1:, :])\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    return decoder_output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wN3g869inv7e",
        "colab_type": "text"
      },
      "source": [
        "# Training\n",
        "This model can be trained in two ways, either using TensorFlow GradientTape to update the model weights manually in a training function, or simply using Keras model.fit() method to start training. The former approach is more explanatory, and the latter is easier in coding. We will demonstrate both methods here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aeCyFxNg9Asc",
        "colab_type": "text"
      },
      "source": [
        "## Gradient Tape\n",
        "First let's try training using GradientTape:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u-FIcYm7waT1",
        "colab_type": "code",
        "outputId": "b1c1d827-b1be-4069-cff9-fdc65d55d4b9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# Demonstration on calling transformer model\n",
        "transformer = Transformer(dropout=.1)\n",
        "print(tf.shape(transformer([np.ones((5, 15)), np.ones((5, 12))], training=False)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor([ 5  1 60], shape=(3,), dtype=int32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pGmzf0sT1wHJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Specify loss, optimizer and training function\n",
        "crossentropy = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
        "\n",
        "def calc_loss(targets, logits):\n",
        "  mask = tf.math.logical_not(tf.math.equal(targets, 0))\n",
        "  mask = tf.cast(mask, dtype=tf.int64)\n",
        "  return crossentropy(targets, logits, sample_weight=mask)\n",
        "\n",
        "optimizer = tf.keras.optimizers.Adam()\n",
        "\n",
        "@tf.function # remove this annotation when debugging\n",
        "def train_step(source_seq, target_seq, target_labels):\n",
        "  \n",
        "  with tf.GradientTape() as tape:\n",
        "    logits = transformer([source_seq, target_seq], training=True) # Set training=True to use dropout in training\n",
        "    loss = calc_loss(target_labels, logits)\n",
        "\n",
        "  variables = transformer.trainable_variables\n",
        "  gradients = tape.gradient(loss, variables)\n",
        "  optimizer.apply_gradients(zip(gradients, variables))\n",
        "\n",
        "  return loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FjjgB1Md2Zby",
        "colab_type": "code",
        "outputId": "7fdc6927-57ff-4442-be66-5020ee9a7350",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 562
        }
      },
      "source": [
        "EPOCHS = 300\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "    \n",
        "    for batch, (source_seq, target_seq, target_labels) in enumerate(dataset):\n",
        "      loss = train_step(source_seq, target_seq, target_labels)\n",
        "      \n",
        "    if epoch % 10 == 0:\n",
        "      print(\"Epoch #%d, Loss %.4f\" % (epoch, loss))\n",
        "      #input_sent, target_sent, translation = translate()\n",
        "      #print(\"Input: %s\\nTarget: %s\\nTranslation: %s\\n\" % (input_sent, target_sent, translation))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch #0, Loss 2.3778\n",
            "Epoch #10, Loss 1.7720\n",
            "Epoch #20, Loss 1.4412\n",
            "Epoch #30, Loss 0.8138\n",
            "Epoch #40, Loss 0.5184\n",
            "Epoch #50, Loss 0.3874\n",
            "Epoch #60, Loss 0.2446\n",
            "Epoch #70, Loss 0.1178\n",
            "Epoch #80, Loss 0.0721\n",
            "Epoch #90, Loss 0.0410\n",
            "Epoch #100, Loss 0.0302\n",
            "Epoch #110, Loss 0.0281\n",
            "Epoch #120, Loss 0.0164\n",
            "Epoch #130, Loss 0.0124\n",
            "Epoch #140, Loss 0.0126\n",
            "Epoch #150, Loss 0.0099\n",
            "Epoch #160, Loss 0.0067\n",
            "Epoch #170, Loss 0.0073\n",
            "Epoch #180, Loss 0.0058\n",
            "Epoch #190, Loss 0.0047\n",
            "Epoch #200, Loss 0.0056\n",
            "Epoch #210, Loss 0.0038\n",
            "Epoch #220, Loss 0.0049\n",
            "Epoch #230, Loss 0.0040\n",
            "Epoch #240, Loss 0.0030\n",
            "Epoch #250, Loss 0.0032\n",
            "Epoch #260, Loss 0.0030\n",
            "Epoch #270, Loss 0.0030\n",
            "Epoch #280, Loss 0.0029\n",
            "Epoch #290, Loss 0.0022\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xs2Y9h-Tn_O2",
        "colab_type": "text"
      },
      "source": [
        "## Keras model.fit()\n",
        "Alternatively we can simply use model.fit() since we built the whole transformer as one keras model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TQS00n7DfcYZ",
        "colab_type": "code",
        "outputId": "eae43f6f-ca59-435a-9ac4-b7f5c44d648b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 11028
        }
      },
      "source": [
        "transformer_2 = Transformer() # Instantiating a new transformer model\n",
        "src_seqs, tgt_seqs, tgt_labels = zip(*dataset_2)\n",
        "train = [tf.cast(src_seqs, dtype=tf.float32), tf.cast(tgt_seqs, dtype=tf.float32)] # Cast the tuples to tensors\n",
        "\n",
        "transformer_2.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "transformer_2.fit(train, tf.cast(tgt_labels, dtype=tf.float32), verbose=2, batch_size=5, epochs=300)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0518 00:35:31.332310 140482633267072 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/300\n",
            "20/20 - 4s - loss: 3.7897 - accuracy: 0.2667\n",
            "Epoch 2/300\n",
            "20/20 - 0s - loss: 3.1120 - accuracy: 0.3542\n",
            "Epoch 3/300\n",
            "20/20 - 0s - loss: 2.9551 - accuracy: 0.3542\n",
            "Epoch 4/300\n",
            "20/20 - 0s - loss: 2.9216 - accuracy: 0.3542\n",
            "Epoch 5/300\n",
            "20/20 - 0s - loss: 2.8347 - accuracy: 0.3542\n",
            "Epoch 6/300\n",
            "20/20 - 0s - loss: 2.6874 - accuracy: 0.3542\n",
            "Epoch 7/300\n",
            "20/20 - 0s - loss: 2.5986 - accuracy: 0.3958\n",
            "Epoch 8/300\n",
            "20/20 - 0s - loss: 2.4959 - accuracy: 0.3917\n",
            "Epoch 9/300\n",
            "20/20 - 0s - loss: 2.4009 - accuracy: 0.4083\n",
            "Epoch 10/300\n",
            "20/20 - 0s - loss: 2.3541 - accuracy: 0.4083\n",
            "Epoch 11/300\n",
            "20/20 - 0s - loss: 2.2928 - accuracy: 0.4333\n",
            "Epoch 12/300\n",
            "20/20 - 0s - loss: 2.2552 - accuracy: 0.4208\n",
            "Epoch 13/300\n",
            "20/20 - 0s - loss: 2.1605 - accuracy: 0.4583\n",
            "Epoch 14/300\n",
            "20/20 - 0s - loss: 2.1244 - accuracy: 0.4708\n",
            "Epoch 15/300\n",
            "20/20 - 0s - loss: 2.1521 - accuracy: 0.4292\n",
            "Epoch 16/300\n",
            "20/20 - 0s - loss: 2.0539 - accuracy: 0.4583\n",
            "Epoch 17/300\n",
            "20/20 - 0s - loss: 1.9400 - accuracy: 0.4917\n",
            "Epoch 18/300\n",
            "20/20 - 0s - loss: 1.9342 - accuracy: 0.4917\n",
            "Epoch 19/300\n",
            "20/20 - 0s - loss: 1.8924 - accuracy: 0.4833\n",
            "Epoch 20/300\n",
            "20/20 - 0s - loss: 1.8224 - accuracy: 0.5167\n",
            "Epoch 21/300\n",
            "20/20 - 0s - loss: 1.7431 - accuracy: 0.5333\n",
            "Epoch 22/300\n",
            "20/20 - 0s - loss: 1.7723 - accuracy: 0.5458\n",
            "Epoch 23/300\n",
            "20/20 - 0s - loss: 1.6737 - accuracy: 0.5583\n",
            "Epoch 24/300\n",
            "20/20 - 0s - loss: 1.6444 - accuracy: 0.5708\n",
            "Epoch 25/300\n",
            "20/20 - 0s - loss: 1.5968 - accuracy: 0.6042\n",
            "Epoch 26/300\n",
            "20/20 - 0s - loss: 1.5845 - accuracy: 0.5708\n",
            "Epoch 27/300\n",
            "20/20 - 0s - loss: 1.5010 - accuracy: 0.6083\n",
            "Epoch 28/300\n",
            "20/20 - 0s - loss: 1.4430 - accuracy: 0.6458\n",
            "Epoch 29/300\n",
            "20/20 - 0s - loss: 1.3921 - accuracy: 0.6458\n",
            "Epoch 30/300\n",
            "20/20 - 0s - loss: 1.3602 - accuracy: 0.6667\n",
            "Epoch 31/300\n",
            "20/20 - 0s - loss: 1.3228 - accuracy: 0.6292\n",
            "Epoch 32/300\n",
            "20/20 - 0s - loss: 1.2651 - accuracy: 0.6542\n",
            "Epoch 33/300\n",
            "20/20 - 0s - loss: 1.2410 - accuracy: 0.6917\n",
            "Epoch 34/300\n",
            "20/20 - 0s - loss: 1.1822 - accuracy: 0.7000\n",
            "Epoch 35/300\n",
            "20/20 - 0s - loss: 1.1349 - accuracy: 0.7042\n",
            "Epoch 36/300\n",
            "20/20 - 0s - loss: 1.0577 - accuracy: 0.7292\n",
            "Epoch 37/300\n",
            "20/20 - 0s - loss: 1.0509 - accuracy: 0.7000\n",
            "Epoch 38/300\n",
            "20/20 - 0s - loss: 0.9782 - accuracy: 0.7250\n",
            "Epoch 39/300\n",
            "20/20 - 0s - loss: 0.9720 - accuracy: 0.7250\n",
            "Epoch 40/300\n",
            "20/20 - 0s - loss: 0.8918 - accuracy: 0.7542\n",
            "Epoch 41/300\n",
            "20/20 - 0s - loss: 0.8629 - accuracy: 0.7625\n",
            "Epoch 42/300\n",
            "20/20 - 0s - loss: 0.8224 - accuracy: 0.7833\n",
            "Epoch 43/300\n",
            "20/20 - 0s - loss: 0.8162 - accuracy: 0.8083\n",
            "Epoch 44/300\n",
            "20/20 - 0s - loss: 0.7582 - accuracy: 0.8083\n",
            "Epoch 45/300\n",
            "20/20 - 0s - loss: 0.7388 - accuracy: 0.7958\n",
            "Epoch 46/300\n",
            "20/20 - 0s - loss: 0.7018 - accuracy: 0.8333\n",
            "Epoch 47/300\n",
            "20/20 - 0s - loss: 0.6369 - accuracy: 0.8583\n",
            "Epoch 48/300\n",
            "20/20 - 0s - loss: 0.6560 - accuracy: 0.8500\n",
            "Epoch 49/300\n",
            "20/20 - 0s - loss: 0.6012 - accuracy: 0.8667\n",
            "Epoch 50/300\n",
            "20/20 - 0s - loss: 0.5860 - accuracy: 0.8708\n",
            "Epoch 51/300\n",
            "20/20 - 0s - loss: 0.5619 - accuracy: 0.8792\n",
            "Epoch 52/300\n",
            "20/20 - 0s - loss: 0.5647 - accuracy: 0.8667\n",
            "Epoch 53/300\n",
            "20/20 - 0s - loss: 0.5209 - accuracy: 0.8792\n",
            "Epoch 54/300\n",
            "20/20 - 0s - loss: 0.5208 - accuracy: 0.8917\n",
            "Epoch 55/300\n",
            "20/20 - 0s - loss: 0.4849 - accuracy: 0.8833\n",
            "Epoch 56/300\n",
            "20/20 - 0s - loss: 0.4649 - accuracy: 0.8958\n",
            "Epoch 57/300\n",
            "20/20 - 0s - loss: 0.4697 - accuracy: 0.9000\n",
            "Epoch 58/300\n",
            "20/20 - 0s - loss: 0.4653 - accuracy: 0.8958\n",
            "Epoch 59/300\n",
            "20/20 - 0s - loss: 0.4220 - accuracy: 0.9083\n",
            "Epoch 60/300\n",
            "20/20 - 0s - loss: 0.4252 - accuracy: 0.9083\n",
            "Epoch 61/300\n",
            "20/20 - 0s - loss: 0.4393 - accuracy: 0.8958\n",
            "Epoch 62/300\n",
            "20/20 - 0s - loss: 0.4250 - accuracy: 0.9042\n",
            "Epoch 63/300\n",
            "20/20 - 0s - loss: 0.3766 - accuracy: 0.9167\n",
            "Epoch 64/300\n",
            "20/20 - 0s - loss: 0.3806 - accuracy: 0.9208\n",
            "Epoch 65/300\n",
            "20/20 - 0s - loss: 0.3691 - accuracy: 0.9208\n",
            "Epoch 66/300\n",
            "20/20 - 0s - loss: 0.3777 - accuracy: 0.9250\n",
            "Epoch 67/300\n",
            "20/20 - 0s - loss: 0.3632 - accuracy: 0.9125\n",
            "Epoch 68/300\n",
            "20/20 - 0s - loss: 0.3651 - accuracy: 0.9250\n",
            "Epoch 69/300\n",
            "20/20 - 0s - loss: 0.3459 - accuracy: 0.9250\n",
            "Epoch 70/300\n",
            "20/20 - 0s - loss: 0.3493 - accuracy: 0.9167\n",
            "Epoch 71/300\n",
            "20/20 - 0s - loss: 0.3474 - accuracy: 0.9208\n",
            "Epoch 72/300\n",
            "20/20 - 0s - loss: 0.3159 - accuracy: 0.9250\n",
            "Epoch 73/300\n",
            "20/20 - 0s - loss: 0.3244 - accuracy: 0.9208\n",
            "Epoch 74/300\n",
            "20/20 - 0s - loss: 0.3081 - accuracy: 0.9250\n",
            "Epoch 75/300\n",
            "20/20 - 0s - loss: 0.3072 - accuracy: 0.9250\n",
            "Epoch 76/300\n",
            "20/20 - 0s - loss: 0.2882 - accuracy: 0.9208\n",
            "Epoch 77/300\n",
            "20/20 - 0s - loss: 0.2720 - accuracy: 0.9292\n",
            "Epoch 78/300\n",
            "20/20 - 0s - loss: 0.2738 - accuracy: 0.9333\n",
            "Epoch 79/300\n",
            "20/20 - 0s - loss: 0.2603 - accuracy: 0.9292\n",
            "Epoch 80/300\n",
            "20/20 - 0s - loss: 0.2358 - accuracy: 0.9458\n",
            "Epoch 81/300\n",
            "20/20 - 0s - loss: 0.2172 - accuracy: 0.9458\n",
            "Epoch 82/300\n",
            "20/20 - 0s - loss: 0.2401 - accuracy: 0.9333\n",
            "Epoch 83/300\n",
            "20/20 - 0s - loss: 0.2163 - accuracy: 0.9583\n",
            "Epoch 84/300\n",
            "20/20 - 0s - loss: 0.2088 - accuracy: 0.9542\n",
            "Epoch 85/300\n",
            "20/20 - 0s - loss: 0.2070 - accuracy: 0.9458\n",
            "Epoch 86/300\n",
            "20/20 - 0s - loss: 0.1956 - accuracy: 0.9625\n",
            "Epoch 87/300\n",
            "20/20 - 0s - loss: 0.1776 - accuracy: 0.9792\n",
            "Epoch 88/300\n",
            "20/20 - 0s - loss: 0.1802 - accuracy: 0.9667\n",
            "Epoch 89/300\n",
            "20/20 - 0s - loss: 0.1581 - accuracy: 0.9833\n",
            "Epoch 90/300\n",
            "20/20 - 0s - loss: 0.1569 - accuracy: 0.9875\n",
            "Epoch 91/300\n",
            "20/20 - 0s - loss: 0.1505 - accuracy: 0.9792\n",
            "Epoch 92/300\n",
            "20/20 - 0s - loss: 0.1372 - accuracy: 0.9833\n",
            "Epoch 93/300\n",
            "20/20 - 0s - loss: 0.1216 - accuracy: 0.9875\n",
            "Epoch 94/300\n",
            "20/20 - 0s - loss: 0.1224 - accuracy: 0.9917\n",
            "Epoch 95/300\n",
            "20/20 - 0s - loss: 0.1039 - accuracy: 1.0000\n",
            "Epoch 96/300\n",
            "20/20 - 0s - loss: 0.1133 - accuracy: 0.9833\n",
            "Epoch 97/300\n",
            "20/20 - 0s - loss: 0.1030 - accuracy: 0.9875\n",
            "Epoch 98/300\n",
            "20/20 - 0s - loss: 0.0908 - accuracy: 0.9958\n",
            "Epoch 99/300\n",
            "20/20 - 0s - loss: 0.0875 - accuracy: 0.9875\n",
            "Epoch 100/300\n",
            "20/20 - 0s - loss: 0.0795 - accuracy: 1.0000\n",
            "Epoch 101/300\n",
            "20/20 - 0s - loss: 0.0818 - accuracy: 0.9958\n",
            "Epoch 102/300\n",
            "20/20 - 0s - loss: 0.0715 - accuracy: 1.0000\n",
            "Epoch 103/300\n",
            "20/20 - 0s - loss: 0.0767 - accuracy: 1.0000\n",
            "Epoch 104/300\n",
            "20/20 - 0s - loss: 0.0588 - accuracy: 0.9958\n",
            "Epoch 105/300\n",
            "20/20 - 0s - loss: 0.0605 - accuracy: 0.9917\n",
            "Epoch 106/300\n",
            "20/20 - 0s - loss: 0.0472 - accuracy: 1.0000\n",
            "Epoch 107/300\n",
            "20/20 - 0s - loss: 0.0534 - accuracy: 1.0000\n",
            "Epoch 108/300\n",
            "20/20 - 0s - loss: 0.0519 - accuracy: 1.0000\n",
            "Epoch 109/300\n",
            "20/20 - 0s - loss: 0.0493 - accuracy: 1.0000\n",
            "Epoch 110/300\n",
            "20/20 - 0s - loss: 0.0445 - accuracy: 1.0000\n",
            "Epoch 111/300\n",
            "20/20 - 0s - loss: 0.0448 - accuracy: 1.0000\n",
            "Epoch 112/300\n",
            "20/20 - 0s - loss: 0.0435 - accuracy: 1.0000\n",
            "Epoch 113/300\n",
            "20/20 - 0s - loss: 0.0380 - accuracy: 1.0000\n",
            "Epoch 114/300\n",
            "20/20 - 0s - loss: 0.0318 - accuracy: 1.0000\n",
            "Epoch 115/300\n",
            "20/20 - 0s - loss: 0.0381 - accuracy: 0.9958\n",
            "Epoch 116/300\n",
            "20/20 - 0s - loss: 0.0348 - accuracy: 1.0000\n",
            "Epoch 117/300\n",
            "20/20 - 0s - loss: 0.0329 - accuracy: 1.0000\n",
            "Epoch 118/300\n",
            "20/20 - 0s - loss: 0.0312 - accuracy: 1.0000\n",
            "Epoch 119/300\n",
            "20/20 - 0s - loss: 0.0290 - accuracy: 1.0000\n",
            "Epoch 120/300\n",
            "20/20 - 0s - loss: 0.0395 - accuracy: 0.9958\n",
            "Epoch 121/300\n",
            "20/20 - 0s - loss: 0.0316 - accuracy: 1.0000\n",
            "Epoch 122/300\n",
            "20/20 - 0s - loss: 0.0299 - accuracy: 1.0000\n",
            "Epoch 123/300\n",
            "20/20 - 0s - loss: 0.0366 - accuracy: 0.9958\n",
            "Epoch 124/300\n",
            "20/20 - 0s - loss: 0.0300 - accuracy: 1.0000\n",
            "Epoch 125/300\n",
            "20/20 - 0s - loss: 0.0409 - accuracy: 0.9958\n",
            "Epoch 126/300\n",
            "20/20 - 0s - loss: 0.0352 - accuracy: 0.9958\n",
            "Epoch 127/300\n",
            "20/20 - 0s - loss: 0.0290 - accuracy: 1.0000\n",
            "Epoch 128/300\n",
            "20/20 - 0s - loss: 0.0275 - accuracy: 0.9958\n",
            "Epoch 129/300\n",
            "20/20 - 0s - loss: 0.0264 - accuracy: 1.0000\n",
            "Epoch 130/300\n",
            "20/20 - 0s - loss: 0.0242 - accuracy: 1.0000\n",
            "Epoch 131/300\n",
            "20/20 - 0s - loss: 0.0231 - accuracy: 1.0000\n",
            "Epoch 132/300\n",
            "20/20 - 0s - loss: 0.0233 - accuracy: 1.0000\n",
            "Epoch 133/300\n",
            "20/20 - 0s - loss: 0.0187 - accuracy: 1.0000\n",
            "Epoch 134/300\n",
            "20/20 - 0s - loss: 0.0216 - accuracy: 1.0000\n",
            "Epoch 135/300\n",
            "20/20 - 0s - loss: 0.0183 - accuracy: 1.0000\n",
            "Epoch 136/300\n",
            "20/20 - 0s - loss: 0.0169 - accuracy: 1.0000\n",
            "Epoch 137/300\n",
            "20/20 - 0s - loss: 0.0182 - accuracy: 1.0000\n",
            "Epoch 138/300\n",
            "20/20 - 0s - loss: 0.0161 - accuracy: 1.0000\n",
            "Epoch 139/300\n",
            "20/20 - 0s - loss: 0.0146 - accuracy: 1.0000\n",
            "Epoch 140/300\n",
            "20/20 - 0s - loss: 0.0153 - accuracy: 1.0000\n",
            "Epoch 141/300\n",
            "20/20 - 0s - loss: 0.0155 - accuracy: 1.0000\n",
            "Epoch 142/300\n",
            "20/20 - 0s - loss: 0.0144 - accuracy: 1.0000\n",
            "Epoch 143/300\n",
            "20/20 - 0s - loss: 0.0144 - accuracy: 1.0000\n",
            "Epoch 144/300\n",
            "20/20 - 0s - loss: 0.0148 - accuracy: 1.0000\n",
            "Epoch 145/300\n",
            "20/20 - 0s - loss: 0.0137 - accuracy: 1.0000\n",
            "Epoch 146/300\n",
            "20/20 - 0s - loss: 0.0121 - accuracy: 1.0000\n",
            "Epoch 147/300\n",
            "20/20 - 0s - loss: 0.0122 - accuracy: 1.0000\n",
            "Epoch 148/300\n",
            "20/20 - 0s - loss: 0.0120 - accuracy: 1.0000\n",
            "Epoch 149/300\n",
            "20/20 - 0s - loss: 0.0120 - accuracy: 1.0000\n",
            "Epoch 150/300\n",
            "20/20 - 0s - loss: 0.0124 - accuracy: 1.0000\n",
            "Epoch 151/300\n",
            "20/20 - 0s - loss: 0.0115 - accuracy: 1.0000\n",
            "Epoch 152/300\n",
            "20/20 - 0s - loss: 0.0108 - accuracy: 1.0000\n",
            "Epoch 153/300\n",
            "20/20 - 0s - loss: 0.0112 - accuracy: 1.0000\n",
            "Epoch 154/300\n",
            "20/20 - 0s - loss: 0.0112 - accuracy: 1.0000\n",
            "Epoch 155/300\n",
            "20/20 - 0s - loss: 0.0111 - accuracy: 1.0000\n",
            "Epoch 156/300\n",
            "20/20 - 0s - loss: 0.0101 - accuracy: 1.0000\n",
            "Epoch 157/300\n",
            "20/20 - 0s - loss: 0.0103 - accuracy: 1.0000\n",
            "Epoch 158/300\n",
            "20/20 - 0s - loss: 0.0103 - accuracy: 1.0000\n",
            "Epoch 159/300\n",
            "20/20 - 0s - loss: 0.0110 - accuracy: 1.0000\n",
            "Epoch 160/300\n",
            "20/20 - 0s - loss: 0.0102 - accuracy: 1.0000\n",
            "Epoch 161/300\n",
            "20/20 - 0s - loss: 0.0102 - accuracy: 1.0000\n",
            "Epoch 162/300\n",
            "20/20 - 0s - loss: 0.0087 - accuracy: 1.0000\n",
            "Epoch 163/300\n",
            "20/20 - 0s - loss: 0.0086 - accuracy: 1.0000\n",
            "Epoch 164/300\n",
            "20/20 - 0s - loss: 0.0099 - accuracy: 1.0000\n",
            "Epoch 165/300\n",
            "20/20 - 0s - loss: 0.0087 - accuracy: 1.0000\n",
            "Epoch 166/300\n",
            "20/20 - 0s - loss: 0.0090 - accuracy: 1.0000\n",
            "Epoch 167/300\n",
            "20/20 - 0s - loss: 0.0090 - accuracy: 1.0000\n",
            "Epoch 168/300\n",
            "20/20 - 0s - loss: 0.0081 - accuracy: 1.0000\n",
            "Epoch 169/300\n",
            "20/20 - 0s - loss: 0.0090 - accuracy: 1.0000\n",
            "Epoch 170/300\n",
            "20/20 - 0s - loss: 0.0070 - accuracy: 1.0000\n",
            "Epoch 171/300\n",
            "20/20 - 0s - loss: 0.0083 - accuracy: 1.0000\n",
            "Epoch 172/300\n",
            "20/20 - 0s - loss: 0.0081 - accuracy: 1.0000\n",
            "Epoch 173/300\n",
            "20/20 - 0s - loss: 0.0076 - accuracy: 1.0000\n",
            "Epoch 174/300\n",
            "20/20 - 0s - loss: 0.0087 - accuracy: 1.0000\n",
            "Epoch 175/300\n",
            "20/20 - 0s - loss: 0.0076 - accuracy: 1.0000\n",
            "Epoch 176/300\n",
            "20/20 - 0s - loss: 0.0082 - accuracy: 1.0000\n",
            "Epoch 177/300\n",
            "20/20 - 0s - loss: 0.0071 - accuracy: 1.0000\n",
            "Epoch 178/300\n",
            "20/20 - 0s - loss: 0.0074 - accuracy: 1.0000\n",
            "Epoch 179/300\n",
            "20/20 - 0s - loss: 0.0069 - accuracy: 1.0000\n",
            "Epoch 180/300\n",
            "20/20 - 0s - loss: 0.0074 - accuracy: 1.0000\n",
            "Epoch 181/300\n",
            "20/20 - 0s - loss: 0.0073 - accuracy: 1.0000\n",
            "Epoch 182/300\n",
            "20/20 - 0s - loss: 0.0074 - accuracy: 1.0000\n",
            "Epoch 183/300\n",
            "20/20 - 0s - loss: 0.0070 - accuracy: 1.0000\n",
            "Epoch 184/300\n",
            "20/20 - 0s - loss: 0.0074 - accuracy: 1.0000\n",
            "Epoch 185/300\n",
            "20/20 - 0s - loss: 0.0063 - accuracy: 1.0000\n",
            "Epoch 186/300\n",
            "20/20 - 0s - loss: 0.0072 - accuracy: 1.0000\n",
            "Epoch 187/300\n",
            "20/20 - 0s - loss: 0.0062 - accuracy: 1.0000\n",
            "Epoch 188/300\n",
            "20/20 - 0s - loss: 0.0067 - accuracy: 1.0000\n",
            "Epoch 189/300\n",
            "20/20 - 0s - loss: 0.0064 - accuracy: 1.0000\n",
            "Epoch 190/300\n",
            "20/20 - 0s - loss: 0.0079 - accuracy: 1.0000\n",
            "Epoch 191/300\n",
            "20/20 - 0s - loss: 0.0064 - accuracy: 1.0000\n",
            "Epoch 192/300\n",
            "20/20 - 0s - loss: 0.0109 - accuracy: 1.0000\n",
            "Epoch 193/300\n",
            "20/20 - 0s - loss: 0.0149 - accuracy: 1.0000\n",
            "Epoch 194/300\n",
            "20/20 - 0s - loss: 0.0095 - accuracy: 1.0000\n",
            "Epoch 195/300\n",
            "20/20 - 0s - loss: 0.0115 - accuracy: 1.0000\n",
            "Epoch 196/300\n",
            "20/20 - 0s - loss: 0.0117 - accuracy: 1.0000\n",
            "Epoch 197/300\n",
            "20/20 - 0s - loss: 0.0087 - accuracy: 1.0000\n",
            "Epoch 198/300\n",
            "20/20 - 0s - loss: 0.0074 - accuracy: 1.0000\n",
            "Epoch 199/300\n",
            "20/20 - 0s - loss: 0.0072 - accuracy: 1.0000\n",
            "Epoch 200/300\n",
            "20/20 - 0s - loss: 0.0072 - accuracy: 1.0000\n",
            "Epoch 201/300\n",
            "20/20 - 0s - loss: 0.0057 - accuracy: 1.0000\n",
            "Epoch 202/300\n",
            "20/20 - 0s - loss: 0.0060 - accuracy: 1.0000\n",
            "Epoch 203/300\n",
            "20/20 - 0s - loss: 0.0064 - accuracy: 1.0000\n",
            "Epoch 204/300\n",
            "20/20 - 0s - loss: 0.0060 - accuracy: 1.0000\n",
            "Epoch 205/300\n",
            "20/20 - 0s - loss: 0.0056 - accuracy: 1.0000\n",
            "Epoch 206/300\n",
            "20/20 - 0s - loss: 0.0051 - accuracy: 1.0000\n",
            "Epoch 207/300\n",
            "20/20 - 0s - loss: 0.0054 - accuracy: 1.0000\n",
            "Epoch 208/300\n",
            "20/20 - 0s - loss: 0.0056 - accuracy: 1.0000\n",
            "Epoch 209/300\n",
            "20/20 - 0s - loss: 0.0060 - accuracy: 1.0000\n",
            "Epoch 210/300\n",
            "20/20 - 0s - loss: 0.0052 - accuracy: 1.0000\n",
            "Epoch 211/300\n",
            "20/20 - 0s - loss: 0.0055 - accuracy: 1.0000\n",
            "Epoch 212/300\n",
            "20/20 - 0s - loss: 0.0049 - accuracy: 1.0000\n",
            "Epoch 213/300\n",
            "20/20 - 0s - loss: 0.0051 - accuracy: 1.0000\n",
            "Epoch 214/300\n",
            "20/20 - 0s - loss: 0.0054 - accuracy: 1.0000\n",
            "Epoch 215/300\n",
            "20/20 - 0s - loss: 0.0055 - accuracy: 1.0000\n",
            "Epoch 216/300\n",
            "20/20 - 0s - loss: 0.0055 - accuracy: 1.0000\n",
            "Epoch 217/300\n",
            "20/20 - 0s - loss: 0.0056 - accuracy: 1.0000\n",
            "Epoch 218/300\n",
            "20/20 - 0s - loss: 0.0048 - accuracy: 1.0000\n",
            "Epoch 219/300\n",
            "20/20 - 0s - loss: 0.0044 - accuracy: 1.0000\n",
            "Epoch 220/300\n",
            "20/20 - 0s - loss: 0.0048 - accuracy: 1.0000\n",
            "Epoch 221/300\n",
            "20/20 - 0s - loss: 0.0049 - accuracy: 1.0000\n",
            "Epoch 222/300\n",
            "20/20 - 0s - loss: 0.0047 - accuracy: 1.0000\n",
            "Epoch 223/300\n",
            "20/20 - 0s - loss: 0.0043 - accuracy: 1.0000\n",
            "Epoch 224/300\n",
            "20/20 - 0s - loss: 0.0042 - accuracy: 1.0000\n",
            "Epoch 225/300\n",
            "20/20 - 0s - loss: 0.0040 - accuracy: 1.0000\n",
            "Epoch 226/300\n",
            "20/20 - 0s - loss: 0.0042 - accuracy: 1.0000\n",
            "Epoch 227/300\n",
            "20/20 - 0s - loss: 0.0047 - accuracy: 1.0000\n",
            "Epoch 228/300\n",
            "20/20 - 0s - loss: 0.0040 - accuracy: 1.0000\n",
            "Epoch 229/300\n",
            "20/20 - 0s - loss: 0.0045 - accuracy: 1.0000\n",
            "Epoch 230/300\n",
            "20/20 - 0s - loss: 0.0044 - accuracy: 1.0000\n",
            "Epoch 231/300\n",
            "20/20 - 0s - loss: 0.0043 - accuracy: 1.0000\n",
            "Epoch 232/300\n",
            "20/20 - 0s - loss: 0.0041 - accuracy: 1.0000\n",
            "Epoch 233/300\n",
            "20/20 - 0s - loss: 0.0041 - accuracy: 1.0000\n",
            "Epoch 234/300\n",
            "20/20 - 0s - loss: 0.0044 - accuracy: 1.0000\n",
            "Epoch 235/300\n",
            "20/20 - 0s - loss: 0.0039 - accuracy: 1.0000\n",
            "Epoch 236/300\n",
            "20/20 - 0s - loss: 0.0043 - accuracy: 1.0000\n",
            "Epoch 237/300\n",
            "20/20 - 0s - loss: 0.0038 - accuracy: 1.0000\n",
            "Epoch 238/300\n",
            "20/20 - 0s - loss: 0.0038 - accuracy: 1.0000\n",
            "Epoch 239/300\n",
            "20/20 - 0s - loss: 0.0045 - accuracy: 1.0000\n",
            "Epoch 240/300\n",
            "20/20 - 0s - loss: 0.0039 - accuracy: 1.0000\n",
            "Epoch 241/300\n",
            "20/20 - 0s - loss: 0.0038 - accuracy: 1.0000\n",
            "Epoch 242/300\n",
            "20/20 - 0s - loss: 0.0036 - accuracy: 1.0000\n",
            "Epoch 243/300\n",
            "20/20 - 0s - loss: 0.0037 - accuracy: 1.0000\n",
            "Epoch 244/300\n",
            "20/20 - 0s - loss: 0.0036 - accuracy: 1.0000\n",
            "Epoch 245/300\n",
            "20/20 - 0s - loss: 0.0038 - accuracy: 1.0000\n",
            "Epoch 246/300\n",
            "20/20 - 0s - loss: 0.0040 - accuracy: 1.0000\n",
            "Epoch 247/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 248/300\n",
            "20/20 - 0s - loss: 0.0035 - accuracy: 1.0000\n",
            "Epoch 249/300\n",
            "20/20 - 0s - loss: 0.0038 - accuracy: 1.0000\n",
            "Epoch 250/300\n",
            "20/20 - 0s - loss: 0.0035 - accuracy: 1.0000\n",
            "Epoch 251/300\n",
            "20/20 - 0s - loss: 0.0036 - accuracy: 1.0000\n",
            "Epoch 252/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 253/300\n",
            "20/20 - 0s - loss: 0.0032 - accuracy: 1.0000\n",
            "Epoch 254/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 255/300\n",
            "20/20 - 0s - loss: 0.0036 - accuracy: 1.0000\n",
            "Epoch 256/300\n",
            "20/20 - 0s - loss: 0.0035 - accuracy: 1.0000\n",
            "Epoch 257/300\n",
            "20/20 - 0s - loss: 0.0033 - accuracy: 1.0000\n",
            "Epoch 258/300\n",
            "20/20 - 0s - loss: 0.0033 - accuracy: 1.0000\n",
            "Epoch 259/300\n",
            "20/20 - 0s - loss: 0.0031 - accuracy: 1.0000\n",
            "Epoch 260/300\n",
            "20/20 - 0s - loss: 0.0033 - accuracy: 1.0000\n",
            "Epoch 261/300\n",
            "20/20 - 0s - loss: 0.0035 - accuracy: 1.0000\n",
            "Epoch 262/300\n",
            "20/20 - 0s - loss: 0.0029 - accuracy: 1.0000\n",
            "Epoch 263/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 264/300\n",
            "20/20 - 0s - loss: 0.0032 - accuracy: 1.0000\n",
            "Epoch 265/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 266/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 267/300\n",
            "20/20 - 0s - loss: 0.0031 - accuracy: 1.0000\n",
            "Epoch 268/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 269/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 270/300\n",
            "20/20 - 0s - loss: 0.0034 - accuracy: 1.0000\n",
            "Epoch 271/300\n",
            "20/20 - 0s - loss: 0.0031 - accuracy: 1.0000\n",
            "Epoch 272/300\n",
            "20/20 - 0s - loss: 0.0028 - accuracy: 1.0000\n",
            "Epoch 273/300\n",
            "20/20 - 0s - loss: 0.0031 - accuracy: 1.0000\n",
            "Epoch 274/300\n",
            "20/20 - 0s - loss: 0.0028 - accuracy: 1.0000\n",
            "Epoch 275/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 276/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 277/300\n",
            "20/20 - 0s - loss: 0.0029 - accuracy: 1.0000\n",
            "Epoch 278/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 279/300\n",
            "20/20 - 0s - loss: 0.0029 - accuracy: 1.0000\n",
            "Epoch 280/300\n",
            "20/20 - 0s - loss: 0.0027 - accuracy: 1.0000\n",
            "Epoch 281/300\n",
            "20/20 - 0s - loss: 0.0028 - accuracy: 1.0000\n",
            "Epoch 282/300\n",
            "20/20 - 0s - loss: 0.0028 - accuracy: 1.0000\n",
            "Epoch 283/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 284/300\n",
            "20/20 - 0s - loss: 0.0027 - accuracy: 1.0000\n",
            "Epoch 285/300\n",
            "20/20 - 0s - loss: 0.0027 - accuracy: 1.0000\n",
            "Epoch 286/300\n",
            "20/20 - 0s - loss: 0.0029 - accuracy: 1.0000\n",
            "Epoch 287/300\n",
            "20/20 - 0s - loss: 0.0029 - accuracy: 1.0000\n",
            "Epoch 288/300\n",
            "20/20 - 0s - loss: 0.0026 - accuracy: 1.0000\n",
            "Epoch 289/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 290/300\n",
            "20/20 - 0s - loss: 0.0028 - accuracy: 1.0000\n",
            "Epoch 291/300\n",
            "20/20 - 0s - loss: 0.0024 - accuracy: 1.0000\n",
            "Epoch 292/300\n",
            "20/20 - 0s - loss: 0.0031 - accuracy: 1.0000\n",
            "Epoch 293/300\n",
            "20/20 - 0s - loss: 0.0027 - accuracy: 1.0000\n",
            "Epoch 294/300\n",
            "20/20 - 0s - loss: 0.0027 - accuracy: 1.0000\n",
            "Epoch 295/300\n",
            "20/20 - 0s - loss: 0.0025 - accuracy: 1.0000\n",
            "Epoch 296/300\n",
            "20/20 - 0s - loss: 0.0029 - accuracy: 1.0000\n",
            "Epoch 297/300\n",
            "20/20 - 0s - loss: 0.0028 - accuracy: 1.0000\n",
            "Epoch 298/300\n",
            "20/20 - 0s - loss: 0.0026 - accuracy: 1.0000\n",
            "Epoch 299/300\n",
            "20/20 - 0s - loss: 0.0030 - accuracy: 1.0000\n",
            "Epoch 300/300\n",
            "20/20 - 0s - loss: 0.0025 - accuracy: 1.0000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7fc461193908>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UcZ9Erkn9Z9h",
        "colab_type": "text"
      },
      "source": [
        "# Translation testing\n",
        "Since we are using only 20 sentences for training demonstration, this model is not expected to work well in arbitrary testing examples. In order to make sure that the model works, we will translate a training source sentence and compare the prediction and the target:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tb6DEeIuLSOt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def translate(model, source_sentence, target_sentence_start=[['<start>']]):\n",
        "  if np.ndim(source_sentence) == 1: # Create a batch of 1 the input is a sentence\n",
        "    source_sentence = [source_sentence]\n",
        "  if np.ndim(target_sentence_start) == 1:\n",
        "    target_sentence_start = [target_sentence_start]\n",
        "  # Tokenizing and padding\n",
        "  source_seq = source_tokenizer.texts_to_sequences(source_sentence)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, padding='post', maxlen=15)\n",
        "  predict_seq = target_tokenizer.texts_to_sequences(target_sentence_start)\n",
        "  \n",
        "  predict_sentence = list(target_sentence_start[0]) # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != '<end>' and len(predict_seq) < max_token_length:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    predict_label = tf.argmax(predict_output, axis=-1) # Pick the label with highest softmax score\n",
        "    predict_seq = tf.concat([predict_seq, predict_label], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(target_tokenizer.index_word[predict_label[0][0].numpy()])\n",
        "  \n",
        "  return predict_sentence"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TUJVgmDhRQLm",
        "colab_type": "code",
        "outputId": "6d4dd7e9-d432-4726-bca2-138dfcfd2953",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "print(\"Source sentence: \", source_sentences[10])\n",
        "print(\"Target sentence: \", target_sentences[10])\n",
        "print(\"Predicted sentence: \", ' '.join(translate(transformer, source_sentences[10].split(' '))))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Source sentence:  <start> Another coffee , please . <end>\n",
            "Target sentence:  <start> Otro cafe , por favor . <end>\n",
            "Predicted sentence:  <start> otro cafe , por favor . <end>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pkF-22akLStG",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "Not included code\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJSi_PJkts7r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#class LayerNormalization(Layer):\n",
        "#  def __init__(self, eps=1e-6, **kwargs):\n",
        "#    self.eps = eps\n",
        "#    super(LayerNormalization, self).__init__(**kwargs)\n",
        "#  def build(self, input_shape):\n",
        "#    self.gamma = self.add_weight(name='gamma', shape=input_shape[-1:],\n",
        "#\t\t\t\t\t\t\t\t\t initializer='ones', trainable=True)\n",
        "#    self.bias = self.add_weight(name='bias', shape=input_shape[-1:],\n",
        "#\t\t\t\t\t\t\t\t\tinitializer='zeros', trainable=True)\n",
        "#    super(LayerNormalization, self).build(input_shape)\n",
        "#  def call(self, x):\n",
        "#    mean = tf.math.reduce_mean(x, axis=-1, keepdims=True)\n",
        "#    std = tf.math.reduce_std(x, axis=-1, keepdims=True)\n",
        "#    return self.gamma * (x - mean) / (std + self.eps) + self.bias\n",
        "#  def compute_output_shape(self, input_shape):\n",
        "#    return input_shape"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}