C3W4_Assignment.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bFWbEb6uGbN-"
      },
      "source": [
        "# Week 4: Predicting the next word\n",
        "\n",
        "Welcome to this assignment! During this week you saw how to create a model that will predict the next word in a text sequence, now you will implement such model and train it using a corpus of Shakespeare's sonnets, while also creating some helper functions to pre-process the data.\n",
        "\n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cp4A-ZBwSN11"
      },
      "source": [
        "_**NOTE:** To prevent errors from the autograder, please avoid editing or deleting non-graded cells in this notebook . Please only put your solutions in between the `### START CODE HERE` and `### END CODE HERE` code comments, and also refrain from adding any new cells._"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "BOwsuGQQY9OL",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTxqlHqKHzhr"
      },
      "source": [
        "For this assignment you will be using the [Shakespeare Sonnets Dataset](https://www.opensourceshakespeare.org/views/sonnets/sonnet_view.php?range=viewrange&sonnetrange1=1&sonnetrange2=154), which contains more than 2000 lines of text extracted from Shakespeare's sonnets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cfB-qcdIsUoL",
        "outputId": "07b48d6c-f82a-451c-9b8b-6cdeb39ad600"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: gdown==5.1.0 in /usr/local/lib/python3.10/dist-packages (5.1.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown==5.1.0) (4.12.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from gdown==5.1.0) (3.15.1)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.10/dist-packages (from gdown==5.1.0) (2.31.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from gdown==5.1.0) (4.66.4)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown==5.1.0) (2.5)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown==5.1.0) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown==5.1.0) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown==5.1.0) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown==5.1.0) (2024.6.2)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown==5.1.0) (1.7.1)\n"
          ]
        }
      ],
      "source": [
        "!pip install gdown==5.1.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "WZ4qOUzujMP6",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c4b2393e-e6ae-4cbb-bafb-10931333a4d4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gdown/__main__.py:132: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.\n",
            "  warnings.warn(\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=108jAePKK4R3BVYBbYJZ32JWUwxeMg20K\n",
            "To: /content/sonnets.txt\n",
            "100% 93.6k/93.6k [00:00<00:00, 51.3MB/s]\n"
          ]
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# sonnets.txt\n",
        "!gdown --id 108jAePKK4R3BVYBbYJZ32JWUwxeMg20K"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "Pfd-nYKij5yY",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc29c71e-470a-4c71-ec24-c09a8b8121e1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 2159 lines of sonnets\n",
            "\n",
            "The first 5 lines look like this:\n",
            "\n",
            "from fairest creatures we desire increase,\n",
            "that thereby beauty's rose might never die,\n",
            "but as the riper should by time decease,\n",
            "his tender heir might bear his memory:\n",
            "but thou, contracted to thine own bright eyes,\n"
          ]
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Define path for file with sonnets\n",
        "SONNETS_FILE = './sonnets.txt'\n",
        "\n",
        "# Read the data\n",
        "with open('./sonnets.txt') as f:\n",
        "    data = f.read()\n",
        "\n",
        "# Convert to lower case and save as a list\n",
        "corpus = data.lower().split(\"\\n\")\n",
        "\n",
        "print(f\"There are {len(corpus)} lines of sonnets\\n\")\n",
        "print(f\"The first 5 lines look like this:\\n\")\n",
        "for i in range(5):\n",
        "  print(corpus[i])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "imB15zrSNhA1"
      },
      "source": [
        "## Tokenizing the text\n",
        "\n",
        "Now fit the Tokenizer to the corpus and save the total number of words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "AAhM_qAZk0o5",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(corpus)\n",
        "total_words = len(tokenizer.word_index) + 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77-0sA46OETa"
      },
      "source": [
        "When converting the text into sequences you can use the `texts_to_sequences` method as you have done throughout this course.\n",
        "\n",
        "In the next graded function you will need to process this corpus one line at a time. Given this, it is important to keep in mind that the way you are feeding the data unto this method affects the result. Check the following example to make this clearer.\n",
        "\n",
        "The first example of the corpus is a string and looks like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "tqhPxdeXlfjh",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "6b0b0587-9e19-4ec7-b5c4-a7eae5fa3330"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'from fairest creatures we desire increase,'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "corpus[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFMP4z11O3os"
      },
      "source": [
        "If you pass this text directly into the `texts_to_sequences` method you will get an unexpected result:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "EMSEhmbzNZCE",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8cb2b239-aa74-42a4-9200-dc14ed4b873a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[],\n",
              " [],\n",
              " [58],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [17],\n",
              " [6],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [17],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [6],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [6],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [17],\n",
              " [],\n",
              " [],\n",
              " []]"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "tokenizer.texts_to_sequences(corpus[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pPZmZtpEPEeI"
      },
      "source": [
        "This happened because `texts_to_sequences` expects a list and you are providing a string. However a string is still and `iterable` in Python so you will get the word index of every character in the string.\n",
        "\n",
        "Instead you need to place the example whithin a list before passing it to the method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "Qmgo-vXhk4nd",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4af18d63-8eae-482c-91d7-9f9c3aed57f7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[34, 417, 877, 166, 213, 517]]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "tokenizer.texts_to_sequences([corpus[0]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0DU7wK-eQ5dc"
      },
      "source": [
        "Notice that you received the sequence wrapped inside a list so in order to get only the desired sequence you need to explicitly get the first item in the list like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "kpTy8WmIQ57P",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "199d249d-9394-4808-fd67-2132cc586a5a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[34, 417, 877, 166, 213, 517]"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "tokenizer.texts_to_sequences([corpus[0]])[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-oqy9KjXRJ9A"
      },
      "source": [
        "## Generating n_grams\n",
        "\n",
        "Now complete the `n_gram_seqs` function below. This function receives the fitted tokenizer and the corpus (which is a list of strings) and should return a list containing the `n_gram` sequences for each line in the corpus:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "deletable": false,
        "id": "iy4baJMDl6kj",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# GRADED FUNCTION: n_gram_seqs\n",
        "def n_gram_seqs(corpus, tokenizer):\n",
        "    \"\"\"\n",
        "    Generates a list of n-gram sequences\n",
        "\n",
        "    Args:\n",
        "        corpus (list of string): lines of texts to generate n-grams for\n",
        "        tokenizer (object): an instance of the Tokenizer class containing the word-index dictionary\n",
        "\n",
        "    Returns:\n",
        "        input_sequences (list of int): the n-gram sequences for each line in the corpus\n",
        "    \"\"\"\n",
        "    input_sequences = []\n",
        "\n",
        "    ### START CODE HERE\n",
        "    # Loop over every line\n",
        "    for line in corpus:\n",
        "      # Tokenize the current line\n",
        "      token_list = tokenizer.texts_to_sequences([line])[0]\n",
        "\n",
        "      # Loop over the line several times to generate the subphrases\n",
        "      for i in range(1, len(token_list)):\n",
        "        # Generate the subphrase\n",
        "        n_gram_sequence = token_list[:i+1]\n",
        "        # Append the subphrase to the sequences list\n",
        "        input_sequences.append(n_gram_sequence)\n",
        "    ### END CODE HERE\n",
        "\n",
        "    return input_sequences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "DlKqW2pfM7G3",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83705e3f-9bb8-4e86-abef-a118cbdc0a3c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "n_gram sequences for first example look like this:\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[34, 417],\n",
              " [34, 417, 877],\n",
              " [34, 417, 877, 166],\n",
              " [34, 417, 877, 166, 213],\n",
              " [34, 417, 877, 166, 213, 517]]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Test your function with one example\n",
        "first_example_sequence = n_gram_seqs([corpus[0]], tokenizer)\n",
        "\n",
        "print(\"n_gram sequences for first example look like this:\\n\")\n",
        "first_example_sequence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0HL8Ug6UU0Jt"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "n_gram sequences for first example look like this:\n",
        "\n",
        "[[34, 417],\n",
        " [34, 417, 877],\n",
        " [34, 417, 877, 166],\n",
        " [34, 417, 877, 166, 213],\n",
        " [34, 417, 877, 166, 213, 517]]\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "wtPpCcBjNc4c",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "96062582-4356-4530-910c-9a1c9db00258"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "n_gram sequences for next 3 examples look like this:\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[8, 878],\n",
              " [8, 878, 134],\n",
              " [8, 878, 134, 351],\n",
              " [8, 878, 134, 351, 102],\n",
              " [8, 878, 134, 351, 102, 156],\n",
              " [8, 878, 134, 351, 102, 156, 199],\n",
              " [16, 22],\n",
              " [16, 22, 2],\n",
              " [16, 22, 2, 879],\n",
              " [16, 22, 2, 879, 61],\n",
              " [16, 22, 2, 879, 61, 30],\n",
              " [16, 22, 2, 879, 61, 30, 48],\n",
              " [16, 22, 2, 879, 61, 30, 48, 634],\n",
              " [25, 311],\n",
              " [25, 311, 635],\n",
              " [25, 311, 635, 102],\n",
              " [25, 311, 635, 102, 200],\n",
              " [25, 311, 635, 102, 200, 25],\n",
              " [25, 311, 635, 102, 200, 25, 278]]"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Test your function with a bigger corpus\n",
        "next_3_examples_sequence = n_gram_seqs(corpus[1:4], tokenizer)\n",
        "\n",
        "print(\"n_gram sequences for next 3 examples look like this:\\n\")\n",
        "next_3_examples_sequence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EIzecMczU9UB"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "n_gram sequences for next 3 examples look like this:\n",
        "\n",
        "[[8, 878],\n",
        " [8, 878, 134],\n",
        " [8, 878, 134, 351],\n",
        " [8, 878, 134, 351, 102],\n",
        " [8, 878, 134, 351, 102, 156],\n",
        " [8, 878, 134, 351, 102, 156, 199],\n",
        " [16, 22],\n",
        " [16, 22, 2],\n",
        " [16, 22, 2, 879],\n",
        " [16, 22, 2, 879, 61],\n",
        " [16, 22, 2, 879, 61, 30],\n",
        " [16, 22, 2, 879, 61, 30, 48],\n",
        " [16, 22, 2, 879, 61, 30, 48, 634],\n",
        " [25, 311],\n",
        " [25, 311, 635],\n",
        " [25, 311, 635, 102],\n",
        " [25, 311, 635, 102, 200],\n",
        " [25, 311, 635, 102, 200, 25],\n",
        " [25, 311, 635, 102, 200, 25, 278]]\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dx3V_RjFWQSu"
      },
      "source": [
        "Apply the `n_gram_seqs` transformation to the whole corpus and save the maximum sequence length to use it later:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "laMwiRUpmuSd",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08ac72a9-7b06-4bca-e7b7-f55f9a32d4c0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "n_grams of input_sequences have length: 15462\n",
            "maximum length of sequences is: 11\n"
          ]
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Apply the n_gram_seqs transformation to the whole corpus\n",
        "input_sequences = n_gram_seqs(corpus, tokenizer)\n",
        "\n",
        "# Save max length\n",
        "max_sequence_len = max([len(x) for x in input_sequences])\n",
        "\n",
        "print(f\"n_grams of input_sequences have length: {len(input_sequences)}\")\n",
        "print(f\"maximum length of sequences is: {max_sequence_len}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2OciMdmEdE9L"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "n_grams of input_sequences have length: 15462\n",
        "maximum length of sequences is: 11\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zHY7HroqWq12"
      },
      "source": [
        "## Add padding to the sequences\n",
        "\n",
        "Now code the `pad_seqs` function which will pad any given sequences to the desired maximum length. Notice that this function receives a list of sequences and should return a numpy array with the padded sequences:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "cellView": "code",
        "deletable": false,
        "id": "WW1-qAZaWOhC",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# GRADED FUNCTION: pad_seqs\n",
        "def pad_seqs(input_sequences, maxlen):\n",
        "    \"\"\"\n",
        "    Pads tokenized sequences to the same length\n",
        "\n",
        "    Args:\n",
        "        input_sequences (list of int): tokenized sequences to pad\n",
        "        maxlen (int): maximum length of the token sequences\n",
        "\n",
        "    Returns:\n",
        "        padded_sequences (array of int): tokenized sequences padded to the same length\n",
        "    \"\"\"\n",
        "    ### START CODE HERE\n",
        "    padded_sequences = pad_sequences(input_sequences, maxlen=maxlen, padding='pre')\n",
        "\n",
        "    return padded_sequences\n",
        "    ### END CODE HERE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "IqVQ0pb3YHLr",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6a5b09fb-edbf-459d-ac5b-09b5ea212539"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0,   0,  34, 417],\n",
              "       [  0,   0,   0,  34, 417, 877],\n",
              "       [  0,   0,  34, 417, 877, 166],\n",
              "       [  0,  34, 417, 877, 166, 213],\n",
              "       [ 34, 417, 877, 166, 213, 517]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Test your function with the n_grams_seq of the first example\n",
        "first_padded_seq = pad_seqs(first_example_sequence, max([len(x) for x in first_example_sequence]))\n",
        "first_padded_seq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Re_avDznXRnU"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "array([[  0,   0,   0,   0,  34, 417],\n",
        "       [  0,   0,   0,  34, 417, 877],\n",
        "       [  0,   0,  34, 417, 877, 166],\n",
        "       [  0,  34, 417, 877, 166, 213],\n",
        "       [ 34, 417, 877, 166, 213, 517]], dtype=int32)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "j56_UCOBYzZt",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6f256875-d413-4a99-b9d7-ac1c09359350"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0,   0,   0,   0,   8, 878],\n",
              "       [  0,   0,   0,   0,   0,   8, 878, 134],\n",
              "       [  0,   0,   0,   0,   8, 878, 134, 351],\n",
              "       [  0,   0,   0,   8, 878, 134, 351, 102],\n",
              "       [  0,   0,   8, 878, 134, 351, 102, 156],\n",
              "       [  0,   8, 878, 134, 351, 102, 156, 199],\n",
              "       [  0,   0,   0,   0,   0,   0,  16,  22],\n",
              "       [  0,   0,   0,   0,   0,  16,  22,   2],\n",
              "       [  0,   0,   0,   0,  16,  22,   2, 879],\n",
              "       [  0,   0,   0,  16,  22,   2, 879,  61],\n",
              "       [  0,   0,  16,  22,   2, 879,  61,  30],\n",
              "       [  0,  16,  22,   2, 879,  61,  30,  48],\n",
              "       [ 16,  22,   2, 879,  61,  30,  48, 634],\n",
              "       [  0,   0,   0,   0,   0,   0,  25, 311],\n",
              "       [  0,   0,   0,   0,   0,  25, 311, 635],\n",
              "       [  0,   0,   0,   0,  25, 311, 635, 102],\n",
              "       [  0,   0,   0,  25, 311, 635, 102, 200],\n",
              "       [  0,   0,  25, 311, 635, 102, 200,  25],\n",
              "       [  0,  25, 311, 635, 102, 200,  25, 278]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Test your function with the n_grams_seq of the next 3 examples\n",
        "next_3_padded_seq = pad_seqs(next_3_examples_sequence, max([len(s) for s in next_3_examples_sequence]))\n",
        "next_3_padded_seq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3rmcDluOXcIU"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "array([[  0,   0,   0,   0,   0,   0,   8, 878],\n",
        "       [  0,   0,   0,   0,   0,   8, 878, 134],\n",
        "       [  0,   0,   0,   0,   8, 878, 134, 351],\n",
        "       [  0,   0,   0,   8, 878, 134, 351, 102],\n",
        "       [  0,   0,   8, 878, 134, 351, 102, 156],\n",
        "       [  0,   8, 878, 134, 351, 102, 156, 199],\n",
        "       [  0,   0,   0,   0,   0,   0,  16,  22],\n",
        "       [  0,   0,   0,   0,   0,  16,  22,   2],\n",
        "       [  0,   0,   0,   0,  16,  22,   2, 879],\n",
        "       [  0,   0,   0,  16,  22,   2, 879,  61],\n",
        "       [  0,   0,  16,  22,   2, 879,  61,  30],\n",
        "       [  0,  16,  22,   2, 879,  61,  30,  48],\n",
        "       [ 16,  22,   2, 879,  61,  30,  48, 634],\n",
        "       [  0,   0,   0,   0,   0,   0,  25, 311],\n",
        "       [  0,   0,   0,   0,   0,  25, 311, 635],\n",
        "       [  0,   0,   0,   0,  25, 311, 635, 102],\n",
        "       [  0,   0,   0,  25, 311, 635, 102, 200],\n",
        "       [  0,   0,  25, 311, 635, 102, 200,  25],\n",
        "       [  0,  25, 311, 635, 102, 200,  25, 278]], dtype=int32)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "rgK-Q_micEYA",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "229cbb1e-0df1-4094-9cd8-14af5a22a586"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "padded corpus has shape: (15462, 11)\n"
          ]
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Pad the whole corpus\n",
        "input_sequences = pad_seqs(input_sequences, max_sequence_len)\n",
        "\n",
        "print(f\"padded corpus has shape: {input_sequences.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59RD1YYNc7CW"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "padded corpus has shape: (15462, 11)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbOidyPrXxf7"
      },
      "source": [
        "## Split the data into features and labels\n",
        "\n",
        "Before feeding the data into the neural network you should split it into features and labels. In this case the features will be the padded n_gram sequences with the last word removed from them and the labels will be the removed word.\n",
        "\n",
        "Complete the `features_and_labels` function below. This function expects the padded n_gram sequences as input and should return a tuple containing the features and the one hot encoded labels.\n",
        "\n",
        "Notice that the function also receives the total of words in the corpus, this parameter will be very important when one hot enconding the labels since every word in the corpus will be a label at least once. If you need a refresh of how the `to_categorical` function works take a look at the [docs](https://www.tensorflow.org/api_docs/python/tf/keras/utils/to_categorical)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "cellView": "code",
        "deletable": false,
        "id": "9WGGbYdnZdmJ",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# GRADED FUNCTION: features_and_labels\n",
        "def features_and_labels(input_sequences, total_words):\n",
        "    \"\"\"\n",
        "    Generates features and labels from n-grams\n",
        "\n",
        "    Args:\n",
        "        input_sequences (list of int): sequences to split features and labels from\n",
        "        total_words (int): vocabulary size\n",
        "\n",
        "    Returns:\n",
        "        features, one_hot_labels (array of int, array of int): arrays of features and one-hot encoded labels\n",
        "    \"\"\"\n",
        "    ### START CODE HERE\n",
        "    features, labels = input_sequences[:,:-1], input_sequences[:,-1]\n",
        "    one_hot_labels = to_categorical(labels, num_classes=total_words)\n",
        "    ### END CODE HERE\n",
        "\n",
        "    return features, one_hot_labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "23DolaBRaIAZ",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7bd44eee-028b-4fde-ab93-3a0d77d49658"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "labels have shape: (5, 3211)\n",
            "\n",
            "features look like this:\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0,   0,  34],\n",
              "       [  0,   0,   0,  34, 417],\n",
              "       [  0,   0,  34, 417, 877],\n",
              "       [  0,  34, 417, 877, 166],\n",
              "       [ 34, 417, 877, 166, 213]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Test your function with the padded n_grams_seq of the first example\n",
        "first_features, first_labels = features_and_labels(first_padded_seq, total_words)\n",
        "\n",
        "print(f\"labels have shape: {first_labels.shape}\")\n",
        "print(\"\\nfeatures look like this:\\n\")\n",
        "first_features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7t4yAx2UaQ43"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "labels have shape: (5, 3211)\n",
        "\n",
        "features look like this:\n",
        "\n",
        "array([[  0,   0,   0,   0,  34],\n",
        "       [  0,   0,   0,  34, 417],\n",
        "       [  0,   0,  34, 417, 877],\n",
        "       [  0,  34, 417, 877, 166],\n",
        "       [ 34, 417, 877, 166, 213]], dtype=int32)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "deletable": false,
        "editable": false,
        "id": "GRTuLEt3bRKa",
        "tags": [
          "graded"
        ],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d519c3a4-88a8-4f97-b7d8-928a60bfeed6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "features have shape: (15462, 10)\n",
            "labels have shape: (15462, 3211)\n"
          ]
        }
      ],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# Split the whole corpus\n",
        "features, labels = features_and_labels(input_sequences, total_words)\n",
        "\n",
        "print(f\"features have shape: {features.shape}\")\n",
        "print(f\"labels have shape: {labels.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xXSMK_HpdLns"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "features have shape: (15462, 10)\n",
        "labels have shape: (15462, 3211)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltxaOCE_aU6J"
      },
      "source": [
        "## Create the model\n",
        "\n",
        "Now you should define a model architecture capable of achieving an accuracy of at least 80%.\n",
        "\n",
        "Some hints to help you in this task:\n",
        "\n",
        "- An appropriate `output_dim` for the first layer (Embedding) is 100, this is already provided for you.\n",
        "- A Bidirectional LSTM is helpful for this particular problem.\n",
        "- The last layer should have the same number of units as the total number of words in the corpus and a softmax activation function.\n",
        "- This problem can be solved with only two layers (excluding the Embedding) so try out small architectures first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "cellView": "code",
        "deletable": false,
        "id": "XrE6kpJFfvRY",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# grader-required-cell\n",
        "\n",
        "# GRADED FUNCTION: create_model\n",
        "def create_model(total_words, max_sequence_len):\n",
        "    \"\"\"\n",
        "    Creates a text generator model\n",
        "\n",
        "    Args:\n",
        "        total_words (int): size of the vocabulary for the Embedding layer input\n",
        "        max_sequence_len (int): length of the input sequences\n",
        "\n",
        "    Returns:\n",
        "        model (tf.keras Model): the text generator model\n",
        "    \"\"\"\n",
        "    model = Sequential()\n",
        "    ### START CODE HERE\n",
        "    import tensorflow as tf\n",
        "    # Hyperparameters\n",
        "    embedding_dim = 100\n",
        "    lstm_units = 150\n",
        "    learning_rate = 0.01\n",
        "    model.add(Embedding(total_words, 100, input_length=max_sequence_len-1))\n",
        "    model.add(Bidirectional(LSTM(lstm_units)))\n",
        "    model.add(Dense(total_words, activation='softmax'))\n",
        "\n",
        "\n",
        "    # Compile the model\n",
        "    model.compile(\n",
        "        loss='categorical_crossentropy',\n",
        "        optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),\n",
        "        metrics=['accuracy']\n",
        "        )\n",
        "\n",
        "    ### END CODE HERE\n",
        "\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0IpX_Gu_gISk",
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e88bd30f-44ed-4c4f-b1aa-8b44d0219166"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/50\n",
            "484/484 [==============================] - 37s 65ms/step - loss: 6.8117 - accuracy: 0.0308\n",
            "Epoch 2/50\n",
            "484/484 [==============================] - 32s 66ms/step - loss: 6.1299 - accuracy: 0.0604\n",
            "Epoch 3/50\n",
            "484/484 [==============================] - 33s 68ms/step - loss: 5.4228 - accuracy: 0.0845\n",
            "Epoch 4/50\n",
            "484/484 [==============================] - 32s 65ms/step - loss: 4.6225 - accuracy: 0.1371\n",
            "Epoch 5/50\n",
            "484/484 [==============================] - 33s 68ms/step - loss: 3.7882 - accuracy: 0.2217\n",
            "Epoch 6/50\n",
            "484/484 [==============================] - 32s 66ms/step - loss: 3.0756 - accuracy: 0.3329\n",
            "Epoch 7/50\n",
            "484/484 [==============================] - 33s 69ms/step - loss: 2.5046 - accuracy: 0.4277\n",
            "Epoch 8/50\n",
            "484/484 [==============================] - 31s 64ms/step - loss: 2.0583 - accuracy: 0.5233\n",
            "Epoch 9/50\n",
            "484/484 [==============================] - 33s 68ms/step - loss: 1.8012 - accuracy: 0.5701\n",
            "Epoch 10/50\n",
            "419/484 [========================>.....] - ETA: 4s - loss: 1.5603 - accuracy: 0.6273"
          ]
        }
      ],
      "source": [
        "# Get the untrained model\n",
        "model = create_model(total_words, max_sequence_len)\n",
        "\n",
        "# Train the model\n",
        "history = model.fit(features, labels, epochs=50, verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gy72RPgly55q"
      },
      "source": [
        "**To pass this assignment, your model should achieve a training accuracy of at least 80%**. If your model didn't achieve this threshold, try training again with a different model architecture, consider increasing the number of unit in your `LSTM` layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1fXTEO3GJ282",
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Take a look at the training curves of your model\n",
        "\n",
        "acc = history.history['accuracy']\n",
        "loss = history.history['loss']\n",
        "\n",
        "epochs = range(len(acc))\n",
        "\n",
        "plt.plot(epochs, acc, 'b', label='Training accuracy')\n",
        "plt.title('Training accuracy')\n",
        "\n",
        "plt.figure()\n",
        "\n",
        "plt.plot(epochs, loss, 'b', label='Training Loss')\n",
        "plt.title('Training loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OjvED5A3qrn2"
      },
      "source": [
        "Before closing the assignment, be sure to also download the `history.pkl` file which contains the information of the training history of your model and will be used to compute your grade. You can download this file by running the cell below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9QRG73l6qE-c",
        "tags": []
      },
      "outputs": [],
      "source": [
        "def download_history():\n",
        "  import pickle\n",
        "  from google.colab import files\n",
        "\n",
        "  with open('history.pkl', 'wb') as f:\n",
        "    pickle.dump(history.history, f)\n",
        "\n",
        "  files.download('history.pkl')\n",
        "\n",
        "download_history()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wdsMszk9zBs_"
      },
      "source": [
        "## See your model in action\n",
        "\n",
        "After all your work it is finally time to see your model generating text.\n",
        "\n",
        "Run the cell below to generate the next 100 words of a seed text.\n",
        "\n",
        "After submitting your assignment you are encouraged to try out training for different amounts of epochs and seeing how this affects the coherency of the generated text. Also try changing the seed text to see what you get!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Vc6PHgxa6Hm",
        "tags": []
      },
      "outputs": [],
      "source": [
        "seed_text = \"Help me Obi Wan Kenobi, you're my only hope\"\n",
        "next_words = 100\n",
        "\n",
        "for _ in range(next_words):\n",
        "    # Convert the text into sequences\n",
        "    token_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "    # Pad the sequences\n",
        "    token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "    # Get the probabilities of predicting a word\n",
        "    predicted = model.predict(token_list, verbose=0)\n",
        "    # Choose the next word based on the maximum probability\n",
        "    predicted = np.argmax(predicted, axis=-1).item()\n",
        "    # Get the actual word from the word index\n",
        "    output_word = tokenizer.index_word[predicted]\n",
        "    # Append to the current text\n",
        "    seed_text += \" \" + output_word\n",
        "\n",
        "print(seed_text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQVDbdcYqSux"
      },
      "source": [
        "## Download your notebook for grading\n",
        "\n",
        "Along with the `history.pkl` file earlier, you will also need to submit your solution notebook for grading. The following code cells will check if this notebook's grader metadata (i.e. hidden data in the notebook needed for grading) is not modified by your workspace. This will ensure that the autograder can evaluate your code properly. Depending on its output, you will either:\n",
        "\n",
        "* *if the metadata is intact*: Download the current notebook. Click on the File tab on the upper left corner of the screen then click on `Download -> Download .ipynb.` You can name it anything you want as long as it is a valid `.ipynb` (jupyter notebook) file.\n",
        "<br>\n",
        "\n",
        "* *if the metadata is missing*: A new notebook with your solutions will be created on this Colab workspace. It should be downloaded automatically and you can submit that to the grader."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZUEiIXZEShHw"
      },
      "outputs": [],
      "source": [
        "# Download metadata checker\n",
        "!wget -nc https://storage.googleapis.com/tensorflow-1-public/colab_metadata_checker.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ek87UXX7Sj6b"
      },
      "outputs": [],
      "source": [
        "import colab_metadata_checker\n",
        "\n",
        "# Please see the output of this cell to see which file you need to submit to the grader\n",
        "colab_metadata_checker.run('C3W4_Assignment_fixed.ipynb')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68Vbslc7sUoT"
      },
      "source": [
        "**Please disregard the following note if the notebook metadata is detected**\n",
        "\n",
        "_Note: Just in case the automatic download fails when the metadata is missing, you can also do these steps:_\n",
        "* _Click the Folder icon on the left side of this screen to open the File Manager._\n",
        "* _Click the Folder Refresh icon in the File Manager to see the latest files in the workspace. You should see a file ending with a `_fixed.ipynb`._\n",
        "* _Right-click on that file to save locally and submit it to the grader._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6r-X-HXtSc8N"
      },
      "source": [
        "**Congratulations on finishing this week's assignment!**\n",
        "\n",
        "You have successfully implemented a neural network capable of predicting the next word in a sequence of text!\n",
        "\n",
        "**We hope to see you in the next course of the specialization! Keep it up!**"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "dlai_version": "1.2.0",
    "jupytext": {
      "main_language": "python"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}