main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab | Text Generation from Shakespeare's Sonnet\n",
    "\n",
    "This notebook explores the fascinating domain of text generation using a deep learning model trained on Shakespeare's sonnets. \n",
    "\n",
    "The objective is to create a neural network capable of generating text sequences that mimic the style and language of Shakespeare.\n",
    "\n",
    "By utilizing a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) layers, this project aims to demonstrate how a model can learn and replicate the complex patterns of early modern English. \n",
    "\n",
    "The dataset used consists of Shakespeare's sonnets, which are preprocessed and tokenized to serve as input for the model.\n",
    "\n",
    "Throughout this notebook, you will see the steps taken to prepare the data, build and train the model, and evaluate its performance in generating text. \n",
    "\n",
    "This lab provides a hands-on approach to understanding the intricacies of natural language processing (NLP) and the potential of machine learning in creative text generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "BOwsuGQQY9OL",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-08-16 22:09:18.737563: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout, Bidirectional\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras import regularizers\n",
    "import tensorflow.keras.utils as ku \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "url = 'https://raw.githubusercontent.com/martin-gorner/tensorflow-rnn-shakespeare/master/shakespeare/sonnets.txt'\n",
    "resp = requests.get(url)\n",
    "with open('sonnets.txt', 'wb') as f:\n",
    "    f.write(resp.content)\n",
    "\n",
    "data = open('sonnets.txt').read()\n",
    "\n",
    "corpus = data.lower().split(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 1: Initialise a tokenizer and fit it on the corpus variable using .fit_on_texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 2: Calculate the Vocabulary Size\n",
    "\n",
    "Let's figure out how many unique words are in your corpus. This will be the size of your vocabulary.\n",
    "\n",
    "Calculate the length of tokenizer.word_index, add 1 to it and store it in a variable called total_words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an empty list called input_sequences.\n",
    "\n",
    "For each sentence in your corpus, convert the text into a sequence of integers using the tokenizer.\n",
    "Then, generate n-gram sequences from these tokens.\n",
    "\n",
    "Store the result in the list input_sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the length of the longest sequence in input_sequences. Assign the result to a variable called max_sequence_len.\n",
    "\n",
    "Now pad the sequences using pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre').\n",
    "Convert it to a numpy array and assign the result back to our variable called input_sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare Predictors and Labels\n",
    "\n",
    "Split the sequences into two parts:\n",
    "\n",
    "- Predictors: All elements from input_sequences except the last one.\n",
    "- Labels: The last element of each sequence in input_sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "PRnDnCW-Z7qv",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One-Hot Encode the Labels :\n",
    "\n",
    "Convert the labels (which are integers) into one-hot encoded vectors. \n",
    "\n",
    "Ensure the length of these vectors matches the total number of unique words in your vocabulary.\n",
    "\n",
    "Use ku.to_categorical() on labels with num_classes = total_words\n",
    "\n",
    "Assign the result back to our variable labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize the Model\n",
    "\n",
    "Start by creating a Sequential model.\n",
    "\n",
    "Add Layers to the Model:\n",
    "\n",
    "Embedding Layer: The first layer is an embedding layer. It converts word indices into dense vectors of fixed size (100 in this case). Set the input length to the maximum sequence length minus one, which corresponds to the number of previous words the model will consider when predicting the next word.\n",
    "\n",
    "Bidirectional LSTM Layer: Add a Bidirectional LSTM layer with 150 units. This layer allows the model to learn context from both directions (past and future) in the sequence. return_sequences=True\n",
    "\n",
    "Dropout Layer: Add a dropout layer with a rate of 0.2 to prevent overfitting by randomly setting 20% of the input units to 0 during training.\n",
    "\n",
    "LSTM Layer: Add a second LSTM layer with 100 units. This layer processes the sequence and passes its output to the next layer.\n",
    "\n",
    "Dense Layer (Intermediate): Add a dense layer with half the total number of words as units, using ReLU activation. A regularization term (L2) is added to prevent overfitting.\n",
    "\n",
    "Dense Layer (Output): The final dense layer has as many units as there are words in the vocabulary, with a softmax activation function to output a probability distribution over all words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential([\n",
    "\n",
    "    # Your code here :\n",
    "    \n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compile the Model:\n",
    "\n",
    "Compile the model using categorical crossentropy as the loss function, the Adam optimizer for efficient training, and accuracy as the metric to evaluate during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Print Model Summary:\n",
    "\n",
    "Use model.summary() to print a summary of the model, which shows the layers, their output shapes, and the number of parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now train the model for 50 epochs and assign it to a variable called history.\n",
    "\n",
    "Training the model with 50 epochs should get you around 40% accuracy.\n",
    "\n",
    "You can train the model for as many epochs as you like depending on the time and computing constraints you are facing. Ideally train it for a larger amount of epochs than 50.\n",
    "\n",
    "That way you will get better text generation at the end.\n",
    "\n",
    "However, dont waste your time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "AIg2f1HBxqof",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use plt from matplotlib to plot the training accuracy over epochs and the loss over epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First you will have to get the accuracy and loss data over epochs, you can do this by using methods on your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "1fXTEO3GJ282",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate text with the model based on a seed text\n",
    "\n",
    "Now you will create two variables :\n",
    "\n",
    "- seed_text = 'Write the text you want the model to use as a starting point to generate the next words'\n",
    "- next_words = number_of_words_you_want_the_model_to_generate\n",
    "\n",
    "Please change number_of_words_you_want_the_model_to_generate by an actual integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a loop that runs based on the next_words variable and generates new text based on your seed_text input string. Print the full text with the generated text at the end.\n",
    "\n",
    "This time you dont get detailed instructions.\n",
    "\n",
    "Have fun!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "6Vc6PHgxa6Hm",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiment with at least 3 different seed_text strings and see what happens!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here :"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "NLP_Week4_Exercise_Shakespeare_Answer.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}