language-model-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "REDeZr0hSeCJ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Assignment #4\n",
    "**CE4719: Deep Learing**\n",
    "\n",
    "\n",
    "*   Spring 2020\n",
    "*   http://ce.sharif.edu/courses/98-99/2/ce719-1/index.php\n",
    "\n",
    "**Please pay attention to these notes:**\n",
    "- the coding parts you have to complete are specified by:\n",
    "```\n",
    "    ################################################################################\n",
    "    # TODO:                                                                        #\n",
    "    ################################################################################\n",
    "    pass\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################ \n",
    "```\n",
    "- We always recommend discussion in groups for assignments. However, each student has to finish all of the questions by him/herself. \n",
    "- All submitted code will be compared against all student's codes using Stanford MOSS.\n",
    "- If you have any questions about this assignment, feel free to drop us a line. You may also post your questions on the course's forum page.\n",
    "- We HIGHLY encourage you to run this notebook on Google Colab.\n",
    "- **Before starting to work on the assignment, please fill your name in the next section AND Remember to RUN the cell.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "mk4Bs7vnsfvG",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "#@title Enter your information & \"RUN the cell!!\"\n",
    "student_id = \"98208824\" #@param {type:\"string\"}\n",
    "student_name = \"Sadroddin Barikbin\" #@param {type:\"string\"}\n",
    "\n",
    "print(\"your student id:\", student_id)\n",
    "print(\"your name:\", student_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "XFWqU5noQjQC",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## 2. Tokenization, Vocabulary, Preprocessing (15 pts)\n",
    "\n",
    "---\n",
    "In this problem, you will practice tokenization, creating vocabulary for a corpus, preprocessing data, and processing data using RNNs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "hu4Yi7q8XwIX",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 2.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6174,
     "status": "ok",
     "timestamp": 1587895793516,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "sxtV0x6rXsNJ",
    "new_sheet": false,
    "outputId": "37707da8-28d8-46f4-8fca-57ebafbb7727",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torch.nn.init as init\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from torch.nn import RNN, RNNCell, Embedding\n",
    "from typing import List, Dict\n",
    "from itertools import chain\n",
    "from collections import Counter\n",
    "import json\n",
    "from pprint import pprint\n",
    "\n",
    "# nltk will be used to tokenize texts\n",
    "import nltk  \n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1431,
     "status": "ok",
     "timestamp": 1587902699324,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "tn5IAVDGhG-f",
    "new_sheet": false,
    "outputId": "13dbad4b-9bb0-4b84-96c7-e226dbc78fdd",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    }
   ],
   "source": [
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(DEVICE)\n",
    "\n",
    "# Function for setting the random seed for reproducibility\n",
    "def set_seed(seed):\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    if DEVICE == torch.device(\"cuda\"):\n",
    "        torch.cuda.manual_seed(seed)\n",
    "        torch.backends.cudnn.deterministic = True\n",
    "        torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "z1yhrX8gXy-E",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 2.2 Tokenize (1 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "ZJEQ9xt6P8AY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Tokenization is the process of tokenizing or splitting a string or text into a list of tokens. Tokenization is one of the common pre-processings in natural language processing. The resulting tokens are then passed on to some other form of processing, which in our case will be deep neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "NaD8qSkxXuiW",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from nltk import word_tokenize\n",
    "\n",
    "\n",
    "def tokenize(corpus: List[str]) -> List[List[str]]:\n",
    "    \"\"\" tokenizes the corpus and returns it as a list of list of tokens\n",
    "    corpus: Input corpus as a list of sentences (each sentence is a string)\n",
    "    \"\"\"\n",
    "    ################################################################################\n",
    "    # TODO: use nltk word_tokenize to tokenize corpus, as a list of sentences,     #\n",
    "    # into its constituent words.                                                  #\n",
    "    # You should first lowercase the characters of sentences (use .lower()         #\n",
    "    # methods of strings.)                                                         #\n",
    "    ################################################################################\n",
    "    return list(map(lambda sent: word_tokenize(sent.lower()),corpus))\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "OqYxSgEdbgOd",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's test your implementation with the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1396,
     "status": "ok",
     "timestamp": 1587896424161,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "zl31abucSYR5",
    "new_sheet": false,
    "outputId": "9a425849-cdcd-4dca-cb4b-514687a7bffb",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_tokenize():\n",
    "    sample_corpus = [\"From fairest creatures we desire increase.\",\n",
    "                     \"Within thine own bud buriest thy content.\",\n",
    "                     \"Thy youth's proud livery so gazed on now.\", \n",
    "                     \"Shall sum my count, and make my old excuse.\"]\n",
    "\n",
    "    correct_answer = [['from', 'fairest', 'creatures', 'we', 'desire', 'increase', '.'],\n",
    "                      ['within', 'thine', 'own', 'bud', 'buriest', 'thy', 'content', '.'],\n",
    "                      ['thy', 'youth', \"'s\", 'proud', 'livery', 'so', 'gazed', 'on', 'now', '.'],\n",
    "                      ['shall', 'sum', 'my', 'count', ',', 'and', 'make', 'my', 'old', 'excuse', '.']]\n",
    "\n",
    "    assert tokenize(sample_corpus) == correct_answer\n",
    "\n",
    "    print('passed!')\n",
    "\n",
    "test_tokenize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "n7MdJpsagYpK",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 2.3 Vocabulary (6 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "hhiS8HnNQfmB",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After tokenizing the corpus, we will find the unique tokens and name them the vocabulary of that corpus. There are a couple of important points to note here:\n",
    "\n",
    "1. When dealing with a corpus in sentence-wise manner, we usually mark the beginning and end of sentences with some special tokens (e.g., `'<START>'` and `'<END>'`).\n",
    "\n",
    "2. To make sequences have the same length, we pad shorter ones with a special token (e.g., `'<PAD>'`).\n",
    "\n",
    "3. Tokens which will be encountered later and do not exist in our vocab will be replaced by a special token (e.g., `'<UNK>'`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "XCvF_KrVX294",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "class Vocab:\n",
    "    def __init__(self, word2id=None):\n",
    "        \"\"\"Constructor of Vocab\n",
    "\n",
    "        word2id: dictionary that maps tokens to their ids.\n",
    "        \"\"\"\n",
    "        self.pad_token = '<PAD>'\n",
    "        self.end_token = '<END>'\n",
    "        self.start_token = '<START>'\n",
    "        self.unk_token = '<UNK>'\n",
    "\n",
    "        if word2id is None:\n",
    "            self.word2id = {self.pad_token: 0,\n",
    "                            self.start_token: 1,\n",
    "                            self.end_token: 2,\n",
    "                            self.unk_token: 3}\n",
    "            self.size = 4\n",
    "        else:\n",
    "            self.word2id = word2id\n",
    "            self.size = len(self.word2id)\n",
    "\n",
    "        self.id2word = {v: k for (k, v) in self.word2id.items()}\n",
    "\n",
    "    def build(self, tokenized_corpus: List[List[str]], size=None, min_freq=None):\n",
    "        \"\"\"Builds the vocab from a tokenized corpus.\n",
    "\n",
    "        tokenized_corpus: corpus as a list of list of tokens (strings)\n",
    "        size: Final size of (number of unique tokens in) our vocab\n",
    "        min_freq: minimum frequency\n",
    "        \"\"\"\n",
    "        tokens2freq = Counter(chain(*tokenized_corpus))  # dict that maps unique tokens to their freqs in the corpus\n",
    "        frequent_tokens = []\n",
    "        ################################################################################\n",
    "        # TODO: use tokens2freq and find the first size frequent tokens and save       #\n",
    "        #       them in frequent_tokens. Remove tokens with a frequency lower than     #\n",
    "        #       min_freq (i.e. if token's occurence in the corpus is less than         # \n",
    "        #       min_freq times, don't put the token in frequent_tokens).               #\n",
    "        #       If size is None, then use all of the tokens. This also applies to      #\n",
    "        #       min_freq.                                                              #\n",
    "        ################################################################################\n",
    "        frequent_tokens=map(lambda x:x[0],filter(lambda x: not min_freq or x[1]>=min_freq,tokens2freq.most_common(size)))\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        # adding tokens to the vocab\n",
    "        for token in frequent_tokens:\n",
    "            self.add_token(token)\n",
    "\n",
    "    def get_token_by_id(self, t_id: int) -> str:\n",
    "        \"\"\"Returns the token with the corresponding id in the vocab.\n",
    "        If the id is not valid, returns None.\n",
    "\n",
    "        t_id: token id\n",
    "        \"\"\"\n",
    "        return self.id2word.get(t_id, None)\n",
    "\n",
    "    def get_id_by_token(self, token: str) -> int:\n",
    "        \"\"\"Returns the id of the token in the vocab. If the token does not exist,\n",
    "        returns the id of <UNK> token.\n",
    "\n",
    "        token: token (as a string) for which the id should be returned.\n",
    "        \"\"\"\n",
    "        return self.word2id.get(token, self.word2id[self.unk_token])\n",
    "\n",
    "    def add_token(self, token: str):\n",
    "        \"\"\"Adds the token to the vocab's data structures\n",
    "        token: token as a string\n",
    "        \"\"\"\n",
    "        ################################################################################\n",
    "        # TODO: If the token is not already in the vocab add it to word2id and id2word #\n",
    "        # Don't forget to update the vocab size afterwards!                            #\n",
    "        ################################################################################\n",
    "        if not token in self.word2id:\n",
    "          self.word2id[token]=self.size\n",
    "          self.id2word[self.size]=token\n",
    "          self.size+=1\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "    def tokens2ids(self, sents):\n",
    "        \"\"\"Convert list of words or list of sentences of tokens \n",
    "        into list or list of list of indices.\n",
    "\n",
    "        sents: input sentences as List[List[str]] (multiple sentences) or List[str]\n",
    "        (single sentence)\n",
    "        \"\"\"\n",
    "        ################################################################################\n",
    "        # TODO: return a new list where each token is repalced by its id.              #\n",
    "        # HINT: try to implement each part in one line of code using list comprehension#\n",
    "        ################################################################################\n",
    "        if type(sents[0]) == list:\n",
    "            return [[self.get_id_by_token(token) for token in sent] for sent in sents]\n",
    "        else:\n",
    "            return [self.get_id_by_token(token) for token in sents]\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "  \n",
    "    def to_tensor(self, sent: List[str]):\n",
    "        \"\"\"Converts a sentence as a list of tokens into a tensor of indices.\n",
    "\n",
    "        sent: a sentence as a list of strings (tokens)\n",
    "        \"\"\"\n",
    "        ################################################################################\n",
    "        # TODO: Use self.tokens2ids to get the sentence as a list of indices and wrap a#\n",
    "        #       tensor around it with dtpye of torch.long and device of DEVICE.        #\n",
    "        ################################################################################\n",
    "        return torch.tensor(self.tokens2ids(sent),dtype=torch.long,device=DEVICE)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "    def pad_sents(self, sents: List[List[str]]) -> List[List[str]]:\n",
    "        \"\"\"Pads list of sentences according to the longest sentence.\n",
    "\n",
    "        sents: sentences as a list of list of tokens (strings).\n",
    "        \"\"\"\n",
    "        sents_padded = []\n",
    "        ################################################################################\n",
    "        # TODO: pad shorter sentences by appending pad token to them.                  #\n",
    "        ################################################################################\n",
    "        maxx=max(map(lambda sent:len(sent),sents))\n",
    "        sents_padded=[sent+['<PAD>']*(maxx-len(sent)) for sent in sents]\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        return sents_padded\n",
    "\n",
    "    def save(self, path: str):\n",
    "        \"\"\"Saves the vocab in a json file.\n",
    "\n",
    "        path: path to save the vocab in\n",
    "        \"\"\"\n",
    "        with open(path, 'w') as f:\n",
    "            json.dump(self.word2id, f)\n",
    "\n",
    "    @staticmethod\n",
    "    def load(path: str):\n",
    "        \"\"\"Loads vocab from a json file.\n",
    "\n",
    "        path: path to load the vocab from\n",
    "        \"\"\"\n",
    "        with open(path, 'r') as f:\n",
    "            word2id = json.load(f)\n",
    "\n",
    "        return Vocab(word2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "1V5TDv5NlCbV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Implement `add_token` and `build` methods. Then, test your implementation with the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1619,
     "status": "ok",
     "timestamp": 1587898773799,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "UYPEzxgsks5z",
    "new_sheet": false,
    "outputId": "72d9f2d7-6d96-408a-890b-718bc1e700be",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_build_vocab():\n",
    "    vocab =  sample_corpus = [\"From fairest creatures we desire increase.\",\n",
    "                              \"Within thine own bud buriest thy content.\",\n",
    "                              \"Thy youth's proud livery so gazed on now.\", \n",
    "                              \"Shall sum my count, and make my old excuse.\"]\n",
    "\n",
    "    vocab = Vocab()\n",
    "    vocab.build(tokenize(sample_corpus))\n",
    "\n",
    "    assert set(vocab.word2id.keys()) == {'old', '<UNK>', 'on', '<END>', 'gazed', 'now',\n",
    "                                         'own', 'my', 'proud', 'content', ',', 'desire',\n",
    "                                         'and', '<PAD>', 'shall', 'from', 'creatures',\n",
    "                                         'sum', 'fairest', 'youth', \"'s\", 'make', 'increase',\n",
    "                                         'excuse', 'we', 'bud', 'thine', '.', '<START>',\n",
    "                                         'livery', 'thy', 'count', 'buriest', 'within', 'so'}\n",
    "\n",
    "    return vocab\n",
    "\n",
    "vocab = test_build_vocab()\n",
    "print('passed!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "lQLJJ0zloJYD",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Implement `pad_sents` method and test it with the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1523,
     "status": "ok",
     "timestamp": 1587898894703,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "coszZZJmoIC4",
    "new_sheet": false,
    "outputId": "284e6da1-f14a-47e4-d80f-00a7f91001ba",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_pad():\n",
    "    vocab = test_build_vocab()\n",
    "\n",
    "    unpadded_sents = [['to', 'say', 'within', 'thine', 'own', 'deep', 'sunken', 'eyes'], \n",
    "                      ['of', 'his', 'self-love', 'to', 'stop', 'posterity',], \n",
    "                      ['die', 'single'],\n",
    "                      ['then', 'beauteous', 'niggard']]\n",
    "\n",
    "    padded_sents = vocab.pad_sents(unpadded_sents)\n",
    "\n",
    "    assert padded_sents == [['to', 'say', 'within', 'thine', 'own', 'deep', 'sunken', 'eyes'],\n",
    "                            ['of', 'his', 'self-love', 'to', 'stop', 'posterity', '<PAD>', '<PAD>'],\n",
    "                            ['die', 'single', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>'],\n",
    "                            ['then', 'beauteous', 'niggard', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']]\n",
    "    print('passed!')\n",
    "\n",
    "test_pad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "6XwEHP9sSw2o",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Implement `tokens2ids` and check your output with the following function (check whether each token is replaced by a correct id):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 731
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1506,
     "status": "ok",
     "timestamp": 1587898905142,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "e1JGewypSt26",
    "new_sheet": false,
    "outputId": "c7b5e718-e701-4496-d8ab-e3475ab3a876",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"'s\": 20,\n",
      " ',': 30,\n",
      " '.': 4,\n",
      " '<END>': 2,\n",
      " '<PAD>': 0,\n",
      " '<START>': 1,\n",
      " '<UNK>': 3,\n",
      " 'and': 31,\n",
      " 'bud': 16,\n",
      " 'buriest': 17,\n",
      " 'content': 18,\n",
      " 'count': 29,\n",
      " 'creatures': 9,\n",
      " 'desire': 11,\n",
      " 'excuse': 34,\n",
      " 'fairest': 8,\n",
      " 'from': 7,\n",
      " 'gazed': 24,\n",
      " 'increase': 12,\n",
      " 'livery': 22,\n",
      " 'make': 32,\n",
      " 'my': 6,\n",
      " 'now': 26,\n",
      " 'old': 33,\n",
      " 'on': 25,\n",
      " 'own': 15,\n",
      " 'proud': 21,\n",
      " 'shall': 27,\n",
      " 'so': 23,\n",
      " 'sum': 28,\n",
      " 'thine': 14,\n",
      " 'thy': 5,\n",
      " 'we': 10,\n",
      " 'within': 13,\n",
      " 'youth': 19}\n",
      "\n",
      "[[3, 3, 13, 14, 15, 3, 3, 3],\n",
      " [12, 3, 10, 3, 3, 11, 0, 0],\n",
      " [3, 3, 0, 0, 0, 0, 0, 0],\n",
      " [3, 21, 22, 0, 0, 0, 0, 0]]\n",
      "\n",
      "[12, 21, 3, 3, 0]\n"
     ]
    }
   ],
   "source": [
    "def test_tokens2id():\n",
    "    vocab = test_build_vocab()\n",
    "    sents = [['to', 'say', 'within', 'thine', 'own', 'deep', 'sunken', 'eyes'],\n",
    "             ['increase', 'his', 'we', 'to', 'stop', 'desire', '<PAD>', '<PAD>'],\n",
    "             ['die', 'single', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>'],\n",
    "             ['then', 'proud', 'livery', '<PAD>', '<PAD>', '<PAD>', '<PAD>', '<PAD>']]\n",
    "\n",
    "    sent = ['increase', 'proud', 'stop', 'sunken', '<PAD>']\n",
    "\n",
    "    pprint(vocab.word2id)\n",
    "    print()\n",
    "    pprint(vocab.tokens2ids(sents))\n",
    "    print()\n",
    "    pprint(vocab.tokens2ids(sent))\n",
    "\n",
    "\n",
    "test_tokens2id()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "OfR1rtMPVbaG",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Implement `to_tensor()` method and test it with the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1426,
     "status": "ok",
     "timestamp": 1587898959308,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "mdMApxnvVZS_",
    "new_sheet": false,
    "outputId": "c5d46206-359f-4a48-fdfc-ebf815bd037d",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([12, 21,  3,  3,  0])\n"
     ]
    }
   ],
   "source": [
    "def test_to_tensor():\n",
    "    sent = ['increase', 'proud', 'stop', 'sunken', '<PAD>']\n",
    "    vocab = test_build_vocab()\n",
    "\n",
    "    out = vocab.to_tensor(sent)\n",
    "\n",
    "    assert torch.is_tensor(out)\n",
    "    print(out)\n",
    "\n",
    "test_to_tensor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "E9-AdlfHgjVy",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 2.4 Pad & Pack and using them with RNNs in PyTorch (8 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "2zzAQos1gs_K",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "You might want to process a batch of data before and after feeding to a RNN (instead of directly feeding it in and using its output). In this section, we will look at some of the ways of doing this in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "AW7ATZE8huk9",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 2.4.1 `pad_sequence()` (2 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "uZ6VqPdt3Qmw",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "`pad_sequence` is used to convert variable length sequences to same size. You can do padding either manually (like what we you did in the previous part) or by using `torch.nn.utils.rnn.pad_sequence()`. <br/>\n",
    "Now let's use this function to pad some data and then feed it to a RNN. With your knowledge of PyTorch at this point, you should be able to read [torch.nn documentation](https://pytorch.org/docs/stable/nn.html) and figure out how to do this task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "OsNHjjeih9HI",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from torch.nn.utils.rnn import pad_sequence\n",
    "\n",
    "def pad_and_feed(data: List[torch.Tensor]) -> torch.Tensor: \n",
    "    \"\"\"Pads shorter sentences in a list of tensors and feeds it to a RNN.\n",
    "\n",
    "    data: data as a list of tensors with torch.int dtype.\n",
    "    \"\"\" \n",
    "    rnn, output = None, None\n",
    "    ################################################################################\n",
    "    # TODO: 1) pads the data using pad_sequence of PyTorch                         #\n",
    "    #       2) Instantiate rnn using torch.nn.RNN with appropriate input size and  #\n",
    "    #       hidden size of 10.                                                     #\n",
    "    #       save the outputs (hidden states) of applying the rnn to the data in    #\n",
    "    #       the output variable.                                                   #         \n",
    "    ################################################################################\n",
    "    in_features=1 if data[0].ndim==1 else data[0].shape[1]\n",
    "    output=pad_sequence(data).reshape((-1,len(data),in_features))\n",
    "    rnn=RNN(in_features,10)\n",
    "    output=rnn(output)[0]\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################ \n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "wrwonn07u3Jr",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's test your implementation with the following function (the test is not exhaustive and is just checking shapes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11485,
     "status": "ok",
     "timestamp": 1587900301197,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "e2OidlVek5FJ",
    "new_sheet": false,
    "outputId": "7f1a549c-fcbd-44f8-c310-a4a44d542b52",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_pad_and_feed():\n",
    "    np.random.seed(42)\n",
    "    data = [torch.empty(np.random.randint(0, 10), 100) for _ in range(5)]\n",
    "    assert pad_and_feed(data).shape == torch.Size([7, 5, 10])\n",
    "    print('passed!')\n",
    "\n",
    "test_pad_and_feed()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "SpGEjxICnEV3",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 2.4.2 `pack_sequence()` (2 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "Rkicrz0Y5Vy_",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After padding the data, there will be a lot of zeros (we often set the id of pad token to zero) representing pads in it. These zeros do not really need to be processed by RNN because they do not represent any meaningful data! In fact, we just use them to make a single tensor from our variable length sentences. Therefore, feeding padded data directly to RNN is inefficient.\n",
    "\n",
    "In order to make the process more efficient, there is a function called `pack_sequence()` that reforms the data so that the model can just process useful tokens and not the paddings.\n",
    "\n",
    "Let's see what exactly pack does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 190
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1679,
     "status": "ok",
     "timestamp": 1587900402235,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "ZejMlf2Twbym",
    "new_sheet": false,
    "outputId": "4fdb81e7-57d5-4a0a-91e0-2be7d6fa0242",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PackedSequence(data=tensor([[0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "        [2., 2., 2., 2., 2., 2., 2., 2.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0.]]), batch_sizes=tensor([3, 2, 2, 1, 1]), sorted_indices=None, unsorted_indices=None)"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch.nn.utils.rnn import pack_sequence\n",
    "\n",
    "def pack():\n",
    "    a = torch.zeros((5, 8))\n",
    "    b = torch.ones((3, 8))\n",
    "    c = 2 * torch.ones((1, 8))\n",
    "    ################################################################################\n",
    "    # TODO:  pack the above three tensors and print the packed output.             #\n",
    "    ################################################################################\n",
    "    return pack_sequence([a,b,c])\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################ \n",
    "\n",
    "pack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "yi7IBIFzEJs6",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "##### **Questions**: \n",
    "\n",
    "1) what pattern do you see in the packed data? \n",
    "\n",
    "**YOUR ANSWER**: First it picks first token of all sentences then picks second token of all sentences and so on and determines place of i'th token rows in tensor by i'th entry of batch_sizes field.\n",
    "\n",
    "2) what does `batch_sizes` mean in the packed data? <br/>\n",
    "\n",
    "**YOUR ANSWER**: Refer to q1.\n",
    "\n",
    "3) How do you think RNN processes packed data?\n",
    "\n",
    "**YOUR ANSWER**: processes them batch by batch(i'th word of all the sentences then i+1'th word)\n",
    "\n",
    "Feel free to write your answers in Persian."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "TuFVHmdYmNaE",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 2.4.3 `pack_padded_sequence()` and `pad_packed_sequence()` (2 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "30tezAidG_lM",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "You will usually see that two functions are frequenlty used to process data before and after feeding it to the RNNs: `pack_padded_sequence()` and `pad_packed_sequence()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "QCc6Q7ycj_er",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# We will import these functions as pack and unpack\n",
    "from torch.nn.utils.rnn import pack_padded_sequence as pack\n",
    "from torch.nn.utils.rnn import pad_packed_sequence as unpack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "_U90CFyAOtMd",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def pad_and_pack(data: List[torch.Tensor]) -> torch.Tensor:\n",
    "    sorted_data = sorted(data, key=lambda element: element.shape[0], reverse=True)\n",
    "    lengths = [d.shape[0] for d in sorted_data]\n",
    "    ################################################################################\n",
    "    # TODO: 1) pad the data using pad_sequence()                                   #\n",
    "    #       2) pack the data and feed it to a RNN with hidden size of 10           #\n",
    "    #       3) unpack the RNN's outputs (hidden states) and return it              #\n",
    "    ################################################################################ \n",
    "    rnn=RNN(data[0].shape[1],10)  \n",
    "    return unpack(rnn(pack(pad_sequence(sorted_data),lengths))[0])[0]\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "0qFG6Qdpzfo3",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's test your implementation with the following function (the test is not exhaustive and is just checking shapes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 877,
     "status": "ok",
     "timestamp": 1587902049717,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "VnyWoSa3zkcy",
    "new_sheet": false,
    "outputId": "77ff64f9-cfcc-4a9e-cf5c-738be9044ff7",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_pad_and_pack():\n",
    "    np.random.seed(42)\n",
    "    data = [torch.empty(np.random.randint(0, 10), 100) for _ in range(5)]\n",
    "    outputs = pad_and_pack(data)\n",
    "    assert outputs.size() == torch.Size([7, 5, 10])\n",
    "    print('passed!')\n",
    "\n",
    "test_pad_and_pack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "7rEzSkHNOGo4",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 2.5 Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "42IqDbUEOFsB",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As you have seen in previous parts, the shape of a batch of our sentences would be $(N, L)$ where $N$ is the number of sentences and $L$ is the length of the longest sequence (Note that after padding all of the sequences have the same length $L$). Each entry of this tensor is the id of a token in our vocab. <br/>\n",
    "\n",
    "Feeding these integer ids directly to RNNs is obviously a bad idea. We need to represent each token of our vocabulary with a dense vector. That's where embeddings comes in. <br/>\n",
    "\n",
    "You can instantiate embedding layers in PyTorch by using `torch.nn.Embedding`. If you feed a Tensor of vocab indices with shape $(d_1, d_2, ..., d_k)$ to an embedding layer, it will return a $(d_1, d_2, ..., d_k, D)$ tensor where $D$ is the embedding dimension of the layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1454,
     "status": "ok",
     "timestamp": 1587902713675,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "sJAsaEK1sfwy",
    "new_sheet": false,
    "outputId": "a3a392e7-73ca-4130-a39f-f01454707b2b",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 0.4632, -1.1937, -0.2864],\n",
      "         [-0.6684,  0.1184, -1.4882],\n",
      "         [ 0.9261, -0.2963, -1.6865],\n",
      "         [ 0.9261, -0.2963, -1.6865]],\n",
      "\n",
      "        [[ 0.4632, -1.1937, -0.2864],\n",
      "         [-1.2028,  0.7015, -1.2490],\n",
      "         [ 0.0000,  0.0000,  0.0000],\n",
      "         [ 0.0000,  0.0000,  0.0000]],\n",
      "\n",
      "        [[-0.6684,  0.1184, -1.4882],\n",
      "         [ 0.0000,  0.0000,  0.0000],\n",
      "         [ 0.0000,  0.0000,  0.0000],\n",
      "         [ 0.0000,  0.0000,  0.0000]]])\n"
     ]
    }
   ],
   "source": [
    "def embed():\n",
    "    set_seed(40719)\n",
    "    data = torch.tensor([[1, 2, 4, 4], [1, 3, 0, 0], [2, 0, 0, 0]])\n",
    "    layer = nn.Embedding(num_embeddings=5, embedding_dim=3, padding_idx=0)  # what is padding_idx?\n",
    "    with torch.no_grad():\n",
    "        print(layer(data))\n",
    "\n",
    "embed() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "kOF6ybSFOMVt",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 2.6 Putting it all together (2 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "bx8RH--bzx0n",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's put all the stuff we learned above together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "QVsaGw5Pm5G3",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def process_data(data: List[str], vocab: Vocab) -> torch.Tensor:\n",
    "    outputs = None\n",
    "    ################################################################################\n",
    "    # TODO: 1) tokenize the data using your tokenize function.                     #                                     \n",
    "    #       2) use vocab's pad_sents method to pad the data.                       #\n",
    "    #       3) replace tokens with their ids in the vocab.                         #\n",
    "    #       4) instantiate an embedding layer with embedding size of 10 and        #\n",
    "    #          vocabulary size equal to vocab.size.                                #\n",
    "    #       5) wrap a tensor around the data and apply the embedding layer to it.  # \n",
    "    #       6) pack the data and feed it to a RNN with hidden size of 5.           #\n",
    "    #       7) unpack the RNN's outputs (hidden states) and save it in outputs.    #\n",
    "    ################################################################################ \n",
    "    sentences=tokenize(data)\n",
    "    lengths = [len(d) for d in sentences]\n",
    "    padded_sents=vocab.pad_sents(sentences)\n",
    "    layer = nn.Embedding(num_embeddings=vocab.size, embedding_dim=10, padding_idx=0)\n",
    "    embedded=layer(vocab.to_tensor(padded_sents)).permute(1,0,2)\n",
    "    outputs=unpack(RNN(10,5)(pack(embedded,lengths,enforce_sorted=False))[0])[0]\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "5leRPmPB4VwR",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's test your implementation (the test is not exhaustive and is just checking shapes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1011,
     "status": "ok",
     "timestamp": 1587905748776,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "1TNt4plR4YZm",
    "new_sheet": false,
    "outputId": "940d41ea-32d5-4af0-eafb-7c5f2559a17e",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_process_data():\n",
    "    vocab = test_build_vocab()\n",
    "    sents = ['to say within thine own deep sunken eyes', \n",
    "             'of his self-love to stop posterity', \n",
    "             'then beauteous niggard'\n",
    "             'die single']\n",
    "\n",
    "    outputs = process_data(sents, vocab)\n",
    "    assert outputs.shape == torch.Size([8, 3, 5])\n",
    "    print('passed!')\n",
    "\n",
    "test_process_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MG0_Hr56pJj8"
   },
   "source": [
    "## Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t3r5rMBbqNpz"
   },
   "source": [
    "- Check and review your answers. Make sure all cells' output are what you have planned.\n",
    "- Select File > Save.\n",
    "- To download the notebook, select File > Download .ipynb.\n",
    "- Create an archive of all notebooks (P1.ipynb, P2.ipynb, and P3.ipynb)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "SpGEjxICnEV3"
   ],
   "name": "HW4_Part2.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}