SNLI Dataset Analysis and Model Comparison.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook attempts to replicate Bowman et al.'s implementation of NLI on their SNLI dataset with a neural network architecture, as well as the neural attention based model seen in Rocktaschel et al."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, due to memory constraints, only a subset of the dataset could be used"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the main steps to completing this process:\n",
    "1. Import SNLI dataset\n",
    " - This may require imprting a portion of the dataset in order to save memory\n",
    "2. Perform an exploratory data analysis\n",
    " - Vocabulary size\n",
    " - Maximum observation length\n",
    " - Distribution of obswrvation lengths based on set type(training, testing, validation)\n",
    " - Word frequencies\n",
    " - Label frequencies\n",
    " - N-grams\n",
    "3. Preprocess data by\n",
    " - Clean -1 labels\n",
    " - Tokenize data\n",
    " - Pad data\n",
    " - Perform embedding\n",
    "     - We will compute sentence embeddings with MPnet for the basic Bowman et al. model\n",
    "     - We will do word embeddings for the neural attention model\n",
    " - DATA CLEANING DOES NOT INVOLVE REMOVING STOP WORDS, AS THEY WILL HAVE AN ACTUAL EFFECT ON THE ENTAILMENT OF TWO SENTENCES\n",
    "4. Create, train, and evaluate models\n",
    "- Basic Bowman et al. model with MPnet sentence embeddings\n",
    "- Basic Bowman et al. model with LSTM sentence embeddings\n",
    "- Rocktaschel's conditional LSTM model with MPnet sentence embeddings"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import SNLI dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import modules for step 1\n",
    "from datasets import load_dataset, Dataset, DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Actually import dataset into RAM\n",
    "\n",
    "dataset_train = load_dataset(\"snli\", split=\"train\")[:]\n",
    "dataset_test = load_dataset(\"snli\", split=\"test\")[:]\n",
    "dataset_valid = load_dataset(\"snli\", split=\"validation\")[:]\n",
    "\n",
    "dataset = DatasetDict({\"train\":Dataset.from_dict(dataset_train), \"test\":Dataset.from_dict(dataset_test), \"validation\":Dataset.from_dict(dataset_valid)})\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exploratory Data Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we attempt to find\n",
    " - Vocabulary size\n",
    " - Maximum observation length\n",
    " - Distribution of obswrvation lengths based on set type(training, testing, validation)\n",
    " - Word frequencies\n",
    " - Label frequencies\n",
    " - N-grams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import modules for step 2\n",
    "from tqdm import tqdm\n",
    "import statistics\n",
    "import numpy as np\n",
    "from datasets import concatenate_datasets\n",
    "import spacy\n",
    "from collections import Counter, defaultdict\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk import ngrams\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "nlp.max_length = 58182256"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_corpus = concatenate_datasets([dataset[\"train\"], dataset[\"test\"], dataset[\"validation\"]])\n",
    "text_full_corpus = full_corpus[\"premise\"]+full_corpus[\"hypothesis\"]\n",
    "text_full_corpus_string = \" \".join(text_full_corpus).lower()\n",
    "# count the number of unique tokens\n",
    "vocab_size = len(set(text_full_corpus_string.split()))\n",
    "print(\"Vocabulary size:\", vocab_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max observation length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_len = 0\n",
    "for text in tqdm(text_full_corpus):\n",
    "    current_length = len(text.split())\n",
    "    if current_length > max_len:\n",
    "        max_len = current_length\n",
    "print(max_len)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text length distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_length_distribution(data, dataset_name:str):\n",
    "    #Get lengths of hypothesis\n",
    "    text_lengths_hypothesis = [len(text.split()) for text in data[\"hypothesis\"]]\n",
    "    #Get lengths of premise\n",
    "    text_lengths_premise = [len(text.split()) for text in data[\"premise\"]]\n",
    "    #Plot lengths with a histogram\n",
    "    plt.hist(text_lengths_hypothesis, color=\"blue\", alpha=0.5, bins=20)\n",
    "    plt.hist(text_lengths_premise, color=\"red\", alpha=0.5,bins=20)\n",
    "    plt.xlabel(\"Text Length\")\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.title(f'Distribution of {dataset_name} Text Lengths')\n",
    "    plt.show()\n",
    "    print(f\"Mean: {statistics.mean(text_lengths_hypothesis+text_lengths_premise)}\")\n",
    "    print(f\"Median: {statistics.median(text_lengths_hypothesis+text_lengths_premise)}\")\n",
    "    print(f\"Max: {max(text_lengths_hypothesis+text_lengths_premise)}\")\n",
    "    print(f\"Min: {min(text_lengths_hypothesis+text_lengths_premise)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_length_distribution(dataset[\"train\"], \"Train\")\n",
    "text_length_distribution(dataset[\"test\"], \"Test\")\n",
    "text_length_distribution(dataset[\"validation\"], \"Validation\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word frequencies\n",
    "This takes a long time with the whole dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_frequency_distribution(data, dataset_name:str):\n",
    "    #Convert hypothesis dataset to a string\n",
    "    data_string_hypothesis = \" \".join(data[\"hypothesis\"])\n",
    "    #Convert premise dataset to a string\n",
    "    data_string_premise = \" \".join(data[\"premise\"])\n",
    "    doc_hyp = nlp(data_string_hypothesis+data_string_premise)\n",
    "    words = [token.text for token in doc_hyp if not token.is_stop and token.is_alpha]\n",
    "    freq_dist = Counter(words)\n",
    "    x, y1 = zip(*freq_dist.most_common(30))\n",
    "    plt.figure(figsize=(10,5))\n",
    "    plt.bar(x, y1, alpha=0.5,color=\"red\")\n",
    "    plt.title(f'Word Frequency Distribution {dataset_name}')\n",
    "    plt.xlabel('Words')\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.xticks(rotation=80)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_frequency_distribution(dataset[\"validation\"], \"Validation\")\n",
    "word_frequency_distribution(dataset[\"train\"], \"Training\")\n",
    "word_frequency_distribution(dataset[\"test\"], \"Testing\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_label_frequencies(data, dataset_name:str):\n",
    "    labels = defaultdict(lambda: 0)\n",
    "    for obs in data:\n",
    "        labels[obs[\"label\"]] += 1\n",
    "    values = list(labels.values())\n",
    "    keys = list(labels.keys())\n",
    "    for val in keys:\n",
    "        print(f\"{val}:{100*(labels[val]/len(data))}%\")\n",
    "    plt.bar(keys, values)\n",
    "    plt.title(f'Frequency of {dataset_name} Labels')\n",
    "    plt.xlabel('Label')\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_label_frequencies(dataset[\"validation\"], \"Validation\")\n",
    "get_label_frequencies(dataset[\"train\"], \"Train\")\n",
    "get_label_frequencies(dataset[\"test\"], \"Test\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "def top_bigrams(dataset):\n",
    "     # Define bigram size\n",
    "    n = 2\n",
    "    # Define list to store bigrams\n",
    "    bigrams_list = []\n",
    "    \n",
    "    # Loop over each list of sentences in the dictionary\n",
    "    for key, sentences in dataset.items():\n",
    "        # Check if key should be excluded\n",
    "        if key in [\"label\"]:\n",
    "            continue\n",
    "        \n",
    "        # Combine all sentences in list into a single string with stop words removed\n",
    "        section = \" \".join(sentences).lower()\n",
    "        section = ' '.join([word for word in section.split() if word.lower() not in stop_words])\n",
    "        \n",
    "        # Generate bigrams for section\n",
    "        words = section.split()\n",
    "        section_bigrams = list(ngrams(words, n))\n",
    "        bigrams_list += section_bigrams\n",
    "    \n",
    "    # Count frequency of each bigram\n",
    "    bigrams_counts = Counter(bigrams_list)\n",
    "\n",
    "    # Get top 20 most common bigrams\n",
    "    top_bigrams = bigrams_counts.most_common(20)\n",
    "    \n",
    "    return top_bigrams\n",
    "    return top_bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_bigrams = \"Train\\n\"\n",
    "for bigram, count in top_bigrams(dataset_train):\n",
    "    training_bigrams += str((bigram, count))+\"\\n\"\n",
    "valid_bigrams = \"Valid\\n\"\n",
    "for bigram, count in top_bigrams(dataset_valid):\n",
    "    valid_bigrams += str((bigram, count))+\"\\n\"\n",
    "test_bigrams = \"Test\\n\"\n",
    "for bigram, count in top_bigrams(dataset_test):\n",
    "    test_bigrams += str((bigram, count))+\"\\n\"\n",
    "zipped = zip(training_bigrams.split(\"\\n\"), valid_bigrams.split(\"\\n\"), test_bigrams.split(\"\\n\"))\n",
    "\n",
    "for count, row in enumerate(zipped):\n",
    "    print(f\"{row[0].ljust(25)}&{row[1].ljust(25)}&{row[2].ljust(25)}\\\\\\\\\")\n",
    "    print(f\"\\hline\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Clean data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules for step 3\n",
    "import numpy as np\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras_preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "from multiprocessing import Pool"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove -1s from dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_dataset = dataset.filter(lambda example: example['label'] != -1)# and example['label'] != 1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test new label frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_label_frequencies(filtered_dataset[\"validation\"], \"Filtered Validation\")\n",
    "get_label_frequencies(filtered_dataset[\"train\"], \"Filtered Train\")\n",
    "get_label_frequencies(filtered_dataset[\"test\"], \"Filtered Test\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find the new vocab size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_corpus = concatenate_datasets([filtered_dataset[\"train\"], filtered_dataset[\"test\"]])\n",
    "text_full_corpus = full_corpus[\"premise\"]+full_corpus[\"hypothesis\"]\n",
    "text_full_corpus_string = \" \".join(text_full_corpus).lower()\n",
    "# count the number of unique tokens\n",
    "vocab_size = len(set(text_full_corpus_string.split()))\n",
    "print(\"Vocabulary size:\", vocab_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### New Max observation length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_len = 0\n",
    "for text in tqdm(text_full_corpus):\n",
    "    current_length = len(text.split())\n",
    "    if current_length > max_len:\n",
    "        max_len = current_length\n",
    "print(max_len)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence embeddings from MPnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "sbert_model = SentenceTransformer('all-mpnet-base-v2', device=\"cuda\")\n",
    "sbert_model.max_seq_length = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_prem = sbert_model.encode(filtered_dataset[\"train\"][\"premise\"], show_progress_bar=True)\n",
    "train_hyp = sbert_model.encode(filtered_dataset[\"train\"][\"hypothesis\"], show_progress_bar=True)\n",
    "\n",
    "test_prem = sbert_model.encode(filtered_dataset[\"test\"][\"premise\"], show_progress_bar=True)\n",
    "test_hyp = sbert_model.encode(filtered_dataset[\"test\"][\"hypothesis\"], show_progress_bar=True)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word embeddings with gloVe"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenize and pad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(full_corpus[\"premise\"]+full_corpus[\"hypothesis\"])\n",
    "train_premise_sequences = tokenizer.texts_to_sequences(filtered_dataset[\"train\"][\"premise\"])\n",
    "train_hypothesis_sequences = tokenizer.texts_to_sequences(filtered_dataset[\"train\"][\"hypothesis\"])\n",
    "test_premise_sequences = tokenizer.texts_to_sequences(filtered_dataset[\"test\"][\"premise\"])\n",
    "test_hypothesis_sequences = tokenizer.texts_to_sequences(filtered_dataset[\"test\"][\"hypothesis\"])\n",
    "\n",
    "padded_train_premise_sequences = pad_sequences(train_premise_sequences, padding='post', maxlen=max_len)\n",
    "padded_train_hypothesis_sequences = pad_sequences(train_hypothesis_sequences, padding='post', maxlen=max_len)\n",
    "\n",
    "padded_test_premise_sequences = pad_sequences(test_premise_sequences, padding='post', maxlen=max_len)\n",
    "padded_test_hypothesis_sequences = pad_sequences(test_hypothesis_sequences, padding='post', maxlen=max_len)\n",
    "\n",
    "tokenizer.word_index[\"NULL\"] = 0\n",
    "tokenizer.index_word[0] = \"NULL\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Glove"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#FROM : https://medium.com/analytics-vidhya/basics-of-using-pre-trained-glove-vectors-in-python-d38905f356db\n",
    "embeddings_dict = {}\n",
    "\n",
    "with open(\"glove.6B.100d.txt\", 'r', encoding=\"utf-8\") as f:\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        vector = np.asarray(values[1:], \"float32\")\n",
    "        embeddings_dict[word] = vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's map the embeddings to our own tokenized words!\n",
    "# Create a dictionary that maps tokens to their embeddings\n",
    "embedding_dict_tokenized = {}\n",
    "for token in tqdm(tokenizer.word_index):\n",
    "    if token in list(embeddings_dict.keys()):\n",
    "        embedding_dict_tokenized[token] = embeddings_dict[token]\n",
    "    else:\n",
    "        embedding_dict_tokenized[token] = np.random.rand(100)\n",
    "embedding_dict_tokenized[\"NULL\"] = np.zeros(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def embedded_sequence(sequences, embedding_dict_tokenized):\n",
    "    embedded_sequences = []\n",
    "    for seq in tqdm(sequences):\n",
    "        embedded_sequence = []\n",
    "        for token in seq:\n",
    "            embedded_sequence.append(embedding_dict_tokenized[tokenizer.index_word[token]])\n",
    "        embedded_sequences.append(embedded_sequence)\n",
    "    return embedded_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_hyp = embedded_sequence(padded_train_hypothesis_sequences, embedding_dict_tokenized)\n",
    "train_prem = embedded_sequence(padded_train_premise_sequences, embedding_dict_tokenized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_hyp = embedded_sequence(padded_test_hypothesis_sequences, embedding_dict_tokenized)\n",
    "test_prem = embedded_sequence(padded_test_premise_sequences, embedding_dict_tokenized)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One hot encode labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = to_categorical(filtered_dataset[\"train\"][\"label\"], num_classes=3)\n",
    "y_test = to_categorical(filtered_dataset[\"test\"][\"label\"], num_classes=3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create, train, and evaluate models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules for step 4\n",
    "from tensorflow.keras.layers import Input, Dense, Concatenate, Embedding, LSTM, Activation, Layer\n",
    "from tensorflow.keras.models import Model\n",
    "from keras.callbacks import EarlyStopping\n",
    "import tensorflow\n",
    "import keras.backend as K"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word-by-Word Attention Based Model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create basic layers for trainable weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Trainable_Matrix(Layer):\n",
    "\n",
    "  def __init__(self, units=32):\n",
    "      super(Trainable_Matrix, self).__init__()\n",
    "      self.units = units\n",
    "\n",
    "  def build(self, input_shape):  # Create the state of the layer (weights)\n",
    "    w_init = tensorflow.random_normal_initializer()\n",
    "    self.w = tensorflow.Variable(\n",
    "        initial_value=w_init(shape=(input_shape[-1], self.units),\n",
    "                             dtype='float32'),\n",
    "        trainable=True)\n",
    "\n",
    "  def call(self, inputs):  # Defines the computation from inputs to outputs\n",
    "      return tensorflow.matmul(inputs, self.w)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_wordword_attention_SNLI_model():\n",
    "    input_premise = Input(shape=(max_len, 100))\n",
    "    input_hypothesis = Input(shape=(max_len, 100))\n",
    "    lstm_premise = LSTM(units=64, return_sequences=True)(input_premise) #key\n",
    "    lstm_hypothesis = LSTM(units=64)(input_hypothesis) #query\n",
    "    attn = tensorflow.keras.layers.Attention()([lstm_premise, lstm_hypothesis])\n",
    "    attn_flat = tensorflow.keras.layers.Flatten()(attn)\n",
    "    output = Dense(units=3, activation=\"softmax\")(attn_flat)\n",
    "    model = tensorflow.keras.models.Model(inputs=[input_premise, input_hypothesis], outputs=output)\n",
    "    return model\n",
    "wordword_attention_model = create_wordword_attention_SNLI_model()\n",
    "optimizer =  tensorflow.keras.optimizers.Adam(learning_rate=0.001)\n",
    "wordword_attention_model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "wordword_attention_model.summary()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stop = EarlyStopping(monitor='loss', patience=2, min_delta=0.0005, restore_best_weights=True)\n",
    "wordword_attention_model.fit([np.array(train_prem), \\\n",
    "           np.array(train_hyp)], \\\n",
    "            y_train, \\\n",
    "            epochs=10, callbacks=[early_stop])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing & Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordword_pred = wordword_attention_model.predict([test_prem, test_hyp])\n",
    "correct_labels = [np.argmax(x) for x in y_test]\n",
    "wordword_pred_labels = [np.argmax(x) for x in wordword_pred]\n",
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(correct_labels, wordword_pred_labels))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conditional LSTM model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_conditional_LSTM_SNLI_model():\n",
    "    input_premise = Input(shape=(max_len, 100))\n",
    "    input_hypothesis = Input(shape=(max_len, 100))\n",
    "    lstm_premise = LSTM(units=64, return_state=True)(input_premise)\n",
    "    lstm_hypothesis = LSTM(units=64)(input_hypothesis, initial_state=lstm_premise[1:])\n",
    "    output = Dense(units=3, activation=\"softmax\")(lstm_hypothesis)\n",
    "    model = tensorflow.keras.models.Model(inputs=[input_premise, input_hypothesis], outputs=output)\n",
    "    return model\n",
    "conditional_lstm_model = create_conditional_LSTM_SNLI_model()\n",
    "optimizer =  tensorflow.keras.optimizers.Adam(learning_rate=0.05)\n",
    "conditional_lstm_model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "conditional_lstm_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditional_lstm_model.fit([np.array(train_prem), \\\n",
    "           np.array(train_hyp)], \\\n",
    "            y_train, \\\n",
    "            epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing & Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditionalLSTM_pred = conditional_lstm_model.predict([test_prem, test_hyp])\n",
    "correct_labels = [np.argmax(x) for x in y_test]\n",
    "conditionalLSTM_pred_labels = [np.argmax(x) for x in conditionalLSTM_pred]\n",
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(correct_labels, conditionalLSTM_pred_labels))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bowman et al. LSTM sentence embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_bowman_LSTM_SNLI_model():\n",
    "    input_premise = Input(shape=(max_len, 100))\n",
    "    input_hypothesis = Input(shape=(max_len, 100))\n",
    "    premise_lstm = LSTM(units=100)(input_premise)\n",
    "    hypothesis_lstm = LSTM(units=100)(input_hypothesis)\n",
    "    concat_premise_hypothesis = Concatenate()([premise_lstm, hypothesis_lstm])\n",
    "    tanh1 = Dense(units=200, activation=\"tanh\", )(concat_premise_hypothesis)\n",
    "    tanh2 = Dense(units=200, activation=\"tanh\")(tanh1)\n",
    "    tanh3 = Dense(units=200, activation=\"tanh\")(tanh2)\n",
    "    output = Dense(units=3, activation=\"softmax\")(tanh3)\n",
    "    model = tensorflow.keras.models.Model(inputs=[input_premise, input_hypothesis], outputs=output)\n",
    "    return model\n",
    "lstm_bowman_model = create_bowman_LSTM_SNLI_model()\n",
    "optimizer =  tensorflow.keras.optimizers.SGD(learning_rate=0.1)\n",
    "lstm_bowman_model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "lstm_bowman_model.summary()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_bowman_model.fit([np.array(train_prem), np.array(train_hyp)], y_train,epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing & Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wordword_pred = create_bowman_LSTM_SNLI_model.predict([test_prem, test_hyp])\n",
    "correct_labels = [np.argmax(x) for x in y_test]\n",
    "wordword_pred_labels = [np.argmax(x) for x in wordword_pred]\n",
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(correct_labels, wordword_pred_labels))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the Bowman \"basic\" model with MPnet sentence embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_bowman_SNLI_model():\n",
    "    input_premise = Input(shape=(768))\n",
    "    input_hypothesis = Input(shape=(768))\n",
    "    concat_premise_hypothesis = Concatenate()([input_premise, input_hypothesis])\n",
    "    tanh1 = Dense(units=200, activation=\"tanh\", )(concat_premise_hypothesis)\n",
    "    tanh2 = Dense(units=200, activation=\"tanh\")(tanh1)\n",
    "    tanh3 = Dense(units=200, activation=\"tanh\")(tanh2)\n",
    "    output = Dense(units=3, activation=\"softmax\")(tanh3)\n",
    "    model = tensorflow.keras.models.Model(inputs=[input_premise, input_hypothesis], outputs=output)\n",
    "    return model\n",
    "bowman_model = create_bowman_SNLI_model()\n",
    "optimizer =  tensorflow.keras.optimizers.SGD(learning_rate=0.1)\n",
    "bowman_model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "bowman_model.summary()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bowman_model.fit([np.array(train_prem), np.array(train_hyp)], y_train,epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing & Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "basic_pred = bowman_model.predict([test_prem, test_hyp])\n",
    "correct_labels = [np.argmax(x) for x in y_test]\n",
    "basic_pred_labels = [np.argmax(x) for x in basic_pred]\n",
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(correct_labels, basic_pred_labels))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}