sample_test_runs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3696c664-18f5-49d0-af49-723dcf711445",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Old Model and Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "99bb1793-b3ea-48d8-ae27-e51302250f30",
   "metadata": {},
   "outputs": [],
   "source": [
    "from question_generator import QuestionGenerator\n",
    "from sentence_embeddings import SentenceEmbeddings\n",
    "from paraphraser_class import Paraphraser\n",
    "\n",
    "model_path = 'model/'\n",
    "tokenizer_path = 'tokenizer/'\n",
    "\n",
    "def qg(context, answer, question_generator, sentence_embeddings, paraphraser):\n",
    "    generated_questions = question_generator.generate(answer, context)\n",
    "\n",
    "    qa_list = [{'question': q, 'answer': answer} for q in generated_questions]\n",
    "    top_questions = sentence_embeddings.get_most_similar(context, qa_list)\n",
    "\n",
    "    for qa in top_questions:\n",
    "        # print(\"Original Question:\", qa['question'])\n",
    "        best_paraphrase = paraphraser.unique_paraphrase(context, qa['question'])\n",
    "        print(\"Generated Question:\", best_paraphrase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1464ecff-576c-4e41-95ff-9f68263f1863",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "question_generator = QuestionGenerator(trained_model_path=model_path, trained_tokenizer_path=tokenizer_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ea73d15d-4b23-4915-b6e0-1bb1affe17d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_embeddings = SentenceEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4174f220-8225-479d-9eeb-66622f601c25",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f8642c42c4440e1ae6b453b3f22743a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-large-uncased-whole-word-masking-finetuned-squad were not used when initializing BertForQuestionAnswering: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
      "- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "paraphraser = Paraphraser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba42bcac-93c5-4731-9008-e9d4fc446c5e",
   "metadata": {},
   "source": [
    "## Sample Run 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc40e557-e7d7-44e6-aad7-d8b9103dafb2",
   "metadata": {},
   "source": [
    "### New Run (Answer Check Implemented: The paraphrases that have the same answer get bonus points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9f806af7-b3c3-4d86-ac96-906efbde7ead",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Google LLC is an American multinational corporation and technology company focusing on \n",
    "search engine technology, cloud computing, computer software, quantum computing and artificial intelligence (AI).\n",
    "It has been referred to as \"the most powerful company in the world\" and is \n",
    "one of the world's most valuable brands due to its market dominance, data collection, and technological advantages in the field of AI.\n",
    "Google's parent company, Alphabet Inc. is one of the five Big Tech companies, alongside Amazon, Apple, Meta, and Microsoft.\n",
    "\"\"\"\n",
    "answer = \"American\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0277da94-68fa-4987-a814-179a9b884891",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/cta/users/serhan.yilmaz/.local/lib/python3.10/site-packages/transformers/generation/utils.py:1141: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What country of origin is Google LLC?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What is Google's national identity?\n",
      "Generated Question: What is the nationality or domicile of Google LLC?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4ea5de5-07cc-4288-8a9c-c23dd281964d",
   "metadata": {},
   "source": [
    "### Old Run (Don't Run Again - Answer Check was not implemented yet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4c7ed3b8-17d6-431f-920d-b1667dd15240",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/cta/users/serhan.yilmaz/.local/lib/python3.10/site-packages/transformers/generation/utils.py:1141: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What country of origin is Google LLC?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What country of origin is Google LLC from?\n",
      "Generated Question: What is the national identity of Google LLC?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44772ac-e39d-48eb-9fb0-5872016bef7e",
   "metadata": {},
   "source": [
    "## Sample Run 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3437067-3eec-47b6-b10f-2c5eba49d608",
   "metadata": {},
   "source": [
    "### What if the answer was something less suitable for Question Generation? \n",
    "### Something less like an answer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8503a374-f39e-4baf-a318-5a2713f73ce1",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"wiki engine\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "493d34e4-74e0-41d4-a1cb-d6d3cad3b1af",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Unlike other web-based systems, the content in this system is generated without a predetermined structure.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wiki software is also known by what alias?\n",
      "Generated Question: Unlike other web-based systems, the content is created without a predetermined owner.\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "068855aa-48f7-4576-8ae8-30dcb7c25ad8",
   "metadata": {},
   "source": [
    "### What if the answer was unrelated to the context? (Hallucination)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "09fa3240-ec6a-48b1-a732-d49919d1989a",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"American\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5e8c6881-4c13-4b72-beef-4f2165360e22",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What is the nationality of the people who developed the wiki software?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What is the nationality of the creators of the wiki software?\n",
      "Generated Question: Which nationality does the average Wiki software user belong to?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4377302-024f-4ab7-bf8e-7295bb6e9b90",
   "metadata": {},
   "source": [
    "### What if the answer was something even less like an answer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "047973bc-092b-4ecb-850e-013b9d29fe5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"emerge\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a17fc54d-efbe-4aaf-a3db-80c72090c6f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wikis adapt their structure to accommodate the needs of their users.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wikis tailor their layout to accommodate the demands of their users.\n",
      "Generated Question: What capabilities does a wiki engine grant to structured content?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46667f13-0970-4fbe-86c0-4572a13d8328",
   "metadata": {},
   "source": [
    "* 3rd Generated Question is not at all bad, does well.\n",
    "* The first 2 are not questions.\n",
    "* Same problem with the first example in Sample Run 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cc197c3-1833-4055-ac48-2d38c1ccf286",
   "metadata": {},
   "source": [
    "I should inspect the Generated Questions, and the returned paraphrases for this.\n",
    "\n",
    "It could be Llama3 hallucinating, or it could be T5 not generating good enough questions.\n",
    "\n",
    "A newer T5 is being trained on an A100 now, should be done in about 23 hours.\n",
    "\n",
    "If it is T5, enhancing the dataset might be needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88e76763-b77d-4dd8-8720-ac31da7d1205",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# New Model and Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "89d5ebd9-3c8b-47b1-93bb-31de2b79fc8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from question_generator import QuestionGenerator\n",
    "from sentence_embeddings import SentenceEmbeddings\n",
    "from paraphraser_class import Paraphraser\n",
    "\n",
    "model_path = 'new_model/model/'\n",
    "tokenizer_path = 'new_model/tokenizer/'\n",
    "\n",
    "def qg(context, answer, question_generator, sentence_embeddings, paraphraser):\n",
    "    generated_questions = question_generator.generate(answer, context)\n",
    "\n",
    "    qa_list = [{'question': q, 'answer': answer} for q in generated_questions]\n",
    "    top_questions = sentence_embeddings.get_most_similar(context, qa_list)\n",
    "\n",
    "    for qa in top_questions:\n",
    "        # print(\"Original Question:\", qa['question'])\n",
    "        best_paraphrase = paraphraser.unique_paraphrase(context, qa['question'])\n",
    "        print(\"Generated Question:\", best_paraphrase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ec877c9d-1f76-43ce-a65d-ced1246e5124",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "question_generator = QuestionGenerator(trained_model_path=model_path, trained_tokenizer_path=tokenizer_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6c3446d6-bfc8-407e-921a-b9384e2c0a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_embeddings = SentenceEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5b810a57-d29d-4b4f-b555-6ea4dd0e40f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d5afabd6a0b4b7b8101b586125e724a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-large-uncased-whole-word-masking-finetuned-squad were not used when initializing BertForQuestionAnswering: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
      "- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "paraphraser = Paraphraser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e733562c-7e65-4b1a-a6b4-33990db6476d",
   "metadata": {},
   "source": [
    "## Sample Run 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "87e4912c-73da-4b39-8bc4-eb0c5c7f4d5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Google LLC is an American multinational corporation and technology company focusing on \n",
    "search engine technology, cloud computing, computer software, quantum computing and artificial intelligence (AI).\n",
    "It has been referred to as \"the most powerful company in the world\" and is \n",
    "one of the world's most valuable brands due to its market dominance, data collection, and technological advantages in the field of AI.\n",
    "Google's parent company, Alphabet Inc. is one of the five Big Tech companies, alongside Amazon, Apple, Meta, and Microsoft.\n",
    "\"\"\"\n",
    "answer = \"American\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "153d888b-3f60-4482-83b7-bd718d92cd5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/cta/users/serhan.yilmaz/.local/lib/python3.10/site-packages/transformers/generation/utils.py:1141: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What is Google LLC's national origin?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What is the national identity of Google LLC?\n",
      "Generated Question: What is the national origin of Google's parent company?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcd392ca-32f8-489e-ad5b-aea3d8cc7563",
   "metadata": {},
   "source": [
    "## Sample Run 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f596d4d1-275a-49c8-a048-2c6699088c04",
   "metadata": {},
   "source": [
    "### What if the answer was something less suitable for Question Generation? \n",
    "### Something less like an answer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "00a05a07-722d-4e57-977e-f9f0ecee349c",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"wiki engine\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2c302301-fb43-4ad1-8e0d-08584ea59a7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wiki software is commonly known as what?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What is wiki software also known as?\n",
      "Generated Question: What are wiki software applications also known as?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c765ba5d-86ea-4c5f-bb50-8dc9ffe39245",
   "metadata": {},
   "source": [
    "### What if the answer was unrelated to the context? (Hallucination)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e2a2815b-b2ee-4f22-852f-ad4750deb720",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"American\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1202a4d5-dea4-4c43-a7f0-521d16f86be3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Who is behind the creation of a wiki?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: What ethnicity is associated with the concept of a wiki?\n",
      "Generated Question: What is the typical nationality of a person who creates and edits content on Wikipedia?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28300d6e-7581-4cad-b74d-00631f7f2375",
   "metadata": {},
   "source": [
    "Hahahhahahaha... what?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18bf5357-2e39-4f63-9190-1c0bdeaa6d40",
   "metadata": {},
   "source": [
    "### What if the answer was something even less like an answer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "71df04e4-0f30-4ae9-93d3-30a52a079ffa",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"emerge\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c11699ca-c384-4bab-a61c-e6c957ac4729",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wikis have no built-in structure, allowing them to evolve according to the needs of their contributors.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wikis' lack of inherent structure allows them to be constantly updated and revised.\n",
      "Generated Question: With no inherent structure, wikis can freely change and develop.\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "357fca75-79df-4d2f-a1b0-26e03cc7ba4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wikis have minimal built-in structure, allowing them to evolve according to the needs of their community.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Question: Wikis' lack of inherent structure enables them to evolve and transform over time.\n",
      "Generated Question: Wikis are characterized by their lack of predetermined structure, which allows it to and what?\n"
     ]
    }
   ],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8ac3d7c-d35e-4144-9bd9-3815a33507d8",
   "metadata": {},
   "source": [
    "* The question generation has gotten relatively better, arguably because of better model performance.\n",
    "* The algorithm still won't ensure 100% that there will be questions though, especially if the answer is unlikely to be an answer. Which is somewhat understandable.\n",
    "* I will train T5 one more time, a bit more harshly without early stopping, and let's observe those results in the next section.\n",
    "* It still bugs me that I take the inputs from the llama3 paraphrasing engine with regex though, unreliable.\n",
    "* In case the non-question generative behaviour continues, an additional scoring might be added to the scoring function, enabling it to give more points to sentences ending with a question mark, whether they give the same answer to the QA or not."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd98f389-7712-404e-9a68-94410fb31854",
   "metadata": {},
   "source": [
    "# Newest Model and Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f894fdd-d0c1-4392-a9b3-3a055ff1c54e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from question_generator import QuestionGenerator\n",
    "from sentence_embeddings import SentenceEmbeddings\n",
    "from paraphraser_class import Paraphraser\n",
    "\n",
    "model_path = 'newest_model/model/'\n",
    "tokenizer_path = 'newest_model/tokenizer/'\n",
    "\n",
    "def qg(context, answer, question_generator, sentence_embeddings, paraphraser):\n",
    "    generated_questions = question_generator.generate(answer, context)\n",
    "\n",
    "    qa_list = [{'question': q, 'answer': answer} for q in generated_questions]\n",
    "    top_questions = sentence_embeddings.get_most_similar(context, qa_list)\n",
    "\n",
    "    for qa in top_questions:\n",
    "        # print(\"Original Question:\", qa['question'])\n",
    "        best_paraphrase = paraphraser.unique_paraphrase(context, qa['question'])\n",
    "        print(\"Generated Question:\", best_paraphrase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8df2d09f-69b8-465a-9268-e4b6bb6bc560",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_generator = QuestionGenerator(trained_model_path=model_path, trained_tokenizer_path=tokenizer_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ad505cd-571b-4caa-8947-27d73b367a56",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_embeddings = SentenceEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98f5cef8-f77c-42d0-b569-474fa55e6dac",
   "metadata": {},
   "outputs": [],
   "source": [
    "paraphraser = Paraphraser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57adb179-16cc-4640-ab05-c5300ae152fe",
   "metadata": {},
   "source": [
    "## Sample Run 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c3ce8c9-e114-47f4-9d6a-32260a937e9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Google LLC is an American multinational corporation and technology company focusing on \n",
    "search engine technology, cloud computing, computer software, quantum computing and artificial intelligence (AI).\n",
    "It has been referred to as \"the most powerful company in the world\" and is \n",
    "one of the world's most valuable brands due to its market dominance, data collection, and technological advantages in the field of AI.\n",
    "Google's parent company, Alphabet Inc. is one of the five Big Tech companies, alongside Amazon, Apple, Meta, and Microsoft.\n",
    "\"\"\"\n",
    "answer = \"American\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94d19b2f-e39d-4862-a71e-6bec69449c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d3fbef-a139-4afc-9611-3b2e386abc3f",
   "metadata": {},
   "source": [
    "## Sample Run 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17ec8ba-876d-498d-bb54-70954e1b32f8",
   "metadata": {},
   "source": [
    "### What if the answer was something less suitable for Question Generation? \n",
    "### Something less like an answer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fe7c3cd-5ddb-45b3-8627-41c587012120",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"wiki engine\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7871fd85-a8bb-417c-abca-fb5a497d53c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d3d246-bacc-4817-84de-572b9fdb0c03",
   "metadata": {},
   "source": [
    "### What if the answer was unrelated to the context? (Hallucination)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fe431b1-6f25-4f1e-9b37-edf740a0fd1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"American\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "904e8ad8-59a9-49b7-9dd7-b6b5f7a15c61",
   "metadata": {},
   "outputs": [],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5f2a5fa-2911-4211-99c8-d47fbfb60760",
   "metadata": {},
   "source": [
    "Hahahhahahaha... what?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "920e2770-6f32-4000-a5fd-d9e3735ae794",
   "metadata": {},
   "source": [
    "### What if the answer was something even less like an answer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "003a25db-ee6f-4237-b34a-1b0f9c4337f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Wikis are enabled by wiki software, otherwise known as wiki engines. \n",
    "A wiki engine, being a form of a content management system, \n",
    "differs from other web-based systems such as blog software or static site generators, \n",
    "in that the content is created without any defined owner or leader, \n",
    "and wikis have little inherent structure, \n",
    "allowing structure to emerge according to the needs of the users.\"\"\"\n",
    "answer = \"emerge\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6da75e8-5a27-4ad1-be3c-991df940f488",
   "metadata": {},
   "outputs": [],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43d070c5-ee55-43d1-b6e6-6592a8127675",
   "metadata": {},
   "outputs": [],
   "source": [
    "qg(context, answer, question_generator, sentence_embeddings, paraphraser)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f8b595f-ebec-4457-a488-b7bd6ddef7ad",
   "metadata": {},
   "source": [
    "* The question generation has gotten relatively better, arguably because of better model performance.\n",
    "* The algorithm now seems to create questions only, but I worry about halucinations.\n",
    "* The questions are generated alright, but in the unlikely case above they don't give the answer.\n",
    "* One might also ask, is there a question that generates the given answer? If not, this can qualify as hallucination.\n",
    "* It still bugs me that I take the inputs from the llama3 paraphrasing engine with regex though, unreliable."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}