generate_dataset_finetune.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Synthetic Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "\n",
    "# Fetch the page\n",
    "url = 'https://www.consumerfinance.gov/rules-policy/regulations/1024/17/'\n",
    "response = requests.get(url)\n",
    "html_content = response.text\n",
    "\n",
    "# Parse the HTML\n",
    "soup = BeautifulSoup(html_content, 'html.parser')\n",
    "\n",
    "# Extract all paragraphs\n",
    "paragraphs = soup.find_all('p')\n",
    "# Combine all text into a single string for easier processing\n",
    "full_text = '\\n'.join([p.get_text() for p in paragraphs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=1250, chunk_overlap=20)\n",
    "\n",
    "chunks = text_splitter.split_text(full_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of chunks: \", len(chunks))\n",
    "print(\"Chunk 1: \", chunks[0])\n",
    "print(\"Chunk 2: \", chunks[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the chunks to a json file for future tracability\n",
    "import json\n",
    "with open('chunks.json', 'w') as f:\n",
    "    json.dump(chunks, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from utility.utils import get_openai_api_key\n",
    "\n",
    "OpenAI.api_key = get_openai_api_key()\n",
    "client = OpenAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SYSTEM_PROMPT = \"\"\"\n",
    "# Role\n",
    "you are a worldclass llm training data generator. \n",
    "\n",
    "# Task\n",
    "You are generating q/a pairs for finetuning our own llm model.\n",
    "create a json of questions and answers. \n",
    "\n",
    "# Specifics\n",
    "Context relecance for the Q/A is really important, else our business will loose money on computation resources \n",
    "\n",
    "         \n",
    "# Example format:\n",
    "{ data: [\n",
    "    {\n",
    "        \"q\": <question>,\n",
    "        \"a\": <answer>\n",
    "    },\n",
    "    {\n",
    "        \"q\": <question>,\n",
    "        \"a\": <answer>\n",
    "    }, ...\n",
    "]}\n",
    "\n",
    "# Notes\n",
    "Make sure to generate JSON format for the Q/A pairs\n",
    "Generate 10 Q/A pairs\n",
    "          \n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_qa_pairs(system_prompt, text):\n",
    "\n",
    "    chat_completion = client.chat.completions.create(\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": system_prompt\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": text\n",
    "            }\n",
    "        ],\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        response_format={\"type\": \"json_object\"}\n",
    "    )\n",
    "\n",
    "    return chat_completion.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### test out the generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets do a test run\n",
    "input = \"(a) General. This section sets out the requirements for an escrow account that a lender establishes in connection with a federally related mortgage loan. It sets limits for escrow accounts using calculations based on monthly payments and disbursements within a calendar year. If an escrow account involves biweekly or any other payment period, the requirements in this section shall be modified accordingly. A Public Guidance Document entitled “Biweekly Payments - Example” provides examples of biweekly accounting and a Public Guidance Document entitled “Annual Escrow Account Disclosure Statement - Example” provides examples of a 3-year accounting cycle that may be used in accordance with paragraph (c)(9) of this section. A Public Guidance Document entitled “Consumer Disclosure for Voluntary Escrow Account Payments” provides a model disclosure format that originators and servicers are encouraged, but not required, to provide to consumers when the originator or servicer anticipates a substantial increase in disbursements from the escrow account after the first year of the loan. The disclosures in that model format may be combined with or included in the Initial Escrow Account Statement required in § 1024.17(g).\"\n",
    "qa_pairs = generate_qa_pairs(SYSTEM_PROMPT, input)\n",
    "qa_pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the chunk and and our generation of 10 q/a pairs data for each chunk, we will effectively have about 410 q/a pairs from the 41 chunks. This is a good start for the training data. We can always add more data later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_chunks = {}\n",
    "qa_chunks[\"data\"] = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# WARNING: This code will take a long time to run (about 7 min) and will use up a lot of OpenAI credits\n",
    "# WARNING: This code will rewrite the existing qa_chunks.json file\n",
    "if False:\n",
    "    import json\n",
    "    count = 0\n",
    "    last_chunk__ending_index= 0\n",
    "    for chunk in chunks:\n",
    "        qa_pairs = generate_qa_pairs(SYSTEM_PROMPT, chunk)\n",
    "        if isinstance(qa_pairs, str):\n",
    "            qa_pairs = json.loads(qa_pairs)\n",
    "        qa_chunk = {\n",
    "                    \"id\": count,\n",
    "                    \"metadata\":{\n",
    "                        \"text\": chunk,\n",
    "                        \"length\": len(chunk),\n",
    "                        \"start_index\": last_chunk__ending_index,\n",
    "                        \"end_index\": last_chunk__ending_index + len(chunk)\n",
    "                    }, \n",
    "                    \"data\": qa_pairs.get('data')}\n",
    "        count+=1\n",
    "        last_chunk__ending_index += len(chunk)\n",
    "        qa_chunks.get('data').append(qa_chunk)\n",
    "        \n",
    "        # save qa_chunks to a json file so we dont loose the data\n",
    "        with open('qa_chunks.json', 'w') as f:\n",
    "            json.dump(qa_chunks, f)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Q/A pairs generated:  368\n",
      "Last chunk id:  40\n"
     ]
    }
   ],
   "source": [
    "# read the qa_chunks from the json file\n",
    "import json\n",
    "with open('generated_qa_data/qa_chunks.json') as f:\n",
    "    generated_qa_chunks = json.load(f)\n",
    "\n",
    "total_qa_generated = 0\n",
    "for chunk in generated_qa_chunks.get('data'):\n",
    "    total_qa_generated += len(chunk.get('data'))\n",
    "print(\"Total Q/A pairs generated: \", total_qa_generated)\n",
    "print(\"Last chunk id: \", generated_qa_chunks.get('data')[-1].get('id'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# since we didnt get expected number of Q/A pairs, lets see what happened\n",
    "for chunk in qa_chunks.get('data'):\n",
    "    print(\"Chunk id: \", chunk.get('id'), \"Q/A pairs: \", len(chunk.get('data')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_chunks.get('data')[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating additional data\n",
    "\n",
    "The generated q/a pairs are less than 400, so we will generate more data (we need atlease 1000 sample points).\n",
    "\n",
    "This time rather then using GPT-3.5 turbo we can use a smaller model as we are not trying to create Q/A from a piece of text, rather we are just generating a similar q/a pairs using the already generated q/a pair with more capable models.\n",
    "\n",
    "for this part we will use llama3-8b-instruct model for the generation of the q/a pairs.\n",
    "\n",
    "lets pull the model using ollama\n",
    "```ollama pull llama3```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ollama pull llama3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import ollama\n",
    "# since we are using smaller model (8b) and like openai's api, we dont have a guarentee of Json output, we will generate the similar output one at a time\n",
    "\n",
    "def generate_similar_question(question, additional_context=None):\n",
    "    \n",
    "    message = f'please reright the question in other words: {question}\\n note: only respond with the question, no answer needed. You can simplify the question as well if needed.'\n",
    "    if additional_context:\n",
    "        message = f'{additional_context} {message}'\n",
    "        \n",
    "    response = ollama.chat(\n",
    "        model='llama3',\n",
    "        messages=[\n",
    "            {\n",
    "                'role': 'user',\n",
    "                'content': message\n",
    "            }\n",
    "        ],\n",
    "        \n",
    "        )\n",
    "    new_question = response['message']['content']\n",
    "    return new_question\n",
    "\n",
    "\n",
    "def generate_10_similar_questions(orignal_question, additional_context=None):\n",
    "\n",
    "    questions = []\n",
    "    print(f\"Generating similar questions like '{orignal_question}'\")\n",
    "    while True:\n",
    "        new_question = generate_similar_question(\n",
    "            orignal_question, additional_context)\n",
    "\n",
    "        # we see some wierd behavour from ollama api, when your request text is the same as the previous one, it may return the same text\n",
    "        # hence we will skip to save the same text\n",
    "        if new_question in questions:\n",
    "            continue\n",
    "        questions.append(new_question)\n",
    "        time.sleep(0.5)\n",
    "        print(\".\", end=\"\")\n",
    "        if len(questions) == 10:\n",
    "            break\n",
    "        \n",
    "    return questions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_10_similar_questions(\"Why is the sky blue?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see above how the local llm is generation similar questions to the origninal, now we will use the same function to generate more data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_10_similar_questions(\n",
    "    \"What is the definition of Aggregate analysis in the context of escrow account analysis?\",\n",
    "    additional_context=\"remove the term escrow account from your resposne\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# WARNING: This code will take a long time to run (about 60 min on macbook m2 pro 32 gb) and will use up a lot of local Compute Resource\n",
    "# now lets do the same for all the qenerated Q/A pairs, and save them to a new json file\n",
    "if False:\n",
    "    for chunk in generated_qa_chunks.get('data'):\n",
    "        for qa_pair in chunk.get('data'):\n",
    "            new_questions = generate_10_similar_questions(qa_pair.get('q'))\n",
    "            qa_pair['similar_questions'] = new_questions\n",
    "        \n",
    "        # save qa_chunks to a json file so we dont loose the data\n",
    "        with open('qa_chunks_with_similar_questions.json', 'w') as f:\n",
    "            json.dump(generated_qa_chunks, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generated_qa_chunks.get('data')[0].get('data')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have 3000+ q/a pairs after we pair the generated data with the GPT-3.5-turbo generated data. ideally it will be good to have similar answers generated as well but in the interest of time we will just use the questions generated by the llama3-8b-instruct model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'question': 'What are the requirements for an escrow account established by a lender in connection with a federally related mortgage loan?', 'answer': 'The requirements for an escrow account set limits based on monthly payments and disbursements within a calendar year. If the escrow account involves a different payment period like biweekly, the requirements are modified accordingly.', 'chunk_id': 0, 'relevent_text': '(a) General. This section sets out the requirements for an escrow account that a lender establishes in connection with a federally related mortgage loan. It sets limits for escrow accounts using calculations based on monthly payments and disbursements within a calendar year. If an escrow account involves biweekly or any other payment period, the requirements in this section shall be modified accordingly. A Public Guidance Document entitled “Biweekly Payments - Example” provides examples of biweekly accounting and a Public Guidance Document entitled “Annual Escrow Account Disclosure Statement - Example” provides examples of a 3-year accounting cycle that may be used in accordance with paragraph (c)(9) of this section. A Public Guidance Document entitled “Consumer Disclosure for Voluntary Escrow Account Payments” provides a model disclosure format that originators and servicers are encouraged, but not required, to provide to consumers when the originator or servicer anticipates a substantial increase in disbursements from the escrow account after the first year of the loan. The disclosures in that model format may be combined with or included in the Initial Escrow Account Statement required in §\\xa01024.17(g).'}\n",
      "8096\n"
     ]
    }
   ],
   "source": [
    "# load the saved qa_chunks_with_similar_questions.json file\n",
    "import json\n",
    "with open('generated_qa_data/qa_chunks_with_similar_questions.json') as f:\n",
    "    generated_qa_chunks = json.load(f)\n",
    "\n",
    "### lets bring everything together, and generate the final dataset for fine-tuning the model.\n",
    "# we need a csv file with cloums: question, answer, chunk_id, relevent_text\n",
    "qa = []\n",
    "for chunk in generated_qa_chunks.get('data'):\n",
    "    for qa_pair in chunk.get('data'):\n",
    "        # orignial q/a pair (generated using gpt-3.5-turbo)\n",
    "        relevant_pair = {\n",
    "            \"question\": qa_pair.get('q'),\n",
    "            \"answer\": qa_pair.get('a'),\n",
    "            \"chunk_id\": chunk.get('id'),\n",
    "            \"relevent_text\": chunk.get('metadata').get('text'),\n",
    "        }\n",
    "        irrelevant_pair = {\n",
    "            \"question\": qa_pair.get('q'),\n",
    "            \"answer\": \"The document does not contain and relevant information for your query\",\n",
    "            \"chunk_id\": chunk.get('id'),\n",
    "            \"relevent_text\": \"No relevant information found\",\n",
    "        }\n",
    "        qa.append(relevant_pair)\n",
    "        qa.append(irrelevant_pair)\n",
    "        \n",
    "        # adding similar questions to the orignal questions to the dataset\n",
    "        # these questions were generated using llama3\n",
    "        # add positive examples\n",
    "        for similar_question in qa_pair.get('similar_questions'):\n",
    "            new_pair = {\n",
    "                \"question\": similar_question,\n",
    "                \"answer\": qa_pair.get('a'),\n",
    "                \"chunk_id\": chunk.get('id'),\n",
    "                \"relevent_text\": chunk.get('metadata').get('text'),\n",
    "            }\n",
    "            qa.append(new_pair)\n",
    "        # add no relevent_text examples\n",
    "        for similar_question in qa_pair.get('similar_questions'):\n",
    "            new_pair = {\n",
    "                \"question\": similar_question,\n",
    "                \"answer\": \"No relevant information found in the document\",\n",
    "                \"chunk_id\": chunk.get('id'),\n",
    "                \"relevent_text\": \"No relevant information found\",\n",
    "            }\n",
    "            qa.append(new_pair)\n",
    "        \n",
    "print(qa[0])\n",
    "print(len(qa))\n",
    "\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving the pairs to a csv file\n",
    "# Save the data to a csv file.\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(qa)\n",
    "\n",
    "data_save_path = 'generated_qa_data/relevant_non_relevant_context_data.json'\n",
    "\n",
    "df.to_csv(data_save_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading the data to make sure it was saved correctly\n",
    "df = pd.read_csv(data_save_path)\n",
    "# adding an extra colum \"input\", and renaming the colum question to \"input\"and answer to \"output\"\n",
    "df = df.drop(columns=['chunk_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>relevent_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the requirements for an escrow accoun...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What are the requirements for an escrow accoun...</td>\n",
       "      <td>The document does not contain and relevant inf...</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What are the regulatory standards and guidelin...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What are the regulations or guidelines that go...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What are the key criteria that must be met whe...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>What are the key guidelines or rules that a le...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>What are the key conditions and guidelines tha...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What are the essential guidelines or standards...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>What are the key factors that a lender must co...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>What are the key criteria for setting up an es...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>What are the essential criteria for a lender's...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>What are the key provisions or standards that ...</td>\n",
       "      <td>The requirements for an escrow account set lim...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>What are the regulatory standards and guidelin...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>What are the regulations or guidelines that go...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What are the key criteria that must be met whe...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>What are the key guidelines or rules that a le...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What are the key conditions and guidelines tha...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>What are the essential guidelines or standards...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>What are the key factors that a lender must co...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>What are the key criteria for setting up an es...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>What are the essential criteria for a lender's...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>What are the key provisions or standards that ...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Are there any specific guidance documents rela...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Are there any specific guidance documents rela...</td>\n",
       "      <td>The document does not contain and relevant inf...</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Are there guidelines or resources provided by ...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Are there any regulatory guidelines or publica...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Are there specific regulatory guidelines or do...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Are there any regulatory guidance documents ad...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Are there any specific guidelines or documenta...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Are there specific guidelines or resource docu...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Are there specific guidelines or resources for...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Are there any relevant guidelines or resources...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Are there any regulatory guidelines or guidanc...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Are there any regulatory guidelines or documen...</td>\n",
       "      <td>Yes, there are Public Guidance Documents title...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Are there guidelines or resources provided by ...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Are there any regulatory guidelines or publica...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Are there specific regulatory guidelines or do...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Are there any regulatory guidance documents ad...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Are there any specific guidelines or documenta...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Are there specific guidelines or resource docu...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Are there specific guidelines or resources for...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Are there any relevant guidelines or resources...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Are there any regulatory guidelines or guidanc...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Are there any regulatory guidelines or documen...</td>\n",
       "      <td>No relevant information found in the document</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>What does the 'Consumer Disclosure for Volunta...</td>\n",
       "      <td>The 'Consumer Disclosure for Voluntary Escrow ...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>What does the 'Consumer Disclosure for Volunta...</td>\n",
       "      <td>The document does not contain and relevant inf...</td>\n",
       "      <td>No relevant information found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>What information does the \"Consumer Disclosure...</td>\n",
       "      <td>The 'Consumer Disclosure for Voluntary Escrow ...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>What does the \"Consumer Disclosure for Volunta...</td>\n",
       "      <td>The 'Consumer Disclosure for Voluntary Escrow ...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>What information or clarification does the \"Co...</td>\n",
       "      <td>The 'Consumer Disclosure for Voluntary Escrow ...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>What information or instructions does the \"Con...</td>\n",
       "      <td>The 'Consumer Disclosure for Voluntary Escrow ...</td>\n",
       "      <td>(a) General. This section sets out the require...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             question  \\\n",
       "0   What are the requirements for an escrow accoun...   \n",
       "1   What are the requirements for an escrow accoun...   \n",
       "2   What are the regulatory standards and guidelin...   \n",
       "3   What are the regulations or guidelines that go...   \n",
       "4   What are the key criteria that must be met whe...   \n",
       "5   What are the key guidelines or rules that a le...   \n",
       "6   What are the key conditions and guidelines tha...   \n",
       "7   What are the essential guidelines or standards...   \n",
       "8   What are the key factors that a lender must co...   \n",
       "9   What are the key criteria for setting up an es...   \n",
       "10  What are the essential criteria for a lender's...   \n",
       "11  What are the key provisions or standards that ...   \n",
       "12  What are the regulatory standards and guidelin...   \n",
       "13  What are the regulations or guidelines that go...   \n",
       "14  What are the key criteria that must be met whe...   \n",
       "15  What are the key guidelines or rules that a le...   \n",
       "16  What are the key conditions and guidelines tha...   \n",
       "17  What are the essential guidelines or standards...   \n",
       "18  What are the key factors that a lender must co...   \n",
       "19  What are the key criteria for setting up an es...   \n",
       "20  What are the essential criteria for a lender's...   \n",
       "21  What are the key provisions or standards that ...   \n",
       "22  Are there any specific guidance documents rela...   \n",
       "23  Are there any specific guidance documents rela...   \n",
       "24  Are there guidelines or resources provided by ...   \n",
       "25  Are there any regulatory guidelines or publica...   \n",
       "26  Are there specific regulatory guidelines or do...   \n",
       "27  Are there any regulatory guidance documents ad...   \n",
       "28  Are there any specific guidelines or documenta...   \n",
       "29  Are there specific guidelines or resource docu...   \n",
       "30  Are there specific guidelines or resources for...   \n",
       "31  Are there any relevant guidelines or resources...   \n",
       "32  Are there any regulatory guidelines or guidanc...   \n",
       "33  Are there any regulatory guidelines or documen...   \n",
       "34  Are there guidelines or resources provided by ...   \n",
       "35  Are there any regulatory guidelines or publica...   \n",
       "36  Are there specific regulatory guidelines or do...   \n",
       "37  Are there any regulatory guidance documents ad...   \n",
       "38  Are there any specific guidelines or documenta...   \n",
       "39  Are there specific guidelines or resource docu...   \n",
       "40  Are there specific guidelines or resources for...   \n",
       "41  Are there any relevant guidelines or resources...   \n",
       "42  Are there any regulatory guidelines or guidanc...   \n",
       "43  Are there any regulatory guidelines or documen...   \n",
       "44  What does the 'Consumer Disclosure for Volunta...   \n",
       "45  What does the 'Consumer Disclosure for Volunta...   \n",
       "46  What information does the \"Consumer Disclosure...   \n",
       "47  What does the \"Consumer Disclosure for Volunta...   \n",
       "48  What information or clarification does the \"Co...   \n",
       "49  What information or instructions does the \"Con...   \n",
       "\n",
       "                                               answer  \\\n",
       "0   The requirements for an escrow account set lim...   \n",
       "1   The document does not contain and relevant inf...   \n",
       "2   The requirements for an escrow account set lim...   \n",
       "3   The requirements for an escrow account set lim...   \n",
       "4   The requirements for an escrow account set lim...   \n",
       "5   The requirements for an escrow account set lim...   \n",
       "6   The requirements for an escrow account set lim...   \n",
       "7   The requirements for an escrow account set lim...   \n",
       "8   The requirements for an escrow account set lim...   \n",
       "9   The requirements for an escrow account set lim...   \n",
       "10  The requirements for an escrow account set lim...   \n",
       "11  The requirements for an escrow account set lim...   \n",
       "12      No relevant information found in the document   \n",
       "13      No relevant information found in the document   \n",
       "14      No relevant information found in the document   \n",
       "15      No relevant information found in the document   \n",
       "16      No relevant information found in the document   \n",
       "17      No relevant information found in the document   \n",
       "18      No relevant information found in the document   \n",
       "19      No relevant information found in the document   \n",
       "20      No relevant information found in the document   \n",
       "21      No relevant information found in the document   \n",
       "22  Yes, there are Public Guidance Documents title...   \n",
       "23  The document does not contain and relevant inf...   \n",
       "24  Yes, there are Public Guidance Documents title...   \n",
       "25  Yes, there are Public Guidance Documents title...   \n",
       "26  Yes, there are Public Guidance Documents title...   \n",
       "27  Yes, there are Public Guidance Documents title...   \n",
       "28  Yes, there are Public Guidance Documents title...   \n",
       "29  Yes, there are Public Guidance Documents title...   \n",
       "30  Yes, there are Public Guidance Documents title...   \n",
       "31  Yes, there are Public Guidance Documents title...   \n",
       "32  Yes, there are Public Guidance Documents title...   \n",
       "33  Yes, there are Public Guidance Documents title...   \n",
       "34      No relevant information found in the document   \n",
       "35      No relevant information found in the document   \n",
       "36      No relevant information found in the document   \n",
       "37      No relevant information found in the document   \n",
       "38      No relevant information found in the document   \n",
       "39      No relevant information found in the document   \n",
       "40      No relevant information found in the document   \n",
       "41      No relevant information found in the document   \n",
       "42      No relevant information found in the document   \n",
       "43      No relevant information found in the document   \n",
       "44  The 'Consumer Disclosure for Voluntary Escrow ...   \n",
       "45  The document does not contain and relevant inf...   \n",
       "46  The 'Consumer Disclosure for Voluntary Escrow ...   \n",
       "47  The 'Consumer Disclosure for Voluntary Escrow ...   \n",
       "48  The 'Consumer Disclosure for Voluntary Escrow ...   \n",
       "49  The 'Consumer Disclosure for Voluntary Escrow ...   \n",
       "\n",
       "                                        relevent_text  \n",
       "0   (a) General. This section sets out the require...  \n",
       "1                       No relevant information found  \n",
       "2   (a) General. This section sets out the require...  \n",
       "3   (a) General. This section sets out the require...  \n",
       "4   (a) General. This section sets out the require...  \n",
       "5   (a) General. This section sets out the require...  \n",
       "6   (a) General. This section sets out the require...  \n",
       "7   (a) General. This section sets out the require...  \n",
       "8   (a) General. This section sets out the require...  \n",
       "9   (a) General. This section sets out the require...  \n",
       "10  (a) General. This section sets out the require...  \n",
       "11  (a) General. This section sets out the require...  \n",
       "12                      No relevant information found  \n",
       "13                      No relevant information found  \n",
       "14                      No relevant information found  \n",
       "15                      No relevant information found  \n",
       "16                      No relevant information found  \n",
       "17                      No relevant information found  \n",
       "18                      No relevant information found  \n",
       "19                      No relevant information found  \n",
       "20                      No relevant information found  \n",
       "21                      No relevant information found  \n",
       "22  (a) General. This section sets out the require...  \n",
       "23                      No relevant information found  \n",
       "24  (a) General. This section sets out the require...  \n",
       "25  (a) General. This section sets out the require...  \n",
       "26  (a) General. This section sets out the require...  \n",
       "27  (a) General. This section sets out the require...  \n",
       "28  (a) General. This section sets out the require...  \n",
       "29  (a) General. This section sets out the require...  \n",
       "30  (a) General. This section sets out the require...  \n",
       "31  (a) General. This section sets out the require...  \n",
       "32  (a) General. This section sets out the require...  \n",
       "33  (a) General. This section sets out the require...  \n",
       "34                      No relevant information found  \n",
       "35                      No relevant information found  \n",
       "36                      No relevant information found  \n",
       "37                      No relevant information found  \n",
       "38                      No relevant information found  \n",
       "39                      No relevant information found  \n",
       "40                      No relevant information found  \n",
       "41                      No relevant information found  \n",
       "42                      No relevant information found  \n",
       "43                      No relevant information found  \n",
       "44  (a) General. This section sets out the require...  \n",
       "45                      No relevant information found  \n",
       "46  (a) General. This section sets out the require...  \n",
       "47  (a) General. This section sets out the require...  \n",
       "48  (a) General. This section sets out the require...  \n",
       "49  (a) General. This section sets out the require...  "
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# experimental (not used for training, but could be used in future)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "NEGATIVE_SYSTEM_PROMPT = \"\"\"\n",
    "# Role\n",
    "you are a worldclass llm training data generator. \n",
    "\n",
    "# Task\n",
    "You are generating negative examples for finetuning our own llm model.\n",
    "User gives you a piece of text and you have to generate a question that is NOT relevant to the text.\n",
    "Your job is to generate 10 questions that are not relevant to the text.\n",
    "\n",
    "# Specifics\n",
    "Context irrelevance for the Q/A is really important, else our business will lose money and reputation as you are serving our customers\n",
    "\n",
    "         \n",
    "# Example format:\n",
    "{ data: [ irrelevant_question1, irrelevant_question2, ...]}\n",
    "]}\n",
    "\n",
    "# Notes\n",
    "- Make sure to generate JSON format for the irrelevant questions\n",
    "- Generate 10 irrelevant questions \n",
    "- The irrelevant question can be anything that is NOT DIRECTLY related to the user prompt.\n",
    "         \n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate negative examples for the dataset\n",
    "# provided a piece of text, generate a questions that are not relevant to the text\n",
    "\n",
    "def generate_irrelevant_questions(system_prompt, text, temperature=0.5):\n",
    "\n",
    "    chat_completion = client.chat.completions.create(\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": system_prompt\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": text\n",
    "            }\n",
    "        ],\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        response_format={\"type\": \"json_object\"},\n",
    "        temperature=temperature\n",
    "    )\n",
    "\n",
    "    data = chat_completion.choices[0].message.content\n",
    "    try:\n",
    "        data = json.loads(data)\n",
    "    except ValueError:\n",
    "        data = None\n",
    "        \n",
    "    return data\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "irrelivant_questions = generate_irrelevant_questions(NEGATIVE_SYSTEM_PROMPT,\"Aggregate (or) composite analysis, hereafter called aggregate analysis, means an accounting method a servicer uses in conducting an escrow account analysis by computing the sufficiency of escrow account funds by analyzing the account as a whole. Appendix E to this part sets forth examples of aggregate escrow account analyses.\\n\\nAnnual escrow account statement means a statement containing all of the information set forth in § 1024.17(i). As noted in § 1024.17(i), a servicer shall submit an annual escrow account statement to the borrower within 30 calendar days of the end of the escrow account computation year, after conducting an escrow account analysis.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': ['What is the best type of cheese for making a grilled cheese sandwich?', 'Have you ever tried bungee jumping?', 'Do you prefer to drink tea or coffee in the morning?', 'How many hours of sleep do you typically get each night?', 'What is your favorite TV show to binge-watch?', 'Have you ever traveled to Antarctica?', 'Do you believe in aliens?', 'What is the capital of Australia?', 'What is your opinion on pineapple pizza?', 'Have you ever participated in a hot dog eating contest?']}\n"
     ]
    }
   ],
   "source": [
    "print(irrelivant_questions)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "context_relevance_prompt= \"\"\"# Role\n",
    "your job is to see if the user query is relevant to the context provided.\n",
    "\n",
    "# Instruction\n",
    "respond only with a number between: 0.0 (not relevant at all) to 1.0 \n",
    "0.0 being not relevant at all and 1.0 being very relevant\n",
    "Note the relevance of the user query to the context provided should be your ONLY concern\n",
    "\n",
    "# Output Example (json)\n",
    "{{relevance: <float>}}\n",
    "\n",
    "# Context\n",
    "{text} \"\"\"\n",
    "\n",
    "def validate_context_relevance(question, text):\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": context_relevance_prompt.format(text=text)\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": question\n",
    "            }\n",
    "        ],\n",
    "        response_format={\"type\": \"json_object\"}\n",
    "    )\n",
    "    data =  response.choices[0].message.content\n",
    "    try:\n",
    "        if isinstance(data, str):\n",
    "            data = json.loads(data)\n",
    "        elif isinstance(data, dict):\n",
    "            pass\n",
    "        else: \n",
    "            raise ValueError\n",
    "    \n",
    "    except ValueError:\n",
    "        data = None\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': ['What is the best type of cheese for making a grilled cheese sandwich?', 'Have you ever tried bungee jumping?', 'Do you prefer to drink tea or coffee in the morning?', 'How many hours of sleep do you typically get each night?', 'What is your favorite TV show to binge-watch?', 'Have you ever traveled to Antarctica?', 'Do you believe in aliens?', 'What is the capital of Australia?', 'What is your opinion on pineapple pizza?', 'Have you ever participated in a hot dog eating contest?']}\n",
      "{'relevance': 0.0}\n"
     ]
    }
   ],
   "source": [
    "print(irrelivant_questions)\n",
    "valid = validate_context_relevance(irrelivant_questions[\"data\"][0], \"Aggregate (or) composite analysis, hereafter called aggregate analysis, means an accounting method a servicer uses in conducting an escrow account analysis by computing the sufficiency of escrow account funds by analyzing the account as a whole. Appendix E to this part sets forth examples of aggregate escrow account analyses.\\n\\nAnnual escrow account statement means a statement containing all of the information set forth in § 1024.17(i). As noted in § 1024.17(i), a servicer shall submit an annual escrow account statement to the borrower within 30 calendar days of the end of the escrow account computation year, after conducting an escrow account analysis.\")\n",
    "print(valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_irrilevant_generated_questions(irrelivant_questions):\n",
    "    valid_irrilevant_q=[] \n",
    "    for question in irrelivant_questions:\n",
    "        print(\"validating question: \", question)\n",
    "        response = validate_context_relevance(question, \"Aggregate (or) composite analysis, hereafter called aggregate analysis, means an accounting method a servicer uses in conducting an escrow account analysis by computing the sufficiency of escrow account funds by analyzing the account as a whole. Appendix E to this part sets forth examples of aggregate escrow account analyses.\\n\\nAnnual escrow account statement means a statement containing all of the information set forth in § 1024.17(i). As noted in § 1024.17(i), a servicer shall submit an annual escrow account statement to the borrower within 30 calendar days of the end of the escrow account computation year, after conducting an escrow account analysis.\")\n",
    "        \n",
    "        if response.get('relevance') < 0.5:\n",
    "            valid_irrilevant_q.append({\"question\": question, \"relevance\": response.get('relevance')})\n",
    "    \n",
    "    return valid_irrilevant_q\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "{'data': ['What is the best type of cheese for making a grilled cheese sandwich?', 'Have you ever tried bungee jumping?', 'Do you prefer to drink tea or coffee in the morning?', 'How many hours of sleep do you typically get each night?', 'What is your favorite TV show to binge-watch?', 'Have you ever traveled to Antarctica?', 'Do you believe in aliens?', 'What is the capital of Australia?', 'What is your opinion on pineapple pizza?', 'Have you ever participated in a hot dog eating contest?']}\n",
      "validating question:  What is the best type of cheese for making a grilled cheese sandwich?\n",
      "validating question:  Have you ever tried bungee jumping?\n",
      "validating question:  Do you prefer to drink tea or coffee in the morning?\n",
      "validating question:  How many hours of sleep do you typically get each night?\n",
      "validating question:  What is your favorite TV show to binge-watch?\n",
      "validating question:  Have you ever traveled to Antarctica?\n",
      "validating question:  Do you believe in aliens?\n",
      "validating question:  What is the capital of Australia?\n",
      "validating question:  What is your opinion on pineapple pizza?\n",
      "validating question:  Have you ever participated in a hot dog eating contest?\n",
      "len of irrilevant questions:  1\n",
      "len of valid irrilevant questions:  10\n",
      "[{'question': 'What is the best type of cheese for making a grilled cheese sandwich?', 'relevance': 0.0}, {'question': 'Have you ever tried bungee jumping?', 'relevance': 0.0}, {'question': 'Do you prefer to drink tea or coffee in the morning?', 'relevance': 0.0}, {'question': 'How many hours of sleep do you typically get each night?', 'relevance': 0.0}, {'question': 'What is your favorite TV show to binge-watch?', 'relevance': 0.0}, {'question': 'Have you ever traveled to Antarctica?', 'relevance': 0.0}, {'question': 'Do you believe in aliens?', 'relevance': 0.0}, {'question': 'What is the capital of Australia?', 'relevance': 0.0}, {'question': 'What is your opinion on pineapple pizza?', 'relevance': 0.0}, {'question': 'Have you ever participated in a hot dog eating contest?', 'relevance': 0.0}]\n"
     ]
    }
   ],
   "source": [
    "print(len(irrelivant_questions))\n",
    "print(irrelivant_questions)\n",
    "valid_irrilevant_q = validate_irrilevant_generated_questions(irrelivant_questions[\"data\"])\n",
    "print(\"len of irrilevant questions: \", len(irrelivant_questions))\n",
    "print(\"len of valid irrilevant questions: \", len(valid_irrilevant_q))\n",
    "print(valid_irrilevant_q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}