03-Quering-AOpenAI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d59d527f-1100-45ff-b051-5f7c9029d94d",
   "metadata": {},
   "source": [
    "# Queries with and without Azure OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb9a9444-dc90-4fc3-aea7-8ee918301aba",
   "metadata": {},
   "source": [
    "Now that we have our Search Engine loaded **from two different data sources in two diferent indexes**, we are going to try some example queries and then use Azure OpenAI service to see if we can get even better results.\n",
    "\n",
    "The idea is that a user can ask a question about Computer Science (first datasource/index) or about Covid (second datasource/index), and the engine will respond accordingly.\n",
    "This **Multi-Index** demo, mimics the scenario where a company loads multiple type of documents of different types and about completly different topics and the search engine must respond with the most relevant results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71f6c7e3-9037-4b1e-ae17-1deaa27b9c08",
   "metadata": {},
   "source": [
    "## Set up variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8e50b404-a061-49e7-a3c7-c6eabc98ff0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib\n",
    "import requests\n",
    "import random\n",
    "from collections import OrderedDict\n",
    "from IPython.display import display, HTML\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.llms import AzureOpenAI\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.vectorstores import FAISS\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain.chains.qa_with_sources import load_qa_with_sources_chain\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "\n",
    "from app.prompts import COMBINE_QUESTION_PROMPT, COMBINE_PROMPT\n",
    "from app.utils import model_tokens_limit, num_tokens_from_docs\n",
    "\n",
    "# Demo Datasource Blob Storage. Change if using your own data\n",
    "DATASOURCE_SAS_TOKEN = \"?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\"\n",
    "\n",
    "# Don't mess with these unless you really know what you are doing\n",
    "AZURE_SEARCH_API_VERSION = '2021-04-30-Preview'\n",
    "AZURE_OPENAI_API_VERSION = \"2023-03-15-preview\"\n",
    "\n",
    "# Change these below with your own services credentials\n",
    "\n",
    "AZURE_SEARCH_ENDPOINT = \"Enter your Azure Cognitive Search Endpoint ...\"\n",
    "AZURE_SEARCH_KEY = \"Enter your Azure Cognitive Search Key ...\" # Make sure is the MANAGEMENT KEY no the query key\n",
    "AZURE_OPENAI_ENDPOINT = \"Enter your Azure OpenAI Endpoint ...\"\n",
    "AZURE_OPENAI_API_KEY = \"Enter your Azure OpenAI Key ...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2f2c22f8-79ab-405c-95e8-77a1978e53bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': AZURE_SEARCH_KEY}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9297d29b-1f61-4dce-858e-bf4272172dba",
   "metadata": {},
   "source": [
    "## Multi-Index Search queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5a46e2d3-298a-4708-83de-9e108b1a117a",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Index that we are going to query (from Notebook 01 and 02)\n",
    "index1_name = \"cogsrch-index-files\"\n",
    "index2_name = \"cogsrch-index-csv\"\n",
    "indexes = [index1_name, index2_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c62ebb2-d7be-4bfb-b1ba-4db86c11839a",
   "metadata": {},
   "source": [
    "Try questions that you think might be answered or addressed in computer science papers in 2020-2021 or that can be addressed by medical publications about COVID in 2020-201. Try comparing the results with the open version of ChatGPT.<br>\n",
    "The idea is that the answers using Azure OpenAI only looks at the information contained on these publications.\n",
    "\n",
    "**Example Questions you can ask**:\n",
    "- What is CLP?\n",
    "- How Markov chains work?\n",
    "- What are some examples of reinforcement learning?\n",
    "- What are the main risk factors for Covid-19?\n",
    "- What medicine reduces inflamation in the lungs?\n",
    "- Why Covid doesn't affect kids that much compared to adults?\n",
    "- Does chloroquine really works against covid?\n",
    "- tell me Use cases where I can use deep learning to solve it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b9b53c14-19bd-451f-aa43-7ad27ccfeead",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"What are some examples of reinforcement learning?\" "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d925eb-7f9c-429e-a62a-4c37d7702caf",
   "metadata": {},
   "source": [
    "### Search on both indexes individually and aggragate results\n",
    "\n",
    "**Note**: In order to standarize the indexes we are setting 4 mandatory fields to be present on each index: id, title, content, pages, language. These fields must be present in each index so that each document can be treated the same along the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "faf2e30f-e71f-4533-ab52-27d048b80a89",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://azure-cog-search-cstevuxaqrxcm.search.windows.net/indexes/cogsrch-index-files/docs?api-version=2021-04-30-Preview&search=What are some examples of reinforcement learning?&select=*&$top=5&queryLanguage=en-us&queryType=semantic&semanticConfiguration=my-semantic-config&$count=true&speller=lexicon&answers=extractive|count-3&captions=extractive|highlight-false\n",
      "200\n",
      "Results Found: 9822, Results Returned: 5\n",
      "https://azure-cog-search-cstevuxaqrxcm.search.windows.net/indexes/cogsrch-index-csv/docs?api-version=2021-04-30-Preview&search=What are some examples of reinforcement learning?&select=*&$top=5&queryLanguage=en-us&queryType=semantic&semanticConfiguration=my-semantic-config&$count=true&speller=lexicon&answers=extractive|count-3&captions=extractive|highlight-false\n",
      "200\n",
      "Results Found: 40503, Results Returned: 5\n"
     ]
    }
   ],
   "source": [
    "agg_search_results = []\n",
    "\n",
    "for index in indexes:\n",
    "    url = AZURE_SEARCH_ENDPOINT + '/indexes/'+ index + '/docs'\n",
    "    url += '?api-version={}'.format(AZURE_SEARCH_API_VERSION)\n",
    "    url += '&search={}'.format(QUESTION)\n",
    "    url += '&select=*'\n",
    "    url += '&$top=5'  # You can change this to anything you need/want\n",
    "    url += '&queryLanguage=en-us'\n",
    "    url += '&queryType=semantic'\n",
    "    url += '&semanticConfiguration=my-semantic-config'\n",
    "    url += '&$count=true'\n",
    "    url += '&speller=lexicon'\n",
    "    url += '&answers=extractive|count-3'\n",
    "    url += '&captions=extractive|highlight-false'\n",
    "\n",
    "    resp = requests.get(url, headers=headers)\n",
    "    print(url)\n",
    "    print(resp.status_code)\n",
    "\n",
    "    search_results = resp.json()\n",
    "    agg_search_results.append(search_results)\n",
    "    print(\"Results Found: {}, Results Returned: {}\".format(search_results['@odata.count'], len(search_results['value'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fd0fe5-4ee0-42e2-a920-72b93a407389",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Display the top results (from both searches) based on the score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9e938337-602d-4b61-8141-b8c92a5d91da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Top Answers</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h4>Top Results</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0212/0212025v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0212025v1  [cs.AI]  10 Dec 2002</a> - score: 1.91</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "A novel reinforcement learning method called plannable RL (pRL) is proposed here. pRL builds a simple model, which is used to search for macro actions. The simplicity of the model makes planning computationally inexpensive. It is shown that pRL finds an optimal policy, and that plannable macro actions found by pRL are near-optimal."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">rl-survey.dvi</a> - score: 1.72</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "techniques like this have been used in several reinforcement learning algorithmsincluding the interval exploration method (kaelbling, 1993b) (described shortly), the ex-ploration bonus in dyna (sutton, 1990), curiosity-driven exploration (schmidhuber, 1991a),and the exploration mechanism in prioritized sweeping (moore & atkeson, 1993).2.2.2 …"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0105/0105027v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0105027v1  [cs.LG]  17 May 2001</a> - score: 1.59</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "One realistic off-line scenario in reinforcement learning is when the data processing and optimization (learning) module is separated (physically) from the data acquisition module (agent). Say we have an ultra-light micro-sensor connected to a central computer ."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0604/0604010v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0604010v1  [cs.AI]  5 Apr 2006</a> - score: 1.57</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "While in general trading off exploration and exploitation in reinforcement learning is hard, under some formulations relatively simple solutions exist."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0207/0207073v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">arXiv:cs/0207073v1  [cs.NI]  21 Jul 2002</a> - score: 1.55</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "The setting of the reinforcement learning problem offers several advantages, including loop resolution, multi-path forwarding capability, cost-sensitive routing, and minimizing state overhead, while maintaining the incremental spirit of current backbone routing algorithms."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h5><a href=\"https://demodatasetsp.blob.core.windows.net/litcovid/train.csv?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">Real-time neural network scheduling of emergency medical mask production during COVID-19.</a> - score: 1.26</h5>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "In this paper, we propose an end-to-end neural network, which takes a sequence of production tasks as inputs and produces a schedule of tasks in a real-time manner. The network is trained by reinforcement learning using the negative total tardiness as the reward signal."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(HTML('<h4>Top Answers</h4>'))\n",
    "\n",
    "for search_results in agg_search_results:\n",
    "    for result in search_results['@search.answers']:\n",
    "        if result['score'] > 0.5: # Show answers that are at least 50% of the max possible score=1\n",
    "            display(HTML('<h5>' + 'Answer - score: ' + str(round(result['score'],2)) + '</h5>'))\n",
    "            display(HTML(result['text']))\n",
    "            \n",
    "print(\"\\n\\n\")\n",
    "display(HTML('<h4>Top Results</h4>'))\n",
    "\n",
    "content = dict()\n",
    "ordered_content = OrderedDict()\n",
    "\n",
    "\n",
    "for search_results in agg_search_results:\n",
    "    for result in search_results['value']:\n",
    "        if result['@search.rerankerScore'] > 1: # Filter results that are at least 25% of the max possible score=4\n",
    "            content[result['id']]={\n",
    "                                    \"title\": result['title'],\n",
    "                                    \"chunks\": result['pages'],\n",
    "                                    \"language\": result['language'], \n",
    "                                    \"caption\": result['@search.captions'][0]['text'],\n",
    "                                    \"score\": result['@search.rerankerScore'],\n",
    "                                    \"name\": result['metadata_storage_name'], \n",
    "                                    \"location\": result['metadata_storage_path']                  \n",
    "                                }\n",
    "    \n",
    "#After results have been filtered we will Sort and add them as an Ordered list\\n\",\n",
    "for id in sorted(content, key= lambda x: content[x][\"score\"], reverse=True):\n",
    "    ordered_content[id] = content[id]\n",
    "    url = ordered_content[id]['location'] + DATASOURCE_SAS_TOKEN\n",
    "    title = str(ordered_content[id]['title']) if (ordered_content[id]['title']) else ordered_content[id]['name']\n",
    "    score = str(round(ordered_content[id]['score'],2))\n",
    "    display(HTML('<h5><a href=\"'+ url + '\">' + title + '</a> - score: '+ score + '</h5>'))\n",
    "    display(HTML(ordered_content[id]['caption']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52a6d3e6-afb2-4fa7-96d3-69bc2373ded5",
   "metadata": {},
   "source": [
    "## Comments on Query results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84e02227-6a92-4944-86f8-6c1e38d90fe4",
   "metadata": {},
   "source": [
    "As seen above the semantic search feature of Azure Cognitive Search service is good. It gives us some answers and also the top results with the corresponding file and the paragraph where the answers is possible located.\n",
    "\n",
    "Let's see if we can make this better with Azure OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8df3e6d4-9a09-4b0f-b328-238738ccfaec",
   "metadata": {},
   "source": [
    "# Using Azure OpenAI\n",
    "\n",
    "To use OpenAI to get a better answer to our question, the thought process is: let's send the the documents of the search result to the GPT model and let it understand the document's content and provide a better response.\n",
    "\n",
    "We will use a genius library call **LangChain** that wraps a lot of boiler plate code.\n",
    "Langchain is one library that does a lot of the prompt engineering for us under the hood, for more information see [here](https://python.langchain.com/en/latest/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e7c720e-ece1-45ad-9d01-2dfd15c182bb",
   "metadata": {},
   "source": [
    "## A gentle intro to chaining LLMs and prompt engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bcd7028-5a6c-4296-8c85-4f420d408d69",
   "metadata": {},
   "source": [
    "Chains are what you get by connecting one or more large language models (LLMs) in a logical way. (Chains can be built of entities other than LLMs but for now, let’s stick with this definition for simplicity).\n",
    "\n",
    "Azure OpenAI is a type of LLM (provider) that you can use but there are others like Cohere, Huggingface, etc.\n",
    "\n",
    "Chains can be simple (i.e. Generic) or specialized (i.e. Utility).\n",
    "\n",
    "* Generic — A single LLM is the simplest chain. It takes an input prompt and the name of the LLM and then uses the LLM for text generation (i.e. output for the prompt).\n",
    "\n",
    "Here’s an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eea62a7d-7e0e-4a93-a89c-20c96560c665",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_BASE\"] = os.environ[\"AZURE_OPENAI_ENDPOINT\"] = AZURE_OPENAI_ENDPOINT\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.environ[\"AZURE_OPENAI_API_KEY\"] = AZURE_OPENAI_API_KEY\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"] = AZURE_OPENAI_API_VERSION\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "13df9247-e784-4e04-9475-55e672efea47",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create our LLM model\n",
    "# Make sure you have the deployment named \"gpt-35-turbo\" for the model \"gpt-35-turbo (0301)\". \n",
    "# Use \"gpt-4\" if you have it available.\n",
    "MODEL = \"gpt-35-turbo\" # options: gpt-35-turbo, gpt-4, gpt-4-32k\n",
    "llm = AzureChatOpenAI(deployment_name=MODEL, temperature=0, max_tokens=500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7b0520b9-83b2-49fd-ad84-624cb0f15ce1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer the following question: \"What are some examples of reinforcement learning?\". Give your response in French\n"
     ]
    }
   ],
   "source": [
    "# Now we create a simple prompt template\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"question\", \"language\"],\n",
    "    template='Answer the following question: \"{question}\". Give your response in {language}',\n",
    ")\n",
    "\n",
    "print(prompt.format(question=QUESTION, language=\"French\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dcc7dae3-6b88-4ea6-be43-b178ebc559dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'What are some examples of reinforcement learning?',\n",
       " 'language': 'French',\n",
       " 'text': \"Quelques exemples d'apprentissage par renforcement sont les voitures autonomes qui apprennent à conduire en utilisant des récompenses pour des actions positives telles que rester sur la voie et éviter les collisions, les robots qui apprennent à accomplir des tâches en utilisant des récompenses pour des actions réussies, et les jeux informatiques où l'ordinateur apprend à jouer en utilisant des récompenses pour des mouvements gagnants.\"}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And finnaly we create our first generic chain\n",
    "chain_chat = LLMChain(llm=llm, prompt=prompt)\n",
    "chain_chat({\"question\": QUESTION, \"language\": \"French\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50ed014c-0c6b-448c-b995-fe7970b92ad5",
   "metadata": {},
   "source": [
    "Great!!, now you know how to create a simple prompt and use a chain in order to answer a general question using ChatGPT knowledge!. \n",
    "\n",
    "It is important to note that we rarely use generic chains as standalone chains. More often they are used as building blocks for Utility chains (as we will see next). Also important to notice is that we are NOT using our documents or the result of the Azure Search yet, just the knowledge of ChatGPT on the data it was trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c48038-b1af-4228-8ffb-720e554fd3b2",
   "metadata": {
    "tags": []
   },
   "source": [
    "**The second type of Chains are Utility:**\n",
    "\n",
    "* Utility — These are specialized chains, comprised of many LLMs to help solve a specific task. For example, LangChain supports some end-to-end chains (such as [QA_WITH_SOURCES](https://python.langchain.com/en/latest/modules/chains/index_examples/qa_with_sources.html) for QnA Doc retrieval, Summarization, etc) and some specific ones (such as GraphQnAChain for creating, querying, and saving graphs). \n",
    "\n",
    "We will look at one specific chain called **qa_with_sources** in this workshop for digging deeper and solve our use case of enhancing the results of Azure Cognitive Search."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0454ddb-44d8-4fa9-929a-5e5563dd28f8",
   "metadata": {},
   "source": [
    "\n",
    "But before dealing with the utility chain needed, we need to deal first with this problem: **the content of the search result files is or can be very lengthy, more than the allowed tokens allowed by the GPT Azure OpenAI models**. So what we need to do is: split in chunks, vectorize those chunks and do a vector semantic search to get the top chunks in order to provide the best and not too lenghy context to the LLM.\n",
    "\n",
    "Notice that **the documents chunks are already done in Azure Search**. *ordered_content* dictionary (created a few cells above) contains the pages (chunks) of each document. So we don't really need to chunk them again, but we still need to make sure that we can be as fast as possible and that we are below the max allowed input token limits of our selected OpenAI model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8f7b41d2-65b0-4058-8a46-c76cf6960720",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of chunks: 70\n"
     ]
    }
   ],
   "source": [
    "# Iterate over each of the results chunks and create a LangChain Document class to use further in the pipeline\n",
    "docs = []\n",
    "for key,value in ordered_content.items():\n",
    "    for page in value[\"chunks\"]:\n",
    "        docs.append(Document(page_content=page, metadata={\"source\": value[\"location\"]}))\n",
    "        \n",
    "print(\"Number of chunks:\",len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "345d35f6-b7c8-4fda-a9e2-94a7da16a18e",
   "metadata": {},
   "source": [
    "We need now to calculate the number of tokens for all the chunks combined to decide what to do:\n",
    "1) Should we embed to vectors and do cosine similarity because there is too much data to fit on the prompt as context?\n",
    "2) If embedding is the decision, should we use OpenAI embedding model or a local parallelizable faster embedder?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "62bd5169-f273-4c66-a91b-6de990dad244",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Custom token limit for gpt-35-turbo : 3000\n",
      "Combined docs tokens count: 77112\n"
     ]
    }
   ],
   "source": [
    "# Calculate number of tokens of our docs\n",
    "if(len(docs)>0):\n",
    "    tokens_limit = model_tokens_limit(MODEL) # this is a custom function we created in app/utils.py\n",
    "    num_tokens = num_tokens_from_docs(docs) # this is a custom function we created in app/utils.py\n",
    "    print(\"Custom token limit for\", MODEL, \":\", tokens_limit)\n",
    "    print(\"Combined docs tokens count:\",num_tokens)\n",
    "        \n",
    "else:\n",
    "    print(\"NO RESULTS FROM AZURE SEARCH\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5403dee-a4c4-420c-9819-68151d973695",
   "metadata": {},
   "source": [
    "Now, depending of the amount of chunks/pages returned from the search result, which is very related to the size of the documents returned, \n",
    "we pick the embedding model that give us fast results.\n",
    "\n",
    "The logic is, if there is less than 50 chunks (of 5000 chars each) to vectorize, then we use \n",
    "OpenAI models, which currently don't offer batch processing, but if there is more than 50 chunks we use a BERT based in-memory model that processes in batches and in parallel (it is recommended a VM of at least 4 cores).\n",
    "\n",
    "For more information on in-memory BERT transformer models that you can use, see [HERE](https://www.sbert.net/docs/pretrained_models.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a03f1f10-32b0-4c1e-8a0e-eee1b1d29ce7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "client=SentenceTransformer(\n",
      "  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: DistilBertModel \n",
      "  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})\n",
      "  (2): Dense({'in_features': 768, 'out_features': 512, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})\n",
      ") model_name='distiluse-base-multilingual-cased-v2' cache_folder=None model_kwargs={}\n",
      "Token count after similarity search: 4644\n",
      "Chain Type selected: map_reduce\n",
      "CPU times: user 19.6 s, sys: 2.38 s, total: 22 s\n",
      "Wall time: 10.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if num_tokens > tokens_limit:\n",
    "    # Select the Embedder model\n",
    "    if len(docs) < 50:\n",
    "        # OpenAI models are accurate but slower, they also only (for now) accept one text at a time (chunk_size)\n",
    "        embedder = OpenAIEmbeddings(deployment=\"text-embedding-ada-002\", chunk_size=1) \n",
    "    else:\n",
    "        # Bert based models are faster (3x-10x) but not as great in accuracy as OpenAI models\n",
    "        # Since this repo supports Multiple languages we need to use a multilingual model. \n",
    "        # But if English only is the requirement, use \"multi-qa-MiniLM-L6-cos-v1\"\n",
    "        # The fastest english model is \"all-MiniLM-L12-v2\"\n",
    "        embedder = HuggingFaceEmbeddings(model_name = 'distiluse-base-multilingual-cased-v2')\n",
    "    \n",
    "    print(embedder)\n",
    "    \n",
    "    # Create our in-memory vector database index from the chunks given by Azure Search.\n",
    "    # We are using FAISS. https://ai.facebook.com/tools/faiss/\n",
    "    db = FAISS.from_documents(docs, embedder)\n",
    "    top_docs = db.similarity_search(QUESTION, k=4)  # Return the top 4 documents\n",
    "    \n",
    "    # Now we need to recalculate the tokens count of the top results from similarity vector search\n",
    "    # in order to select the chain type: stuff (all chunks in one prompt) or \n",
    "    # map_reduce (multiple calls to the LLM to summarize/reduce the chunks and then combine them)\n",
    "    \n",
    "    num_tokens = num_tokens_from_docs(top_docs)\n",
    "    print(\"Token count after similarity search:\", num_tokens)\n",
    "    chain_type = \"map_reduce\" if num_tokens > tokens_limit else \"stuff\"\n",
    "    \n",
    "else:\n",
    "    # if total tokens is less than our limit, we don't need to vectorize and do similarity search\n",
    "    top_docs = docs\n",
    "    chain_type = \"stuff\"\n",
    "    \n",
    "print(\"Chain Type selected:\", chain_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17247488-7d14-4178-9add-31eb1afcbcbe",
   "metadata": {},
   "source": [
    "At this point we already have the top most similar chunks (in order of relevance) in **top_docs**\n",
    "\n",
    "Now we need Azure OpenAI GPT model to understand these top chunks and provide us an answer to the question."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f8c6ad1-82b8-4fd8-80a3-0276b81d7231",
   "metadata": {},
   "source": [
    "For this task, we need to come back to the Utility Chain: **qa_with_sources** that we mentioned before. See [HERE](https://python.langchain.com/en/latest/modules/chains/index_examples/qa_with_sources.html) for reference.\n",
    "\n",
    "We created our own custom prompts so we can add translation to a specified language. But, for more information on the different types of prompts for this utility chain please see [HERE](https://github.com/hwchase17/langchain/tree/master/langchain/chains/question_answering)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3ab16c86-9863-4001-89af-6819c6f3240a",
   "metadata": {},
   "outputs": [],
   "source": [
    "if chain_type == \"stuff\":\n",
    "    chain = load_qa_with_sources_chain(llm, chain_type=chain_type, \n",
    "                                       prompt=COMBINE_PROMPT)\n",
    "elif chain_type == \"map_reduce\":\n",
    "    chain = load_qa_with_sources_chain(llm, chain_type=chain_type, \n",
    "                                       question_prompt=COMBINE_QUESTION_PROMPT,\n",
    "                                       combine_prompt=COMBINE_PROMPT,\n",
    "                                       return_intermediate_steps=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "28926219-74c2-4538-8493-129463ac40a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below line if you want to check our custom COMBINE_PROMPT\n",
    "# print(chain.combine_document_chain.llm_chain.prompt.template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a1e619b8-1dcf-431b-8aad-f1696a09c2ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 25.5 ms, sys: 309 µs, total: 25.8 ms\n",
      "Wall time: 16.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Try with other language as well\n",
    "response = chain({\"input_documents\": top_docs, \"question\": QUESTION, \"language\": \"English\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8cddb1cb-a4a0-4e2f-9f0c-4216b0f232b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Azure OpenAI ChatGPT Answer:</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Some examples of reinforcement learning include shaping, hierarchical reinforcement learning, local reinforcement signals, and decomposing a huge learning problem into smaller ones. Other examples include dynamic-programming approach, Gittins allocation indices, and learning automata such as the Tsetlin automaton. Reinforcement learning has also been used in routing algorithms, such as the ant-based algorithms and the stigmergetic routing framework. The exploration-exploitation trade-off is a common dilemma in reinforcement learning. Sequential sampling theory and Bayesian decision problems are also relevant to reinforcement learning. \n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<u>Sources</u>: <sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/9605/9605103v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[1]</a></sup><sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0207/0207073v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[2]</a></sup><sup><a href=\"https://demodatasetsp.blob.core.windows.net/arxivcs/0604/0604010v1.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rltfx&se=2023-11-29T01:50:59Z&st=2023-05-10T16:50:59Z&spr=https&sig=ZT7MLy%2BnlvAxUKKj5v0RwoWObXaab3gO4ec2%2Fci6iL0%3D\">[3]</a></sup>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answer = response['output_text']\n",
    "\n",
    "display(HTML('<h4>Azure OpenAI ChatGPT Answer:</h4>'))\n",
    "display(HTML(answer.split(\"SOURCES:\")[0]))\n",
    "\n",
    "sources_list = answer.split(\"SOURCES:\")[1].replace(\" \",\"\").split(\",\")\n",
    "\n",
    "sources_html = '<u>Sources</u>: '\n",
    "for index, value in enumerate(sources_list):\n",
    "    url = value + DATASOURCE_SAS_TOKEN\n",
    "    sources_html +='<sup><a href=\"'+ url + '\">[' + str(index+1) + ']</a></sup>'\n",
    "    \n",
    "display(HTML(sources_html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "11345374-6420-4b36-b061-795d2a804c85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Chunk Summary:</b> There are a variety of reinforcement-learning techniques that work effectively on a variety of small problems. Shaping has been used in supervised-learning systems, and can be used to train hierarchical reinforcement-learning systems from the bottom up, and to alleviate problems of delayed reinforcement by decreasing the delay until the problem is well understood. Whenever possible, agents should be given reinforcement signals that are local. An agent can learn by \"watching\" another agent perform the task. Decomposing a huge learning problem into a collection of smaller ones, and providing useful reinforcement signals for the subproblems is a very powerful technique for biasing learning. One thing that keeps agents that know nothing from learning anything is that they have a hard time even finding the interesting parts of the space. These problems can be ameliorated by programming a set of \"reflexes\" that cause the agent to act initially in some way that is reasonable."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b>Chunk Summary:</b> \"Reinforcement learning (RL) [14] is a branch of machine learning that is increasingly finding use in many important applications, including routing. The ant-based algorithms of Subramanian et al. [21] and the stigmergetic routing framework described in [10] are examples of reinforcement learning algorithms for routing.\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b>Chunk Summary:</b> There are several examples of reinforcement learning discussed in the text, including the dynamic-programming approach, Gittins allocation indices, and learning automata such as the Tsetlin automaton."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b>Chunk Summary:</b> In reinforcement learning, the dilemma between selecting actions to maximise the expected return according to the current world model and to improve the world model such as to potentially be able to achieve a higher expected return is referred to as the exploration-exploitation trade-off. This has been the subject of much interest before, one of the earliest developments being the theory of sequential sampling in statistics, as developed by [1]. A more general overview of sequential decision problems from a Bayesian viewpoint is offered in [2]."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Uncomment if you want to inspect the results from map_reduce chain type, each top similar chunk summary (k=4 by default)\n",
    "\n",
    "if chain_type == \"map_reduce\":\n",
    "    for step in response['intermediate_steps']:\n",
    "        display(HTML(\"<b>Chunk Summary:</b> \" + step))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f347373a-a5be-473d-b64e-0f6b6dbcd0e0",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### This answer is way better than taking just the result from Azure Cognitive Search. So the summary is:\n",
    "- Azure Cognitive Search give us the top results (context)\n",
    "- Azure OpenAI takes these results and understand the content and uses it as context to give the best answer\n",
    "- Best of two worlds!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdc6e2fe-1c34-4952-99ad-14940f022379",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We just added a smart layer on top of Azure Cognitive Search. This is the backend for a GPT Smart Search Engine.\n",
    "\n",
    "However, we are missing something: **How to have a conversation with this engine?**\n",
    "\n",
    "On the next Notebook, we are going to understand the concept of **memory**. This is necessary in order to have a chatbot that can establish a conversation with the user. Without memory, there is no real conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4d5356b-ae05-4b41-9013-5341f248dce7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}