04-Complex-Docs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "60ec6048-44e4-4118-b16a-9c4c9cc78a3b",
   "metadata": {},
   "source": [
    "# How to deal with complex/large Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9281ac79-47cd-49d4-bdd4-7f5c173a947d",
   "metadata": {},
   "source": [
    "In the previous notebook, we developed a solution for various types of files and data formats commonly found in organizations, and this covers 90% of the use cases. However, you will find that there are issues when dealing with questions that require answers from complex files. The complexity of these files arises from their length and the way information is distributed within them. Large documents are always a challenge for Search Engines.\n",
    "\n",
    "One example of such complex files is Technical Specification Guides or Product Manuals, which can span hundreds of pages and contain information in the form of images, tables, forms, and more. Books are also complex due to their length and the presence of images or tables.\n",
    "\n",
    "These files are typically in PDF format. To better handle these PDFs, we need a smarter parsing method that treats each document as a special source and processes them page by page. The objective is to obtain more accurate and faster answers from our system. Fortunately, there are usually not many of these types of documents in an organization, allowing us to make exceptions and treat them differently.\n",
    "\n",
    "If your use case is just PDFs, for example, you can just use [PyPDF library](https://pypi.org/project/pypdf/) or [Azure AI Document Intelligence SDK (former Form Recognizer)](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview?view=doc-intel-3.0.0), vectorize using OpenAI API and push the content to a vector-based index. And this is problably the simplest and fastest way to go.  However if your use case entails connecting to a datalake, or Sharepoint libraries or any other document data source with thousands of documents with multiple file types and that can change dynamically, then you would want to use the Ingestion and Document Cracking and AI-Enrichment capabilities of Azure Search engine, Notebooks 1-3, and avoid a lot of painful custom code. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "15f6044e-463f-4988-bc46-a3c3d641c15c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import time\n",
    "import requests\n",
    "import random\n",
    "from collections import OrderedDict\n",
    "import urllib.request\n",
    "from tqdm import tqdm\n",
    "import langchain\n",
    "from langchain.embeddings import AzureOpenAIEmbeddings\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.chains.qa_with_sources import load_qa_with_sources_chain\n",
    "\n",
    "from common.utils import parse_pdf, read_pdf_files, text_to_base64\n",
    "from common.prompts import COMBINE_QUESTION_PROMPT, COMBINE_PROMPT, COMBINE_PROMPT_TEMPLATE\n",
    "from common.utils import (\n",
    "    get_search_results,\n",
    "    model_tokens_limit,\n",
    "    num_tokens_from_docs,\n",
    "    num_tokens_from_string\n",
    ")\n",
    "\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "    \n",
    "os.makedirs(\"data/books/\",exist_ok=True)\n",
    "    \n",
    "\n",
    "BLOB_CONTAINER_NAME = \"books\"\n",
    "BASE_CONTAINER_URL = \"https://datasetsgptsmartsearch.blob.core.windows.net/\" + BLOB_CONTAINER_NAME + \"/\"\n",
    "LOCAL_FOLDER = \"./data/books/\"\n",
    "\n",
    "MODEL = \"gpt-35-turbo-16k\" # options: gpt-35-turbo, gpt-35-turbo-16k, gpt-4, gpt-4-32k\n",
    "\n",
    "os.makedirs(LOCAL_FOLDER,exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "331692ba-b68e-4b99-9bae-5057da9a389d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "594ff0d4-56e3-4bed-843d-28c7a092069b",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedder = AzureOpenAIEmbeddings(deployment=\"text-embedding-ada-002\", chunk_size=1) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb87c647-158c-4f85-b569-5b9462f06c83",
   "metadata": {},
   "source": [
    "## 1 - Manual Document Cracking with Push to Vector-based Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75551868-1546-421b-a14e-e42618d88e61",
   "metadata": {},
   "source": [
    "Within our demo storage account, we have a container named `books`, which holds 5 books of different lengths, languages, and complexities. Let's create a `cogsrch-index-books-vector` and load it with the pages of all these books.\n",
    "\n",
    "We begin by downloading these books to our local machine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0999e24b-6a75-4fa1-9a5f-426cf0f0bdba",
   "metadata": {},
   "outputs": [],
   "source": [
    "books = [\"Azure_Cognitive_Search_Documentation.pdf\", \n",
    "         \"Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf\",\n",
    "         \"Fundamentals_of_Physics_Textbook.pdf\",\n",
    "         \"Made_To_Stick.pdf\",\n",
    "         \"Pere_Riche_Pere_Pauvre.pdf\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd867b2f-b5a1-443c-aa0a-ce914a66b3c9",
   "metadata": {},
   "source": [
    "Let's download the files to the local `./data/` folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3554f0b7-fee8-4446-a155-5d22dc0f0888",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:02<00:00,  2.02it/s]\n"
     ]
    }
   ],
   "source": [
    "for book in tqdm(books):\n",
    "    book_url = BASE_CONTAINER_URL + book + os.environ['BLOB_SAS_TOKEN']\n",
    "    urllib.request.urlretrieve(book_url, LOCAL_FOLDER+ book)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "788cc0db-9dae-45f2-8943-2b6fa32fcc75",
   "metadata": {},
   "source": [
    "### What to use: pyPDF or AI Documment Intelligence API (Form Recognizer)?\n",
    "\n",
    "In `utils.py` there is a **parse_pdf()** function. This utility function can parse local files using PyPDF library and can also parse local or from_url PDFs files using Azure AI Document Intelligence (Former Form Recognizer).\n",
    "\n",
    "If `form_recognizer=False`, the function will parse the PDF using the python pyPDF library, which 75% of the time does a good job.<br>\n",
    "\n",
    "Setting `form_recognizer=True`, is the best (and slower) parsing method using AI Documment Intelligence API (former known as Form Recognizer). You can specify the prebuilt model to use, the default is `model=\"prebuilt-document\"`. However, if you have a complex document with tables, charts and figures , you can try\n",
    "`model=\"prebuilt-layout\"`, and it will capture all of the nuances of each page (it takes longer of course).\n",
    "\n",
    "**Note: Many PDFs are scanned images. For example, any signed contract that was scanned and saved as PDF will NOT be parsed by pyPDF. Only AI Documment Intelligence API will work.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c1c63a2f-7a53-4346-8a1f-483cfd159d34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting Text from Azure_Cognitive_Search_Documentation.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 32.466733 seconds\n",
      "Azure_Cognitive_Search_Documentation.pdf contained 1947 pages\n",
      "\n",
      "Extracting Text from Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 1.773100 seconds\n",
      "Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf contained 357 pages\n",
      "\n",
      "Extracting Text from Fundamentals_of_Physics_Textbook.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 96.595367 seconds\n",
      "Fundamentals_of_Physics_Textbook.pdf contained 1450 pages\n",
      "\n",
      "Extracting Text from Made_To_Stick.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 7.461382 seconds\n",
      "Made_To_Stick.pdf contained 225 pages\n",
      "\n",
      "Extracting Text from Pere_Riche_Pere_Pauvre.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 0.995148 seconds\n",
      "Pere_Riche_Pere_Pauvre.pdf contained 225 pages\n",
      "\n"
     ]
    }
   ],
   "source": [
    "book_pages_map = dict()\n",
    "for book in books:\n",
    "    print(\"Extracting Text from\",book,\"...\")\n",
    "    \n",
    "    # Capture the start time\n",
    "    start_time = time.time()\n",
    "    \n",
    "    # Parse the PDF\n",
    "    book_path = LOCAL_FOLDER+book\n",
    "    book_map = parse_pdf(file=book_path, form_recognizer=False, verbose=True)\n",
    "    book_pages_map[book]= book_map\n",
    "    \n",
    "    # Capture the end time and Calculate the elapsed time\n",
    "    end_time = time.time()\n",
    "    elapsed_time = end_time - start_time\n",
    "\n",
    "    print(f\"Parsing took: {elapsed_time:.6f} seconds\")\n",
    "    print(f\"{book} contained {len(book_map)} pages\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de0a722-ae0c-4b57-802a-518f5d4d93fd",
   "metadata": {},
   "source": [
    "Now let's check a random page of each book to make sure the parsing was done correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f2a5d62f-b664-4662-a6c9-a1eb2a3c5e11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure_Cognitive_Search_Documentation.pdf \n",
      " chunk text: 3. In Description, choose a field that provides details that might help someone  ...\n",
      "\n",
      "Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf \n",
      " chunk text: 14\n",
      "11:59 A.M.\n",
      "The rest of Sherrie’s morning proceeded fairly well. A tal-\n",
      "ented  ...\n",
      "\n",
      "Fundamentals_of_Physics_Textbook.pdf \n",
      " chunk text: 192-2INSTANTANEOUS VELOCITY AND SPEED\n",
      "Figure 2-6(a) The x(t) curve for an elevat ...\n",
      "\n",
      "Made_To_Stick.pdf \n",
      " chunk text: the artistic feel of the movie will change. When st ars are hired to play \n",
      "the p ...\n",
      "\n",
      "Pere_Riche_Pere_Pauvre.pdf \n",
      " chunk text: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for bookname,bookmap in book_pages_map.items():\n",
    "    print(bookname,\"\\n\",\"chunk text:\",bookmap[random.randint(10, 50)][2][:80],\"...\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcdc1ee-71fc-49d2-8e7c-0964bc3a4370",
   "metadata": {},
   "source": [
    "As we can see above, all books were parsed except `Pere_Riche_Pere_Pauvre.pdf` (this book is \"Rich Dad, Poor Dad\" written in French), why? Well, as we mentioned above, this book was scanned, so each page is an image and with a very unique font. We need a good PDF parser with good OCR capabilities in order to extract the content of this PDF. \n",
    "Let's try to parse this book again, but this time using Azure Document Intelligence API (former Form Recognizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "801c6bc2-467c-4418-aa7e-ef89a1e20e1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting text using Azure Document Intelligence\n",
      "CPU times: user 10.6 s, sys: 160 ms, total: 10.8 s\n",
      "Wall time: 42.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "book = \"Pere_Riche_Pere_Pauvre.pdf\"\n",
    "book_path = LOCAL_FOLDER+book\n",
    "book_map = parse_pdf(file=book_path, form_recognizer=True, model=\"prebuilt-document\",from_url=False, verbose=True)\n",
    "book_pages_map[book]= book_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "52f8b4b9-c167-49cf-b88b-c6be705200b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Note: If the above command throws an error - Create another form recognizer resource in the azure portal in the same resource group, don't delete it. And try again.\n",
    "# This seems to be a transient error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "97f9c5bb-c44b-4a4d-9780-591f9f8d128a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pere_Riche_Pere_Pauvre.pdf \n",
      " chunk text: règles différentes. Les employés sont perdants; les patrons et les investisseurs ...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(book,\"\\n\",\"chunk text:\",book_map[random.randint(10, 50)][2][:80],\"...\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c279dfb-4fed-41b8-89e1-0ca2cefbcdc9",
   "metadata": {},
   "source": [
    "As demonstrated above, Azure Document Intelligence proves to be superior to pyPDF. **For production scenarios, we strongly recommend using Azure Document Intelligence consistently**. When doing so, it's important to make a wise choice between the available models, such as \"prebuilt-document,\" \"prebuilt-layout,\" or others. You can find more information on model selection [HERE](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/choose-model-feature?view=doc-intel-3.0.0).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f5f9b7d-99e6-426d-a47e-343c7e8b492e",
   "metadata": {},
   "source": [
    "## Create Vector-based index\n",
    "\n",
    "\n",
    "Now that we have the content of the book's chunks (each page of each book) in the dictionary `book_pages_map`, let's create the Vector-based index in our Azure Search Engine where this content is going to land"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7d46e7c5-49c4-40f3-bb2d-79a9afeab4b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "book_index_name = \"cogsrch-index-books-vector\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1b07e84b-d306-4bc9-9124-e64f252dd7b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Create Azure Search Vector-based Index\n",
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}\n",
    "params = {'api-version': os.environ['AZURE_SEARCH_API_VERSION']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2df4db6b-969b-4b91-963f-9334e17a4e3c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "index_payload = {\n",
    "    \"name\": book_index_name,\n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"filterable\": \"true\" },\n",
    "        {\"name\": \"title\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunk\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunkVector\",\"type\": \"Collection(Edm.Single)\",\"searchable\": \"true\",\"retrievable\": \"true\",\"dimensions\": 1536,\"vectorSearchConfiguration\": \"vectorConfig\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"page_num\",\"type\": \"Edm.Int32\",\"searchable\": \"false\",\"retrievable\": \"true\"},\n",
    "        \n",
    "    ],\n",
    "    \"vectorSearch\": {\n",
    "        \"algorithmConfigurations\": [\n",
    "            {\n",
    "                \"name\": \"vectorConfig\",\n",
    "                \"kind\": \"hnsw\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": {\n",
    "                        \"fieldName\": \"title\"\n",
    "                    },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        {\n",
    "                            \"fieldName\": \"chunk\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"prioritizedKeywordsFields\": []\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + book_index_name,\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "36691ff0-c4c8-49d0-bfa8-3e076ece0ce5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to debug errors\n",
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc7dda9-4725-410e-9465-54f0298fc758",
   "metadata": {},
   "source": [
    "## Upload the Document chunks and its vectors to the Vector-Based Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d73e7600-7902-48d4-b199-9d9dc0a17aa0",
   "metadata": {},
   "source": [
    "The following code will iterate over each chunk of each book and use the Azure Search Rest API upload method to insert each document with its corresponding vector (using OpenAI embedding model) to the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f5c8aa55-1b60-4057-93db-0d4a89993a57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from Azure_Cognitive_Search_Documentation.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 357/357 [01:36<00:00,  3.71it/s]s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from Fundamentals_of_Physics_Textbook.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1450/1450 [07:10<00:00,  3.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from Made_To_Stick.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 225/225 [01:03<00:00,  3.56it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from Pere_Riche_Pere_Pauvre.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 225/225 [01:02<00:00,  3.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 6s, sys: 4.32 s, total: 3min 10s\n",
      "Wall time: 19min 29s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for bookname,bookmap in book_pages_map.items():\n",
    "    print(\"Uploading chunks from\",bookname)\n",
    "    for page in tqdm(bookmap):\n",
    "        try:\n",
    "            page_num = page[0] + 1\n",
    "            content = page[2]\n",
    "            book_url = BASE_CONTAINER_URL + bookname\n",
    "            upload_payload = {\n",
    "                \"value\": [\n",
    "                    {\n",
    "                        \"id\": text_to_base64(bookname + str(page_num)),\n",
    "                        \"title\": f\"{bookname}_page_{str(page_num)}\",\n",
    "                        \"chunk\": content,\n",
    "                        \"chunkVector\": embedder.embed_query(content if content!=\"\" else \"-------\"),\n",
    "                        \"name\": bookname,\n",
    "                        \"location\": book_url,\n",
    "                        \"page_num\": page_num,\n",
    "                        \"@search.action\": \"upload\"\n",
    "                    },\n",
    "                ]\n",
    "            }\n",
    "\n",
    "            r = requests.post(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + book_index_name + \"/docs/index\",\n",
    "                                 data=json.dumps(upload_payload), headers=headers, params=params)\n",
    "            if r.status_code != 200:\n",
    "                print(r.status_code)\n",
    "                print(r.text)\n",
    "        except Exception as e:\n",
    "            print(\"Exception:\",e)\n",
    "            print(content)\n",
    "            continue"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "715cddcf-af7b-4006-a047-853fc7a66be3",
   "metadata": {},
   "source": [
    "## Query the Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8b408798-5527-44ca-9dba-cad2ee726aca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# QUESTION = \"what normally rich dad do that is different from poor dad?\"\n",
    "# QUESTION = \"Tell me a summary of the book Boundaries\"\n",
    "# QUESTION = \"Dime que significa la radiacion del cuerpo negro\"\n",
    "# QUESTION = \"what is the acronym of the main point of Made to Stick book\"\n",
    "QUESTION = \"Tell me a python example of how do I push documents with vectors to an index using the python SDK?\"\n",
    "# QUESTION = \"who won the soccer worldcup in 1994?\" # this question should have no answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1b182ade-0ddd-47a1-b1eb-2cbf435c317f",
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_indexes = [book_index_name]\n",
    "\n",
    "ordered_results = get_search_results(QUESTION, vector_indexes, \n",
    "                                        k=10,\n",
    "                                        reranker_threshold=1,\n",
    "                                        vector_search=True, \n",
    "                                        similarity_k=10,\n",
    "                                        query_vector = embedder.embed_query(QUESTION)\n",
    "                                        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd2f3f2-2d66-4bd4-b90b-d30970b71af4",
   "metadata": {},
   "source": [
    "**Note**: that we are picking a larger k=10 since these chunks are NOT of 5000 chars each like prior notebooks, but instead each page is a chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "410ff796-dab1-4817-a3a5-82eeff6c0c57",
   "metadata": {},
   "outputs": [],
   "source": [
    "COMPLETION_TOKENS = 1000\n",
    "llm = AzureChatOpenAI(deployment_name=MODEL, temperature=0.5, max_tokens=COMPLETION_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "744aba20-b3fd-4286-8d58-2ddfccc77734",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of chunks: 10\n"
     ]
    }
   ],
   "source": [
    "top_docs = []\n",
    "for key,value in ordered_results.items():\n",
    "    location = value[\"location\"] if value[\"location\"] is not None else \"\"\n",
    "    top_docs.append(Document(page_content=value[\"chunk\"], metadata={\"source\": location+os.environ['BLOB_SAS_TOKEN']}))\n",
    "        \n",
    "print(\"Number of chunks:\",len(top_docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "db1c4d56-8c2d-47d6-8717-810f156f1c0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System prompt token count: 1669\n",
      "Max Completion Token count: 1000\n",
      "Combined docs (context) token count: 3485\n",
      "--------\n",
      "Requested token count: 6154\n",
      "Token limit for gpt-35-turbo-16k : 16384\n",
      "Chain Type selected: stuff\n"
     ]
    }
   ],
   "source": [
    "# Calculate number of tokens of our docs\n",
    "if(len(top_docs)>0):\n",
    "    tokens_limit = model_tokens_limit(MODEL) # this is a custom function we created in common/utils.py\n",
    "    prompt_tokens = num_tokens_from_string(COMBINE_PROMPT_TEMPLATE) # this is a custom function we created in common/utils.py\n",
    "    context_tokens = num_tokens_from_docs(top_docs) # this is a custom function we created in common/utils.py\n",
    "    \n",
    "    requested_tokens = prompt_tokens + context_tokens + COMPLETION_TOKENS\n",
    "    \n",
    "    chain_type = \"map_reduce\" if requested_tokens > 0.9 * tokens_limit else \"stuff\"  \n",
    "    \n",
    "    print(\"System prompt token count:\",prompt_tokens)\n",
    "    print(\"Max Completion Token count:\", COMPLETION_TOKENS)\n",
    "    print(\"Combined docs (context) token count:\",context_tokens)\n",
    "    print(\"--------\")\n",
    "    print(\"Requested token count:\",requested_tokens)\n",
    "    print(\"Token limit for\", MODEL, \":\", tokens_limit)\n",
    "    print(\"Chain Type selected:\", chain_type)\n",
    "        \n",
    "else:\n",
    "    print(\"NO RESULTS FROM AZURE SEARCH\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "62cf3a3f-2b4d-4806-8b92-eb982c52b0cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "if chain_type == \"stuff\":\n",
    "    chain = load_qa_with_sources_chain(llm, chain_type=chain_type, \n",
    "                                       prompt=COMBINE_PROMPT)\n",
    "elif chain_type == \"map_reduce\":\n",
    "    chain = load_qa_with_sources_chain(llm, chain_type=chain_type, \n",
    "                                       question_prompt=COMBINE_QUESTION_PROMPT,\n",
    "                                       combine_prompt=COMBINE_PROMPT,\n",
    "                                       return_intermediate_steps=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "3b412c56-650f-4ca4-a868-9954f83679fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 21.9 ms, sys: 0 ns, total: 21.9 ms\n",
      "Wall time: 7.75 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Try with other language as well\n",
    "response = chain({\"input_documents\": top_docs, \"question\": QUESTION, \"language\": \"English\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "63f07b08-87bd-4518-b2f2-03ee1096f59f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "To push documents with vectors to an index using the Python SDK, you can use the following code example:\n",
       "\n",
       "```python\n",
       "from azure.core.credentials import AzureKeyCredential\n",
       "from azure.search.documents import SearchClient\n",
       "\n",
       "# Set up the connection\n",
       "endpoint = \"your_search_service_endpoint\"\n",
       "index_name = \"your_index_name\"\n",
       "api_key = \"your_api_key\"\n",
       "\n",
       "# Create the search client\n",
       "search_client = SearchClient(endpoint=endpoint, index_name=index_name, credential=AzureKeyCredential(api_key))\n",
       "\n",
       "# Define your documents with vectors\n",
       "documents = [\n",
       "    {\n",
       "        \"@search.action\": \"upload\",\n",
       "        \"id\": \"1\",\n",
       "        \"title\": \"Document 1\",\n",
       "        \"vector\": [0.1, 0.2, 0.3]\n",
       "    },\n",
       "    {\n",
       "        \"@search.action\": \"upload\",\n",
       "        \"id\": \"2\",\n",
       "        \"title\": \"Document 2\",\n",
       "        \"vector\": [0.4, 0.5, 0.6]\n",
       "    },\n",
       "    # Add more documents as needed\n",
       "]\n",
       "\n",
       "# Upload the documents to the index\n",
       "result = search_client.upload_documents(documents=documents)\n",
       "\n",
       "# Check if the upload succeeded\n",
       "print(\"Upload of new documents succeeded:\", result[0].succeeded)\n",
       "```\n",
       "\n",
       "This code creates a search client using the endpoint, index name, and API key. It then defines the documents to be uploaded, including a unique ID, a title, and a vector. The `@search.action` field specifies that the action is an upload. Finally, the `upload_documents` method is called to push the documents to the index.\n",
       "\n",
       "Please note that you need to replace `\"your_search_service_endpoint\"`, `\"your_index_name\"`, and `\"your_api_key\"` with your own values.\n",
       "\n",
       "[1]<sup><a href=\"https://demodatasetsp.blob.core.windows.net/books/Azure_Cognitive_Search_Documentation.pdf?sv=2022-11-02&ss=bf&srt=sco&sp=rl&se=2025-11-06T23:27:04Z&st=2023-11-06T15:27:04Z&spr=https&sig=IxmYt1nWtSI0MtBHeQBC1t%2F4VeoN19HqQM1Xu6tvacU%3D\" target=\"_blank\">Source</a></sup>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(response['output_text']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3941796c-7655-4888-a358-8a62e380bd7e",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "In this notebook we learned how to deal with complex and large Documents and make them available for Q&A over them using [Hybrid Search](https://learn.microsoft.com/en-us/azure/search/search-get-started-vector#hybrid-search) (text + vector search).\n",
    "\n",
    "We also learned the power of Azure Document Inteligence API and why it is recommended for production scenarios where manual Document parsing (instead of Azure Search Indexer Document Cracking) is necessary.\n",
    "\n",
    "Using Azure Cognitive Search with its Vector capabilities and hybrid search features eliminates the need for other vector databases such as Weaviate, Qdrant, Milvus, Pinecone, and so on.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d9a7d1-f029-416b-8eb2-00a8afb9151d",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "So far we have learned how to use OpenAI vectors and completion APIs in order to get an excelent answer from our documents stored in Azure Cognitive Search. This is the backbone for a GPT Smart Search Engine.\n",
    "\n",
    "However, we are missing something: **How to have a conversation with this engine?**\n",
    "\n",
    "On the next Notebook, we are going to understand the concept of **memory**. This is necessary in order to have a chatbot that can establish a conversation with the user. Without memory, there is no real conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20ef4e89-a769-4e68-9f28-66fab1ec873f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}