04-Complex-Docs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "60ec6048-44e4-4118-b16a-9c4c9cc78a3b",
   "metadata": {},
   "source": [
    "# How to deal with complex/large Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9281ac79-47cd-49d4-bdd4-7f5c173a947d",
   "metadata": {},
   "source": [
    "In the previous notebook, we developed a solution for various types of files and data formats commonly found in organizations, and this covers big majority of the use cases. However, you will find that there are issues when dealing with questions that require answers from complex files. The complexity of these files arises from their length and the way information is distributed within them. Large documents are always a challenge for Search Engines.\n",
    "\n",
    "One example of such complex files is Technical Specification Guides or Product Manuals, which can span hundreds of pages and contain information in the form of images, tables, forms, and more. Books are also complex due to their length and the presence of images or tables.\n",
    "\n",
    "These files are typically in PDF format. To better handle these PDFs, we need a smarter parsing method that treats each document as a special source and processes them page by page (1 page = 1 chunk). The objective is to obtain more accurate and faster answers from our system. Fortunately, there are usually not many of these types of documents in an organization, allowing us to make exceptions and treat them differently.\n",
    "\n",
    "If your use case is just PDFs, for example, you can just use [PyPDF library](https://pypi.org/project/pypdf/) or [Azure AI Document Intelligence SDK (former Form Recognizer)](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview?view=doc-intel-3.0.0), vectorize using OpenAI API and push the content to a vector-based index. And this is problably the simplest and fastest way to go.  However if your use case entails connecting to a datalake, or Sharepoint libraries or any other document data source with thousands of documents with multiple file types and that can change dynamically, then you would want to use the Ingestion and Document Cracking and AI-Enrichment capabilities of Azure Search engine, Notebooks 1-3, and avoid a lot of painful custom code. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "15f6044e-463f-4988-bc46-a3c3d641c15c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import json\n",
    "import time\n",
    "import requests\n",
    "import random\n",
    "import uuid\n",
    "import shutil\n",
    "import zipfile\n",
    "from collections import OrderedDict\n",
    "import urllib.request\n",
    "from tqdm import tqdm\n",
    "\n",
    "from typing import List\n",
    "\n",
    "from langchain_openai import AzureChatOpenAI, AzureOpenAIEmbeddings\n",
    "from langchain_core.runnables import ConfigurableField\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "\n",
    "from operator import itemgetter\n",
    "\n",
    "from common.utils import upload_file_to_blob, extract_zip_file, upload_directory_to_blob\n",
    "from common.utils import parse_pdf, read_pdf_files\n",
    "from common.prompts import DOCSEARCH_PROMPT_TEXT\n",
    "from common.utils import CustomAzureSearchRetriever\n",
    "\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "331692ba-b68e-4b99-9bae-5057da9a389d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa901f14-adf6-4575-8c75-72569ca4f256",
   "metadata": {},
   "source": [
    "## Upload local dataset to Blob Container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0cd2cff9-de28-4656-a154-18c5bc9975e2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./data/books.zip ... \n",
      "Extracted ./data/books.zip to ./data/temp_extract\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Uploading Files: 100%|████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.42it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Temp Folder: ./data/temp_extract removed\n",
      "CPU times: user 315 ms, sys: 190 ms, total: 506 ms\n",
      "Wall time: 4.33 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Define connection string and other parameters\n",
    "BLOB_CONTAINER_NAME = \"books\"\n",
    "BLOB_NAME = \"books.zip\"\n",
    "LOCAL_FILE_PATH = \"./data/\" + BLOB_NAME  # Path to the local file you want to upload\n",
    "upload_directory = \"./data/temp_extract\"  # Temporary directory to extract the zip file\n",
    "\n",
    "# Extract the zip file\n",
    "extract_zip_file(LOCAL_FILE_PATH, upload_directory)\n",
    "\n",
    "# Upload the extracted files and folder structure\n",
    "upload_directory_to_blob(upload_directory, BLOB_CONTAINER_NAME)\n",
    "\n",
    "# Clean up: Optionally, you can remove the temp folder after uploading\n",
    "shutil.rmtree(upload_directory)\n",
    "print(f\"Temp Folder: {upload_directory} removed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb87c647-158c-4f85-b569-5b9462f06c83",
   "metadata": {},
   "source": [
    "## Manual Document Cracking with Push to Vector-based Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "788cc0db-9dae-45f2-8943-2b6fa32fcc75",
   "metadata": {},
   "source": [
    "### What to use: pyPDF or AI Documment Intelligence API (Form Recognizer)?\n",
    "\n",
    "In `utils.py` there is a **parse_pdf()** function. This utility function can parse local files using PyPDF library and can also parse local or from_url PDFs files using Azure AI Document Intelligence (Former Form Recognizer).\n",
    "\n",
    "If `form_recognizer=False`, the function will parse the PDF using the python pyPDF library, which 75% of the time does a good job.<br>\n",
    "\n",
    "Setting `form_recognizer=True`, is the best (and slower) parsing method using AI Documment Intelligence API (former known as Form Recognizer). You can specify the prebuilt model to use, the default is `model=\"prebuilt-document\"`. However, if you have a complex document with tables, charts and figures , you can try\n",
    "`model=\"prebuilt-layout\"`, and it will capture all of the nuances of each page (it takes longer of course).\n",
    "\n",
    "**Note: Many PDFs are scanned images. For example, any signed contract that was scanned and saved as PDF will NOT be parsed by pyPDF. Only AI Documment Intelligence API will work.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0e0c21a6-bf09-48ca-b47c-27b8a2045d45",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "BLOB_NAME = \"books.zip\"\n",
    "LOCAL_FILE_PATH = \"./data/\" + BLOB_NAME  # Path to the local file you want to upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "050418ea-9b0e-4c76-8a11-e59b3d4429a0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting Text from books/Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 1.727564 seconds\n",
      "books/Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf contained 357 pages\n",
      "\n",
      "Extracting Text from books/Fundamentals_of_Physics_Textbook.pdf ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Ignoring wrong pointing object 23 0 (offset 0)\n",
      "Ignoring wrong pointing object 27 0 (offset 0)\n",
      "Ignoring wrong pointing object 53 0 (offset 0)\n",
      "Ignoring wrong pointing object 198 0 (offset 0)\n",
      "Ignoring wrong pointing object 205 0 (offset 0)\n",
      "Ignoring wrong pointing object 272 0 (offset 0)\n",
      "Ignoring wrong pointing object 307 0 (offset 0)\n",
      "Ignoring wrong pointing object 326 0 (offset 0)\n",
      "Ignoring wrong pointing object 345 0 (offset 0)\n",
      "Ignoring wrong pointing object 637 0 (offset 0)\n",
      "Ignoring wrong pointing object 638 0 (offset 0)\n",
      "Ignoring wrong pointing object 640 0 (offset 0)\n",
      "Ignoring wrong pointing object 641 0 (offset 0)\n",
      "Ignoring wrong pointing object 1638 0 (offset 0)\n",
      "Ignoring wrong pointing object 1685 0 (offset 0)\n",
      "Ignoring wrong pointing object 2511 0 (offset 0)\n",
      "Ignoring wrong pointing object 2516 0 (offset 0)\n",
      "Ignoring wrong pointing object 2780 0 (offset 0)\n",
      "Ignoring wrong pointing object 2816 0 (offset 0)\n",
      "Ignoring wrong pointing object 3617 0 (offset 0)\n",
      "Ignoring wrong pointing object 3757 0 (offset 0)\n",
      "Ignoring wrong pointing object 3759 0 (offset 0)\n",
      "Ignoring wrong pointing object 3781 0 (offset 0)\n",
      "Ignoring wrong pointing object 3789 0 (offset 0)\n",
      "Ignoring wrong pointing object 6314 0 (offset 0)\n",
      "Ignoring wrong pointing object 6327 0 (offset 0)\n",
      "Ignoring wrong pointing object 6395 0 (offset 0)\n",
      "Ignoring wrong pointing object 6424 0 (offset 0)\n",
      "Ignoring wrong pointing object 6481 0 (offset 0)\n",
      "Ignoring wrong pointing object 7720 0 (offset 0)\n",
      "Ignoring wrong pointing object 7782 0 (offset 0)\n",
      "Ignoring wrong pointing object 7972 0 (offset 0)\n",
      "Ignoring wrong pointing object 7979 0 (offset 0)\n",
      "Ignoring wrong pointing object 7990 0 (offset 0)\n",
      "Ignoring wrong pointing object 7998 0 (offset 0)\n",
      "Ignoring wrong pointing object 8336 0 (offset 0)\n",
      "Ignoring wrong pointing object 8353 0 (offset 0)\n",
      "Ignoring wrong pointing object 8803 0 (offset 0)\n",
      "Ignoring wrong pointing object 8819 0 (offset 0)\n",
      "Ignoring wrong pointing object 8945 0 (offset 0)\n",
      "Ignoring wrong pointing object 8948 0 (offset 0)\n",
      "Ignoring wrong pointing object 8951 0 (offset 0)\n",
      "Ignoring wrong pointing object 8974 0 (offset 0)\n",
      "Ignoring wrong pointing object 9022 0 (offset 0)\n",
      "Ignoring wrong pointing object 9028 0 (offset 0)\n",
      "Ignoring wrong pointing object 9127 0 (offset 0)\n",
      "Ignoring wrong pointing object 9348 0 (offset 0)\n",
      "Ignoring wrong pointing object 9557 0 (offset 0)\n",
      "Ignoring wrong pointing object 9578 0 (offset 0)\n",
      "Ignoring wrong pointing object 9595 0 (offset 0)\n",
      "Ignoring wrong pointing object 9716 0 (offset 0)\n",
      "Ignoring wrong pointing object 9747 0 (offset 0)\n",
      "Ignoring wrong pointing object 10122 0 (offset 0)\n",
      "Ignoring wrong pointing object 10132 0 (offset 0)\n",
      "Ignoring wrong pointing object 10169 0 (offset 0)\n",
      "Ignoring wrong pointing object 10196 0 (offset 0)\n",
      "Ignoring wrong pointing object 10337 0 (offset 0)\n",
      "Ignoring wrong pointing object 10426 0 (offset 0)\n",
      "Ignoring wrong pointing object 10439 0 (offset 0)\n",
      "Ignoring wrong pointing object 10446 0 (offset 0)\n",
      "Ignoring wrong pointing object 10453 0 (offset 0)\n",
      "Ignoring wrong pointing object 10561 0 (offset 0)\n",
      "Ignoring wrong pointing object 10612 0 (offset 0)\n",
      "Ignoring wrong pointing object 10722 0 (offset 0)\n",
      "Ignoring wrong pointing object 10842 0 (offset 0)\n",
      "Ignoring wrong pointing object 10864 0 (offset 0)\n",
      "Ignoring wrong pointing object 11010 0 (offset 0)\n",
      "Ignoring wrong pointing object 11023 0 (offset 0)\n",
      "Ignoring wrong pointing object 11127 0 (offset 0)\n",
      "Ignoring wrong pointing object 11139 0 (offset 0)\n",
      "Ignoring wrong pointing object 11242 0 (offset 0)\n",
      "Ignoring wrong pointing object 11270 0 (offset 0)\n",
      "Ignoring wrong pointing object 11284 0 (offset 0)\n",
      "Ignoring wrong pointing object 11324 0 (offset 0)\n",
      "Ignoring wrong pointing object 11342 0 (offset 0)\n",
      "Ignoring wrong pointing object 11350 0 (offset 0)\n",
      "Ignoring wrong pointing object 11384 0 (offset 0)\n",
      "Ignoring wrong pointing object 11418 0 (offset 0)\n",
      "Ignoring wrong pointing object 11430 0 (offset 0)\n",
      "Ignoring wrong pointing object 11780 0 (offset 0)\n",
      "Ignoring wrong pointing object 11793 0 (offset 0)\n",
      "Ignoring wrong pointing object 11853 0 (offset 0)\n",
      "Ignoring wrong pointing object 11981 0 (offset 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting text using PyPDF\n",
      "Parsing took: 98.872403 seconds\n",
      "books/Fundamentals_of_Physics_Textbook.pdf contained 1450 pages\n",
      "\n",
      "Extracting Text from books/Made_To_Stick.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 6.983265 seconds\n",
      "books/Made_To_Stick.pdf contained 225 pages\n",
      "\n",
      "Extracting Text from books/Pere_Riche_Pere_Pauvre.pdf ...\n",
      "Extracting text using PyPDF\n",
      "Parsing took: 0.775370 seconds\n",
      "books/Pere_Riche_Pere_Pauvre.pdf contained 225 pages\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Dictionary to store the parsed data for each book\n",
    "book_pages_map = dict()\n",
    "\n",
    "# Open the zip file\n",
    "with zipfile.ZipFile(LOCAL_FILE_PATH, 'r') as zip_ref:\n",
    "    # Iterate over the PDF files inside the zip archive\n",
    "    for file_info in zip_ref.infolist():\n",
    "        if file_info.filename.endswith('.pdf'):\n",
    "            book = file_info.filename\n",
    "            \n",
    "            print(\"Extracting Text from\", book, \"...\")\n",
    "            \n",
    "            # Read the PDF file directly into memory (as a binary stream)\n",
    "            with zip_ref.open(file_info) as file:\n",
    "                file_stream = io.BytesIO(file.read())  # Convert file to BytesIO for in-memory file handling\n",
    "\n",
    "                # Capture the start time\n",
    "                start_time = time.time()\n",
    "\n",
    "                # Parse the PDF (you would use your actual parse_pdf function here)\n",
    "                book_map = parse_pdf(file_stream, form_recognizer=False, verbose=True)\n",
    "                book_pages_map[book] = book_map\n",
    "                \n",
    "                # Capture the end time and calculate the elapsed time\n",
    "                end_time = time.time()\n",
    "                elapsed_time = end_time - start_time\n",
    "\n",
    "                print(f\"Parsing took: {elapsed_time:.6f} seconds\")\n",
    "                print(f\"{book} contained {len(book_map)} pages\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de0a722-ae0c-4b57-802a-518f5d4d93fd",
   "metadata": {},
   "source": [
    "Now let's check a random page of each book to make sure the parsing was done correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f2a5d62f-b664-4662-a6c9-a1eb2a3c5e11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "books/Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf \n",
      " chunk text: 14\n",
      "11:59 A.M.\n",
      "The rest of Sherrie’s morning proceeded fairly well. A tal-\n",
      "ented fashion designer, Sherrie had an unerrin ...\n",
      "\n",
      "books/Fundamentals_of_Physics_Textbook.pdf \n",
      " chunk text: 71-3 MASS\n",
      "international agreement, a mass of 1 kilogram. Accurate copies have been sent\n",
      "to standardizing laboratories in ...\n",
      "\n",
      "books/Made_To_Stick.pdf \n",
      " chunk text: of both fruits rather than all of either. Should we trade? If so, how do \n",
      "we go about doing it?\"  \n",
      "Students are initiall ...\n",
      "\n",
      "books/Pere_Riche_Pere_Pauvre.pdf \n",
      " chunk text: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for bookname,bookmap in book_pages_map.items():\n",
    "    print(bookname,\"\\n\",\"chunk text:\",bookmap[random.randint(10, 50)][2][:120],\"...\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcdc1ee-71fc-49d2-8e7c-0964bc3a4370",
   "metadata": {},
   "source": [
    "As we can see above, all books were parsed except `Pere_Riche_Pere_Pauvre.pdf` (this book is \"Rich Dad, Poor Dad\" written in French), why? Well, as we mentioned above, this book was scanned, so each page is an image and with a very unique font. We need a good PDF parser with good OCR capabilities in order to extract the content of this PDF. \n",
    "Let's try to parse this book again, but this time using Azure Document Intelligence API (former Form Recognizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "801c6bc2-467c-4418-aa7e-ef89a1e20e1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting text using Azure Document Intelligence\n",
      "Parsing took: 45.923739 seconds\n",
      "books/Pere_Riche_Pere_Pauvre.pdf contained 225 pages\n",
      "\n",
      "CPU times: user 11.7 s, sys: 184 ms, total: 11.9 s\n",
      "Wall time: 46 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "book = \"books/Pere_Riche_Pere_Pauvre.pdf\"\n",
    "with zipfile.ZipFile(LOCAL_FILE_PATH, 'r') as zip_ref:\n",
    "    with zip_ref.open(book) as file:\n",
    "                file_stream = io.BytesIO(file.read())  # Convert file to BytesIO for in-memory file handling\n",
    "\n",
    "                # Capture the start time\n",
    "                start_time = time.time()\n",
    "\n",
    "                # Parse the PDF (you would use your actual parse_pdf function here)\n",
    "                book_map = parse_pdf(file_stream, form_recognizer=True, model=\"prebuilt-document\",from_url=False, verbose=True)\n",
    "                book_pages_map[book] = book_map\n",
    "                \n",
    "                # Capture the end time and calculate the elapsed time\n",
    "                end_time = time.time()\n",
    "                elapsed_time = end_time - start_time\n",
    "\n",
    "                print(f\"Parsing took: {elapsed_time:.6f} seconds\")\n",
    "                print(f\"{book} contained {len(book_map)} pages\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "97f9c5bb-c44b-4a4d-9780-591f9f8d128a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "books/Pere_Riche_Pere_Pauvre.pdf \n",
      " chunk text: « Comment donc devrions-nous nous y prendre pour faire de l'argent? » demanda Mi ...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(book,\"\\n\",\"chunk text:\",book_map[random.randint(10, 50)][2][:80],\"...\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c279dfb-4fed-41b8-89e1-0ca2cefbcdc9",
   "metadata": {},
   "source": [
    "As demonstrated above, Azure Document Intelligence proves to be superior to pyPDF. **For production scenarios, we strongly recommend using Azure Document Intelligence consistently**. When doing so, it's important to make a wise choice between the available models, such as \"prebuilt-document,\" \"prebuilt-layout,\" or others. You can find more information on model selection [HERE](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/choose-model-feature?view=doc-intel-3.0.0).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f5f9b7d-99e6-426d-a47e-343c7e8b492e",
   "metadata": {},
   "source": [
    "## Create Vector-based index\n",
    "\n",
    "\n",
    "Now that we have the content of the book's chunks (each page of each book) in the dictionary `book_pages_map`, let's create the Vector index in our Azure Search Engine where this content is going to land"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "594ff0d4-56e3-4bed-843d-28c7a092069b",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 75\n",
    "embedder = AzureOpenAIEmbeddings(deployment=os.environ[\"EMBEDDING_DEPLOYMENT_NAME\"], chunk_size=batch_size, \n",
    "                                 max_retries=2, \n",
    "                                 retry_min_seconds= 60,\n",
    "                                 retry_max_seconds= 70)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7d46e7c5-49c4-40f3-bb2d-79a9afeab4b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "book_index_name = \"srch-index-books\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1b07e84b-d306-4bc9-9124-e64f252dd7b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Create Azure Search Vector-based Index\n",
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}\n",
    "params = {'api-version': os.environ['AZURE_SEARCH_API_VERSION']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2faab899-977b-40d0-b36e-26f75ac07e54",
   "metadata": {},
   "source": [
    "\n",
    "Please note the following points regarding the index:\n",
    "\n",
    "- The ParentKey field is absent.\n",
    "- The page_num field is present.\n",
    "\n",
    "The absence of the ParentKey field is due to the utilization of a PUSH method, rather than a PULL method. This approach indicates that we are not leveraging the integrated indexing provided by the Azure AI Search engine. Instead, we are engaging in the process of parsing, performing OCR, and manually creating and pushing the content along with its vectors.\n",
    "\n",
    "This manual parsing process involves the use of either, the pyPDF library, or the Azure Document Intelligence API. These APIs allow for the segmentation of content by page rather than by a specified number of characters, which is the method employed by the Azure AI search indexer. Consequently, this enables the inclusion of page_num as a field in our index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75d63e68-69a5-4b3b-8eb0-86da02cb7230",
   "metadata": {},
   "source": [
    "The latest Azure AI Search API supports external and internal vectorization. This Notebook assumes an external vectorization strategy. This API also supports:\n",
    "    \n",
    "- vectorSearch algorithms, hnsw and exhaustiveKnn nearest neighbors, with parameters for indexing and scoring.\n",
    "- vectorProfiles for multiple combinations of algorithm configurations.\n",
    "\n",
    "Vector search algorithms include **exhaustive k-nearest neighbors (KNN)** and **Hierarchical Navigable Small World (HNSW)**. Exhaustive KNN performs a brute-force search that scans the entire vector space. HNSW performs an approximate nearest neighbor (ANN) search. While KNN provides exact nearest neighbor search results with high accuracy, its computational cost and poor scalability make it impractical for large datasets or real-time applications. HNSW, on the other hand, offers a highly efficient and scalable solution for nearest neighbor searches by finding approximate nearest neighbors quickly, making it more suitable for large-scale and high-dimensional data applications.\n",
    "\n",
    "\n",
    "check [HERE](https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-create-index?tabs=config-2023-10-01-Preview%2Crest-2023-11-01%2Cpush%2Cportal-check-index) for the details of the vector configuration.\n",
    "\n",
    "**Note**: Unlike Notebooks 1 and 2, we will not add any vector compression to this index. This approach allows you to compare the resulting index sizes across all three indexes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2df4db6b-969b-4b91-963f-9334e17a4e3c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "index_payload = {\n",
    "    \"name\": book_index_name,\n",
    "    \"vectorSearch\": {\n",
    "        \"algorithms\": [  # We are showing here 3 types of search algorithms configurations that you can do\n",
    "             {\n",
    "                 \"name\": \"my-hnsw-config-1\",\n",
    "                 \"kind\": \"hnsw\",\n",
    "                 \"hnswParameters\": {\n",
    "                     \"m\": 4,\n",
    "                     \"efConstruction\": 400,\n",
    "                     \"efSearch\": 500,\n",
    "                     \"metric\": \"cosine\"\n",
    "                 }\n",
    "             },\n",
    "             {\n",
    "                 \"name\": \"my-hnsw-config-2\",\n",
    "                 \"kind\": \"hnsw\",\n",
    "                 \"hnswParameters\": {\n",
    "                     \"m\": 8,\n",
    "                     \"efConstruction\": 800,\n",
    "                     \"efSearch\": 800,\n",
    "                     \"metric\": \"cosine\"\n",
    "                 }\n",
    "             },\n",
    "             {\n",
    "                 \"name\": \"my-eknn-config\",\n",
    "                 \"kind\": \"exhaustiveKnn\",\n",
    "                 \"exhaustiveKnnParameters\": {\n",
    "                     \"metric\": \"cosine\"\n",
    "                 }\n",
    "             }\n",
    "        ],\n",
    "        \"vectorizers\": [\n",
    "            {\n",
    "                \"name\": \"openai\",\n",
    "                \"kind\": \"azureOpenAI\",\n",
    "                \"azureOpenAIParameters\":\n",
    "                {\n",
    "                    \"resourceUri\" : os.environ['AZURE_OPENAI_ENDPOINT'],\n",
    "                    \"apiKey\" : os.environ['AZURE_OPENAI_API_KEY'],\n",
    "                    \"deploymentId\" : os.environ['EMBEDDING_DEPLOYMENT_NAME'],\n",
    "                    \"modelName\" : os.environ['EMBEDDING_DEPLOYMENT_NAME']\n",
    "                }\n",
    "            }\n",
    "        ],\n",
    "        \"profiles\": [  # profiles is the diferent kind of combinations of algos and vectorizers\n",
    "            {\n",
    "             \"name\": \"my-vector-profile-1\",\n",
    "             \"algorithm\": \"my-hnsw-config-1\",\n",
    "             \"vectorizer\":\"openai\"\n",
    "            },\n",
    "            {\n",
    "             \"name\": \"my-vector-profile-2\",\n",
    "             \"algorithm\": \"my-hnsw-config-2\",\n",
    "             \"vectorizer\":\"openai\"\n",
    "            },\n",
    "            {\n",
    "             \"name\": \"my-vector-profile-3\",\n",
    "             \"algorithm\": \"my-eknn-config\",\n",
    "             \"vectorizer\":\"openai\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": {\n",
    "                        \"fieldName\": \"title\"\n",
    "                    },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        {\n",
    "                            \"fieldName\": \"chunk\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"prioritizedKeywordsFields\": []\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"filterable\": \"true\" },\n",
    "        {\"name\": \"title\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunk\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"page_num\",\"type\": \"Edm.Int32\",\"searchable\": \"false\",\"retrievable\": \"true\"},\n",
    "        {\n",
    "            \"name\": \"chunkVector\",\n",
    "            \"type\": \"Collection(Edm.Single)\",\n",
    "            \"dimensions\": 3072,\n",
    "            \"vectorSearchProfile\": \"my-vector-profile-3\", # we picked profile 3 to show that this index uses eKNN vs HNSW (on prior notebooks)\n",
    "            \"searchable\": \"true\",\n",
    "            \"retrievable\": \"true\",\n",
    "            \"filterable\": \"false\",\n",
    "            \"sortable\": \"false\",\n",
    "            \"facetable\": \"false\"\n",
    "        }\n",
    "        \n",
    "    ],\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + book_index_name,\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "36691ff0-c4c8-49d0-bfa8-3e076ece0ce5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to debug errors\n",
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc7dda9-4725-410e-9465-54f0298fc758",
   "metadata": {},
   "source": [
    "## Upload the Document chunks and its vectors to the Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d73e7600-7902-48d4-b199-9d9dc0a17aa0",
   "metadata": {},
   "source": [
    "The following code will iterate over each chunk of each book and use the Azure Search Rest API upload method to insert each document with its corresponding vector (using OpenAI embedding model) to the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a94911cf-c95f-4306-8574-b56296f29b88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to process a batch of pages\n",
    "def process_batch(bookname, pages):\n",
    "    try:\n",
    "        contents = [page[2] for page in pages]\n",
    "        chunk_vectors = embedder.embed_documents(contents)\n",
    "        \n",
    "        upload_payload = {\"value\": []}\n",
    "        for i, page in enumerate(pages):\n",
    "            page_num = page[0] + 1\n",
    "            content = page[2]\n",
    "            book_url = os.environ['BASE_CONTAINER_URL'] + bookname\n",
    "            \n",
    "            payload = {\n",
    "                \"@search.action\": \"upload\",\n",
    "                \"id\": str(uuid.uuid5(uuid.NAMESPACE_DNS, f\"{bookname}{page_num}\")),\n",
    "                \"title\": f\"{bookname}_page_{str(page_num)}\",\n",
    "                \"chunk\": content,\n",
    "                \"chunkVector\": chunk_vectors[i],\n",
    "                \"name\": bookname,\n",
    "                \"location\": book_url,\n",
    "                \"page_num\": page_num\n",
    "            }\n",
    "            upload_payload[\"value\"].append(payload)\n",
    "        \n",
    "        r = requests.post(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + book_index_name + \"/docs/index\",\n",
    "                          data=json.dumps(upload_payload), headers=headers, params=params)\n",
    "        if r.status_code != 200:\n",
    "            print(f\"Failed to upload batch of pages from {bookname}: {r.status_code}\")\n",
    "            print(r.text)\n",
    "    except Exception as e:\n",
    "        print(f\"Exception processing batch of pages from {bookname}: {e}\")\n",
    "        time.sleep(10)  # Wait before retrying\n",
    "        process_batch(bookname, pages)  # Retry the same batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "793a3171-f8f0-4070-8a54-8a540828333c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from books/Boundaries_When_to_Say_Yes_How_to_Say_No_to_Take_Control_of_Your_Life.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:26<00:00,  5.30s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from books/Fundamentals_of_Physics_Textbook.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20/20 [06:04<00:00, 18.22s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from books/Made_To_Stick.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:17<00:00,  5.91s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading chunks from books/Pere_Riche_Pere_Pauvre.pdf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:40<00:00, 13.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13.9 s, sys: 196 ms, total: 14.1 s\n",
      "Wall time: 7min 29s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for bookname, bookmap in book_pages_map.items():\n",
    "        print(\"Uploading chunks from\", bookname)\n",
    "        # Split bookmap into chunks of size chunk_size\n",
    "        for i in tqdm(range(0, len(bookmap), batch_size)):\n",
    "            batch = bookmap[i:i + batch_size]\n",
    "            process_batch(bookname, batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "715cddcf-af7b-4006-a047-853fc7a66be3",
   "metadata": {},
   "source": [
    "## Query the Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8b408798-5527-44ca-9dba-cad2ee726aca",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"what normally rich dad do that is different from poor dad?\"\n",
    "# QUESTION = \"Dime que significa la radiacion del cuerpo negro\"\n",
    "# QUESTION = \"what is the acronym of the main point of Made to Stick book\"\n",
    "# QUESTION = \"Tell me a python example of how do I push documents with vectors to an index using the python SDK?\"\n",
    "# QUESTION = \"who won the soccer worldcup in 1994?\" # this question should have no answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1b182ade-0ddd-47a1-b1eb-2cbf435c317f",
   "metadata": {},
   "outputs": [],
   "source": [
    "indexes = [book_index_name]\n",
    "k=50 # in this index k corresponds to the top pages as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d50eecb2-ce26-4127-a62b-79735b937046",
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = CustomAzureSearchRetriever(indexes=[book_index_name], topK=k, reranker_threshold=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd2f3f2-2d66-4bd4-b90b-d30970b71af4",
   "metadata": {},
   "source": [
    "**Note**: that we are picking a larger k=20 since these chunks are NOT of 5000 chars each like prior notebooks, but instead each page is a chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "410ff796-dab1-4817-a3a5-82eeff6c0c57",
   "metadata": {},
   "outputs": [],
   "source": [
    "COMPLETION_TOKENS = 2500\n",
    "llm = AzureChatOpenAI(deployment_name=os.environ[\"GPT4oMINI_DEPLOYMENT_NAME\"], temperature=0.5, max_tokens=COMPLETION_TOKENS).configurable_alternatives(\n",
    "    ConfigurableField(id=\"model\"),\n",
    "    default_key=\"gpt4omini\",\n",
    "    gpt4o=AzureChatOpenAI(deployment_name=os.environ[\"GPT4o_DEPLOYMENT_NAME\"], temperature=0, max_tokens=COMPLETION_TOKENS),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c33c2851-08c5-4e7c-ba7b-4655a6021e1e",
   "metadata": {},
   "source": [
    "In `utils.py` we created the **CustomAzureSearchRetriever** class that we will use going forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9168d828-6519-4f1b-a243-56f75fa86160",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DOCSEARCH_PROMPT = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", DOCSEARCH_PROMPT_TEXT + \"\\n\\nCONTEXT:\\n{context}\\n\\n\"),\n",
    "        (\"human\", \"{question}\"),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "26f47c69-44d8-48e3-974e-7989b4a8b7c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = (\n",
    "    {\n",
    "        \"context\": itemgetter(\"question\") | retriever, # Passes the question to the retriever and the results are assign to context\n",
    "        \"question\": itemgetter(\"question\")\n",
    "    }\n",
    "    | DOCSEARCH_PROMPT  # Passes the 4 variables above to the prompt template\n",
    "    | llm   # Passes the finished prompt to the LLM\n",
    "    | StrOutputParser()  # converts the output (Runnable object) to the desired output (string)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "765df250-af7f-46c9-8d7a-15c0522969ec",
   "metadata": {},
   "source": [
    "#### With GPT4o-mini"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "73f34192-519d-45b9-a0e2-a8b2de51ee1e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The differences between the mindsets and actions of the \"rich dad\" and the \"poor dad\" in Robert Kiyosaki's \"Père riche, père pauvre\" are quite pronounced and can be summarized as follows:\n",
      "\n",
      "1. **Attitude Towards Money**: The rich dad views money as a tool that can work for him, whereas the poor dad believes that money is something to be earned through hard work. The rich dad teaches that \"money works for you\" whereas the poor dad thinks \"you work for money\" [[6]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "2. **Financial Education**: The rich dad emphasizes the importance of financial education and understanding how money works. He believes that learning about money management and investments is crucial for wealth creation. In contrast, the poor dad focuses on traditional education and securing a stable job, believing that good grades will lead to a good job [[6]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "3. **Investment and Assets**: The rich dad teaches the importance of acquiring assets that generate income, while the poor dad considers his home as the most significant investment. The rich dad understands the difference between assets and liabilities, emphasizing that true wealth comes from accumulating assets that produce cash flow [[5]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "4. **Mindset and Language**: The rich dad encourages a mindset that focuses on possibilities and opportunities, teaching his children to ask, \"How can I afford this?\" instead of saying, \"I can't afford this,\" which limits thinking. The poor dad, however, often expresses limitations regarding money [[6]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "5. **Risk and Fear**: The rich dad teaches how to manage risks and leverage them for opportunity, while the poor dad tends to avoid risks and focuses on job security [[6]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "These contrasting beliefs and behaviors highlight how mindset plays a crucial role in financial success and the ability to build wealth over time."
     ]
    }
   ],
   "source": [
    "for chunk in chain.with_config(configurable={\"model\": \"gpt4omini\"}).stream(\n",
    "    {\"question\": QUESTION, \"language\": \"English\"}):\n",
    "    print(chunk, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4a8761d-2c1e-4369-b7c4-c3571a0793e9",
   "metadata": {},
   "source": [
    "#### With GPT4-o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "14b77511-b178-4c9b-9fa5-fdddb0d3e586",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In \"Père riche, Père pauvre,\" the rich dad and the poor dad have fundamentally different approaches to money and life. The rich dad emphasizes the importance of financial education and making money work for you, rather than working for money. He believes in acquiring assets that generate income and encourages learning about how money works to achieve financial independence. The rich dad also stresses the importance of understanding the law and using it to one's advantage, often employing financial advisors and lawyers to minimize taxes and protect wealth [[1]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "On the other hand, the poor dad, despite being well-educated, focuses on job security and working for a stable salary. He believes in the traditional path of getting a good education to secure a good job with benefits. The poor dad often views the house as the most significant investment, whereas the rich dad sees it as a liability unless it generates income [[2]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf).\n",
      "\n",
      "The rich dad encourages thinking about how to afford things by asking \"How can I afford it?\" rather than saying \"I can't afford it,\" which is a mindset the poor dad often has. This difference in mindset leads to different financial outcomes and life paths [[3]](https://blobstorageixqo5iaqmpzwc.blob.core.windows.net/books/Pere_Riche_Pere_Pauvre.pdf)."
     ]
    }
   ],
   "source": [
    "for chunk in chain.with_config(configurable={\"model\": \"gpt4o\"}).stream(\n",
    "    {\"question\": QUESTION, \"language\": \"English\"}):\n",
    "    print(chunk, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3941796c-7655-4888-a358-8a62e380bd7e",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "In this notebook we learned how to deal with complex and large Documents and make them available for Q&A over them using [Hybrid Search](https://learn.microsoft.com/en-us/azure/search/search-get-started-vector#hybrid-search) (text + vector search).\n",
    "\n",
    "We also learned the power of Azure Document Inteligence API and why it is recommended for production scenarios where manual Document parsing (instead of Azure Search Indexer Document Cracking) is necessary.\n",
    "\n",
    "Using Azure AI Search with its Vector capabilities and hybrid search features eliminates the need for other vector databases such as Weaviate, Qdrant, Milvus, Pinecone, and so on.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d9a7d1-f029-416b-8eb2-00a8afb9151d",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "So far we have learned how to use OpenAI vectors and completion APIs in order to get an excelent answer from our documents stored in Azure AI Search. This is the backbone for a GPT Smart Search Engine.\n",
    "\n",
    "However, we are missing something: **How to have a conversation with this engine?**\n",
    "\n",
    "On the next Notebook, we are going to understand the concept of **memory**. This is necessary in order to have a chatbot that can establish a conversation with the user. Without memory, there is no real conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ed733fb-dec4-4a8f-bff0-c7cbbcc0328e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "GPTSearch3 (Python 3.12)",
   "language": "python",
   "name": "gptsearch3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}