language/examples/document-summarization/summarization_large_documents.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uxCkB_DXTHzf"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hny4I-ODTIS6"
      },
      "source": [
        "# Text Summarization of Large Documents\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/language/examples/reference-architectures/text_summarization_for_large_documents.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/reference-architectures/text_summarization_for_large_documents.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/blob/main/language/examples/reference-architectures/text_summarization_for_large_documents.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-nLS57E2TO5y"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Text summarization is the process of creating a shorter version of a text document while still preserving the most important information. This can be useful for a variety of purposes, such as quickly skimming a long document, getting the gist of an article, or sharing a summary with others.\n",
        "\n",
        "Although summarizing a short paragraph is a non-trivial task, there are a few challenges to overcome if you want to summarize a large document, such as a PDF file with multiple pages. In this notebook, you will go through a few examples of how you can use generative models to summarize large documents."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXsvgIuwTPZw"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you will learn how to use generative models to summarize information from text by working through the following examples:\n",
        "\n",
        "- Stuffing method\n",
        "- MapReduce method\n",
        "- MapReduce with Overlapping Chunks method\n",
        "- MapReduce with Rolling Summary method"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skXAu__iqks_"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "- Vertex AI Generative AI Studio\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Generative AI pricing](https://cloud.google.com/vertex-ai/pricing#generative_ai_models), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mvKl-BtQTRiQ"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwFMpIMrTV_4"
      },
      "source": [
        "### Install Vertex AI SDK, other packages and their dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WYUu8VMdJs3V"
      },
      "outputs": [],
      "source": [
        "!pip install google-cloud-aiplatform PyPDF2 ratelimit backoff --upgrade --quiet --user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWwtjLV5TY6H"
      },
      "source": [
        "**Colab only**: Uncomment the following cell to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opUxT_k5TdgP"
      },
      "source": [
        "### Authenticating your notebook environment\n",
        "* If you are using **Colab** to run this notebook, uncomment the cell below and continue.\n",
        "* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vbNgv4q1T2Mi"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n5fXfvzhTkYN"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uCGlGzd1ihnf"
      },
      "source": [
        "**Colab only:** Uncomment the following cell to initialize the Vertex AI SDK. For Vertex AI Workbench, you don't need to run this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GjSsu6cmUdEx"
      },
      "outputs": [],
      "source": [
        "# import vertexai\n",
        "\n",
        "# PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "# vertexai.init(project=PROJECT_ID, location=\"us-central1\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cRkcfnQMT9vD"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "import urllib\n",
        "import warnings\n",
        "from pathlib import Path\n",
        "\n",
        "import backoff\n",
        "import pandas as pd\n",
        "import PyPDF2\n",
        "import ratelimit\n",
        "from google.api_core import exceptions\n",
        "from tqdm import tqdm\n",
        "from vertexai.preview.language_models import TextGenerationModel\n",
        "\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAGaTjPVTmhP"
      },
      "source": [
        "### Import models\n",
        "\n",
        "Here you load the pre-trained text generation model called `text-bison@001`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ITUmZiNZcMUW"
      },
      "outputs": [],
      "source": [
        "generation_model = TextGenerationModel.from_pretrained(\"text-bison@001\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZkLDRTjTcfm"
      },
      "source": [
        "### Preparing data files\n",
        "\n",
        "To begin, you will need to download a pdf file for the summarizing tasks below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7H0zINHpTaSu"
      },
      "outputs": [],
      "source": [
        "# Define a folder to store the files\n",
        "data_folder = \"data\"\n",
        "Path(data_folder).mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "# Define a pdf link to download and place to store the download file\n",
        "pdf_url = \"https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf\"\n",
        "pdf_file = Path(data_folder, pdf_url.split(\"/\")[-1])\n",
        "\n",
        "# Download the file using `urllib` library\n",
        "urllib.request.urlretrieve(pdf_url, pdf_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d_g1xzysoU18"
      },
      "source": [
        "Here you will take a peak at a few pages of the downloaded pdf file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hLtMd97SoTBE"
      },
      "outputs": [],
      "source": [
        "# Read the PDF file and create a list of pages\n",
        "reader = PyPDF2.PdfReader(pdf_file)\n",
        "pages = reader.pages\n",
        "\n",
        "# Print three pages from the pdf\n",
        "for i in range(3):\n",
        "    text = pages[i].extract_text().strip()\n",
        "    print(f\"Page {i}: {text} \\n\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZDVwBFSjZ7ws"
      },
      "source": [
        "## Method 1: Stuffing\n",
        "\n",
        "The simplest way to pass data to a language model is to \"stuff\" it all into the prompt as context. This means simply including all of the relevant information in the prompt, in the order that you want the model to process it.\n",
        "\n",
        "Here you will extract the text from all the pages in the pdf file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eH3Y3X8hJs3X"
      },
      "outputs": [],
      "source": [
        "# Read the PDF file and create a list of pages\n",
        "reader = PyPDF2.PdfReader(pdf_file)\n",
        "pages = reader.pages\n",
        "\n",
        "# Entry string to concatenate all the extacted texts\n",
        "concatenated_text = \"\"\n",
        "\n",
        "# Loop through the pages\n",
        "for page in tqdm(pages):\n",
        "\n",
        "    # Extract the text from the page and remove any leading or trailing whitespace\n",
        "    text = page.extract_text().strip()\n",
        "\n",
        "    # Concate the extracted text to the concatenated text\n",
        "    concatenated_text += text\n",
        "\n",
        "print(f\"There are {len(concatenated_text)} characters in the pdf\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fOMGiUAaiy3Y"
      },
      "source": [
        "You will now create a prompt template that can be used later in the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eDBlvprWizgW"
      },
      "outputs": [],
      "source": [
        "prompt_template = \"\"\"\n",
        "    Write a concise summary of the following text delimited by triple backquotes.\n",
        "    Return your response in bullet points which covers the key points of the text.\n",
        "\n",
        "    ```{text}```\n",
        "\n",
        "    BULLET POINT SUMMARY:\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_xMwRLuyDrj"
      },
      "source": [
        "Here you will use LLM via the API to summarize the extracted texts. Please note that LLMs currently have input text limit and stuffing a large input text might not be accepted. You can read more about quotas and limits [here](https://cloud.google.com/vertex-ai/docs/quotas).\n",
        "\n",
        "The following code will cause **an exception**!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RtgemmBzkddX"
      },
      "outputs": [],
      "source": [
        "# Define the prompt using the prompt template\n",
        "prompt = prompt_template.format(text=concatenated_text)\n",
        "\n",
        "# Use the model to summarize the text using the prompt\n",
        "summary = generation_model.predict(prompt=prompt, max_output_tokens=1024).text\n",
        "\n",
        "print(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5aVrDWkJs3Y"
      },
      "source": [
        "#### Retrying\n",
        "\n",
        "The model responded with an error message: **400 Request contains an invalid argument** because the extracted text is too long for the generative model to process.\n",
        "\n",
        "To avoid this issue, you will only input a chunk of the extracted text (e.g. the first 30,000 words)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MmUwTIjMk58J"
      },
      "outputs": [],
      "source": [
        "# Define the prompt using the prompt template\n",
        "prompt = prompt_template.format(text=concatenated_text[:30000])\n",
        "\n",
        "# Use the model to summarize the text using the prompt\n",
        "summary = generation_model.predict(prompt=prompt, max_output_tokens=1024).text\n",
        "\n",
        "print(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vtp21WX3T7d_"
      },
      "source": [
        "### Recap\n",
        "\n",
        "Although full text is too large for the model, you have managed to create a concise, bulleted list of the most important information from a portion of the PDF using the model. Thus, here are the pros and cons of using the stuffing method:\n",
        "\n",
        "**Pros:**\n",
        "- Only required a single call to the model.\n",
        "- When summarizing text, the model has access to all the data at once so that the result may be better.\n",
        "\n",
        "**Cons:**\n",
        "- Most models have a context length, and for large documents (or many documents) this will not work as it will result in a prompt larger than the context length.\n",
        "- This method only works on smaller pieces of data and not suitable to large documents most of the time.\n",
        "\n",
        "In the following session, you will explore approaches which designed to help deal with having longer text than context lengh limit of LLMs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOY5LHl9Js3Y"
      },
      "source": [
        "### Adding rate limit to model calls\n",
        "\n",
        "When you use MapReduce or other similar methods, you will be making multiple API calls to the model in a short period of time. There is a limit on the number of API calls you can make per minute, so you will need to add a safety measure to your code to prevent exceeding the limit. This will help to ensure that your code runs smoothly and does not encounter any errors.\n",
        "\n",
        "For this method, here are a few specific things that you will do:\n",
        "1. You will make use of a Python library called [ratelimit](https://pypi.org/project/ratelimit/) to limit the number of API calls per minute\n",
        "2. You will make use of a Python library called [backoff](https://pypi.org/project/backoff/) to retry until the maximum time limit has reached\n",
        "\n",
        "The following function improves the API call process by limiting the number of calls to **20 per minute**. It also back offs and retries calling the API after encountering **Resource Exhausted** exception. The wait duration grows **exponentially until the 5-minute mark**, and then the function will give up on retrying."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0uziSPeEJs3Y"
      },
      "outputs": [],
      "source": [
        "CALL_LIMIT = 20  # Number of calls to allow within a period\n",
        "ONE_MINUTE = 60  # One minute in seconds\n",
        "FIVE_MINUTE = 5 * ONE_MINUTE\n",
        "\n",
        "# A function to print a message when the function is retrying\n",
        "def backoff_hdlr(details):\n",
        "    print(\n",
        "        \"Backing off {} seconds after {} tries\".format(\n",
        "            details[\"wait\"], details[\"tries\"]\n",
        "        )\n",
        "    )\n",
        "\n",
        "\n",
        "@backoff.on_exception(  # Retry with exponential backoff strategy when exceptions occur\n",
        "    backoff.expo,\n",
        "    (\n",
        "        exceptions.ResourceExhausted,\n",
        "        ratelimit.RateLimitException,\n",
        "    ),  # Exceptions to retry on\n",
        "    max_time=FIVE_MINUTE,\n",
        "    on_backoff=backoff_hdlr,  # Function to call when retrying\n",
        ")\n",
        "@ratelimit.limits(  # Limit the number of calls to the model per minute\n",
        "    calls=CALL_LIMIT, period=ONE_MINUTE\n",
        ")\n",
        "\n",
        "# This function will call the `generation_model.predict` function, but it will retry if defined exceptions occur.\n",
        "def model_with_limit_and_backoff(**kwargs):\n",
        "    return generation_model.predict(**kwargs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RM3V1JARZ9-k"
      },
      "source": [
        "## Method 2: MapReduce\n",
        "\n",
        "This method works by first splitting the large data into chunks, then running a prompt on each chunk of text. For summarization tasks, the output from the initial prompt would be a summary of that chunk. Once all the initial outputs have been generated, a different prompt is run to combine them.\n",
        "\n",
        "This method is a bit more complex than the first method, but it can be more effective for large datasets. Here you will prepare two prompt templates: one for the initial summary step and another for the final combine step. You will be using these two templates later in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fs4rmcL-Js3Y"
      },
      "outputs": [],
      "source": [
        "initial_prompt_template = \"\"\"\n",
        "    Write a concise summary of the following text delimited by triple backquotes.\n",
        "\n",
        "    ```{text}```\n",
        "\n",
        "    CONCISE SUMMARY:\n",
        "\"\"\"\n",
        "\n",
        "final_prompt_template = \"\"\"\n",
        "    Write a concise summary of the following text delimited by triple backquotes.\n",
        "    Return your response in bullet points which covers the key points of the text.\n",
        "\n",
        "    ```{text}```\n",
        "\n",
        "    BULLET POINT SUMMARY:\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eo5NkotOJs3Y"
      },
      "source": [
        "#### Map step\n",
        "\n",
        "In this section, you will read the PDF file again and use the model to summarize each page individually using the initial prompt template."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aRub85PIJs3Y"
      },
      "outputs": [],
      "source": [
        "# Read the PDF file and create a list of pages\n",
        "reader = PyPDF2.PdfReader(pdf_file)\n",
        "pages = reader.pages\n",
        "\n",
        "# Create an empty list to store the summaries\n",
        "initial_summary = []\n",
        "\n",
        "# Iterate over the pages and generate a summary for each page\n",
        "for page in tqdm(pages):\n",
        "\n",
        "    # Extract the text from the page and remove any leading or trailing whitespace\n",
        "    text = page.extract_text().strip()\n",
        "\n",
        "    # Create a prompt for the model using the extracted text and a prompt template\n",
        "    prompt = initial_prompt_template.format(text=text)\n",
        "\n",
        "    # Generate a summary using the model and the prompt\n",
        "    summary = model_with_limit_and_backoff(prompt=prompt, max_output_tokens=1024).text\n",
        "\n",
        "    # Append the summary to the list of summaries\n",
        "    initial_summary.append(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pLLAUkuNDLbp"
      },
      "source": [
        "Take a look at the first few summaries of from the initial Map phrase."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3CpkQtgJs3Z"
      },
      "outputs": [],
      "source": [
        "print(\"\\n\\n\".join(initial_summary[:10]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KlPC414VJs3Z"
      },
      "source": [
        "Here you will count the number of characters in the initial summary to see if they are small enough to fit in a prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GtetmxSRJs3Z"
      },
      "outputs": [],
      "source": [
        "len(\"\\n\".join(initial_summary))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TRAxL2PxJs3Z"
      },
      "source": [
        "As you managed to input 30,000 characters in a prompt previously, you can input this whole summary which has fewer characters to a prompt directly too. You will do that in the next step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdgZs93AJs3Z"
      },
      "source": [
        "#### Reduce step\n",
        "\n",
        "Here you will create a reduce function that concatenate the summaries from the inital summarization step (Map step) and use the final prompt template to summarize the summaries again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QS4caPjNJs3Z"
      },
      "outputs": [],
      "source": [
        "# Define a function to create a summary of the summaries\n",
        "def reduce(summary, prompt_template):\n",
        "\n",
        "    # Concatenate the summaries from the inital step\n",
        "    concat_summary = \"\\n\".join(initial_summary)\n",
        "\n",
        "    # Create a prompt for the model using the concatenated text and a prompt template\n",
        "    prompt = prompt_template.format(text=concat_summary)\n",
        "\n",
        "    # Generate a summary using the model and the prompt\n",
        "    summary = model_with_limit_and_backoff(prompt=prompt, max_output_tokens=1024).text\n",
        "\n",
        "    return summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9OxwetDgKoUg"
      },
      "source": [
        "You are ready to proceed on to the next step to combine all the summary into an even smaller summary using the final prompt template and the function that you created earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uvd7MUrSKfu-"
      },
      "outputs": [],
      "source": [
        "# Use defined `reduce` function to summarize the summaries\n",
        "summary = reduce(initial_summary, final_prompt_template)\n",
        "\n",
        "print(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AjkbEdTYJs3Z"
      },
      "source": [
        "#### Recap\n",
        "\n",
        "You just summarized the whole paper into a few bullet points using the MapReduce method. Here are the pros and cons of using such method:\n",
        "\n",
        "**Pros:**\n",
        "- Can summarize a large document\n",
        "- Can work well with parallel processing as the processes to summarize pages are independent to each other\n",
        "\n",
        "**Cons:**\n",
        "- Multiple calls to the model is needed\n",
        "- As the pages are summarized individually, the context between the pages could be loss\n",
        "\n",
        "\n",
        "In the next section, you will try another method which makes use of more than one chunk (page) per prompt to summarize."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rpQV6viOlUgH"
      },
      "source": [
        "## Method 3: MapReduce with Overlapping Chunks\n",
        "\n",
        "It is similar to MapReduce, but with one key difference: overlapping chunks. This means that a few pages will be summarized together, rather than each page being summarized separately. This helps to preserve more context or information between chunks, which can improve the accuracy of the results.\n",
        "\n",
        "It is important to note that combining chunks may sometimes exceed the token limit imposed by the model. If this occurs, you can either implement the chunk splitting method showor creatively solve the issue (e.g. removing a few initial chunks)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SiPq5CqJJs3c"
      },
      "source": [
        "#### Map step\n",
        "\n",
        "In this section, you will read the PDF file again and use the model to summarize <b>a few pages</b> together using the initial prompt template that you defined earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ux6pPmPoJs3c"
      },
      "outputs": [],
      "source": [
        "# Read the PDF file and create a list of pages\n",
        "reader = PyPDF2.PdfReader(pdf_file)\n",
        "pages = reader.pages\n",
        "\n",
        "# Create an empty list to store the extracted text from the pages\n",
        "text_from_pages = []\n",
        "\n",
        "# Iterate over the pages and generate a summary for each page\n",
        "for page in tqdm(pages):\n",
        "\n",
        "    # Extract the text from the page and remove any leading or trailing whitespace\n",
        "    text = page.extract_text().strip()\n",
        "\n",
        "    # Append the extracted text to the list of extracted text\n",
        "    text_from_pages.append(text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZD7HMdQgXuP0"
      },
      "source": [
        "Here you will define the chunk size (number of pages to combine in this example) and summarize the chunks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZOA0Aq1nJs3c"
      },
      "outputs": [],
      "source": [
        "CHUNK_SIZE = 2  # number of overlapping pages\n",
        "\n",
        "# Read the PDF file and create a list of pages\n",
        "reader = PyPDF2.PdfReader(pdf_file)\n",
        "pages = reader.pages\n",
        "\n",
        "# Create an empty list to store the summaries\n",
        "initial_summary = []\n",
        "\n",
        "# Iterate over the pages and generate a summary for a few pages as one chunk based on `CHUNK_SIZE`\n",
        "for i in tqdm(range(len(pages))):\n",
        "\n",
        "    # Select a list of pages to merge as one chunk\n",
        "    pages_to_merge = [x for x in range(i, i + CHUNK_SIZE) if x < len(pages)]\n",
        "\n",
        "    extracted_texts = [text_from_pages[x] for x in pages_to_merge]\n",
        "\n",
        "    # Concatenate the\n",
        "    text = \"\\n\".join(extracted_texts)\n",
        "\n",
        "    # Create a prompt for the model using the concatenated text and a prompt template\n",
        "    prompt = initial_prompt_template.format(text=text)\n",
        "\n",
        "    # Generate a summary using the model and the prompt\n",
        "    summary = model_with_limit_and_backoff(prompt=prompt, max_output_tokens=1024).text\n",
        "\n",
        "    # Append the summary to the list of summaries\n",
        "    initial_summary.append(summary)\n",
        "\n",
        "    # If the last page is reached, break the loop\n",
        "    if pages_to_merge[-1] == len(reader.pages):\n",
        "        break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KVUVGBfbekfL"
      },
      "source": [
        "Take a look at the first few summaries of from the initial Map phrase."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uxgPKJ7BefyX"
      },
      "outputs": [],
      "source": [
        "print(\"\\n\\n\".join(initial_summary[:10]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oL7aV6U2Js3d"
      },
      "source": [
        "#### Reduce step\n",
        "\n",
        "You are ready to proceed on to the next step to combine all the summary into an even smaller summary using the final prompt template and the function that you created earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AxqSucUgJs3d"
      },
      "outputs": [],
      "source": [
        "# Use defined `reduce` function to summarize the summaries\n",
        "summary = reduce(initial_summary, final_prompt_template)\n",
        "\n",
        "print(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6UpRIozJs3d"
      },
      "source": [
        "#### Recap\n",
        "\n",
        "The model was able to summary the whole paper into a few bullet points using the MapReduce with Overlapping Chunks method. Here are the pros and cons of using such method:\n",
        "\n",
        "**Pros:**\n",
        "- Can summarize a large document\n",
        "- As the sequential pages are summarized together, the context between the pages are preserved\n",
        "- Can use parallel processing as the results are independent to each other\n",
        "\n",
        "**Cons:**\n",
        "- Multiple calls to the model is needed\n",
        "- Slightly slower than pure MapReduce method\n",
        "- Create larger input text\n",
        "\n",
        "\n",
        "In the next section, you will try a different approach that make use of a summary from the previous page instead of the entire text."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFqa_qCmJs3d"
      },
      "source": [
        "## Method 4: MapReduce with Rolling Summary (Refine)\n",
        "\n",
        "On some occasions, combining a few pages might be too large to summarize. To resolve that issue, you will now a different approach that uses an initial summary from the previous step along with the next page to summarize each prompt. This helps to ensure that the summary is complete and accurate, as it takes into account the context of the previous page."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zfK3SQMSJs3d"
      },
      "outputs": [],
      "source": [
        "initial_prompt_template = \"\"\"\n",
        "    Taking the following context delimited by triple backquotes into consideration:\n",
        "\n",
        "    ```{context}```\n",
        "\n",
        "    Write a concise summary of the following text delimited by triple backquotes.\n",
        "\n",
        "    ```{text}```\n",
        "\n",
        "    CONCISE SUMMARY:\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9sCN849gJs3d"
      },
      "outputs": [],
      "source": [
        "# Read the PDF file and create a list of pages.\n",
        "reader = PyPDF2.PdfReader(pdf_file)\n",
        "pages = reader.pages\n",
        "\n",
        "# Create an empty list to store the summaries.\n",
        "initial_summary = []\n",
        "\n",
        "# Iterate over the pages and generate a summary\n",
        "for idx, page in enumerate(tqdm(pages)):\n",
        "\n",
        "    # Extract the text from the page and remove any leading or trailing whitespace.\n",
        "    text = page.extract_text().strip()\n",
        "\n",
        "    if idx == 0:  # if current page is the first page, no previous context\n",
        "        prompt = initial_prompt_template.format(context=\"\", text=text)\n",
        "\n",
        "    else:  # if current page is not the first page, previous context is the summary of the previous page\n",
        "        prompt = initial_prompt_template.format(\n",
        "            context=initial_summary[idx - 1], text=text\n",
        "        )\n",
        "\n",
        "    # Generate a summary using the model and the prompt\n",
        "    summary = model_with_limit_and_backoff(prompt=prompt, max_output_tokens=1024).text\n",
        "\n",
        "    # Append the summary to the list of summaries\n",
        "    initial_summary.append(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NJ3cOxWOJs3d"
      },
      "source": [
        "Here you will list out a few entries from the initial summary list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L5yOZikVJs3d"
      },
      "outputs": [],
      "source": [
        "initial_summary[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OYYZM4QOJs3d"
      },
      "source": [
        "It is expected that there will be a few duplicate entries in the list, as you are rolling in context from previous pages to the next. You can easily remove these duplicates by using the set function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KxKpvvEzJs3d"
      },
      "outputs": [],
      "source": [
        "initial_summary = set(initial_summary)  # set() function removes duplicate items"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9HczirNnJs3d"
      },
      "source": [
        "#### Reduce step\n",
        "You are ready to proceed on to the next step to combine all the summary into an even smaller summary using the final prompt template and the function that you created earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I8Mg8MaaJs3d"
      },
      "outputs": [],
      "source": [
        "# Use defined `reduce` function to summarize the summaries\n",
        "summary = reduce(initial_summary, final_prompt_template)\n",
        "\n",
        "print(summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_5wFY2SLJs3d"
      },
      "source": [
        "#### Recap\n",
        "\n",
        "The model was able to summarize the whole paper into a few bullet points using the MapReduce with Rolling Summary method. Here are the pros and cons of using such method:\n",
        "\n",
        "**Pros:**\n",
        "- Can summarize a large document\n",
        "- As the sequential pages are summarized using the context from previous pages, the context between the pages are preserved\n",
        "\n",
        "**Cons:**\n",
        "- Multiple calls to the model is needed\n",
        "- Cannot work well with parallel processing as the processes to summarize pages are dependent to each other"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LNyorWQgJs3d"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "You have successfully summarized a long document, even though it was initially impossible due to an input prompt limit. You have also learned several methods for summarizing long documents, along with their advantages and disadvantages.\n",
        "\n",
        "Summarizing a long document can be challenging. It requires you to identify the main points of the document, synthesize the information, and present it in a concise and coherent way. This can be especially difficult if the document is complex or technical. Additionally, summarizing a long document can be time-consuming, as you need to carefully read and analyze the text to ensure that the summary is accurate and complete.\n",
        "\n",
        "While these methods allow you to interact with LLMs and summarize long documents in a flexible way, you may sometimes want to speed up the process by using bootstrapping or pre-built methods. This is where libraries like LangChain come in. You can read more about LangChain support on Vertex AI [here](https://python.langchain.com/en/latest/modules/models/llms/integrations/google_vertex_ai_palm.html)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "summarization_large_documents.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}