03-video-summarization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "500968f8",
   "metadata": {},
   "source": [
    "# AI Video Summarization\n",
    "\n",
    "Publishers and broadcasters can leverage short-form video across social media platforms such as Facebook, Instagram, and TikTok to attract new audiences and create additional revenue opportunities.\n",
    "\n",
    "However, generating video summaries is a manual and time-consuming process due to challenges like understanding complex content, maintaining coherence, diverse video types, and lack of scalability when dealing with a large volume of videos. Introducing automation through the use of artificial intelligence (AI) and machine learning (ML) can make this process more viable and scalable with automatic content analysis, real-time processing, contextual adaptation, customization, and continuous AI/ML system improvement.\n",
    "\n",
    "### High level workflow\n",
    "\n",
    "![video summarization diagram](static/images/video-summarization-diagram.png)\n",
    "\n",
    "In this notebook, we'll break down each step and show you in detail how video summarization can be achieved using AWS native services such as [Amazon Transcribe](https://aws.amazon.com/pm/transcribe), [Amazon Bedrock](https://aws.amazon.com/bedrock), [Amazon Polly](https://aws.amazon.com/polly/) and [AWS Elemental MediaConvert](https://aws.amazon.com/mediaconvert/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a4657c2",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "088571c4",
   "metadata": {},
   "source": [
    "To run this notebook, you need to have run the previous notebooks: [00-prerequisites](00-prerequisites.ipynb) and [01-video-time-segmentation](01-video-time-segmentation.ipynb), where you installed package dependencies and gathered some information from the SageMaker environment as well as segmented the video using audio, visual and semantic information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab95822",
   "metadata": {},
   "source": [
    "### Retrieve saved values from previous notebooks\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f733d62a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfe018e1",
   "metadata": {},
   "source": [
    "### Import python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6202fc75",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "import json\n",
    "import time\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError\n",
    "import json_repair\n",
    "from termcolor import colored\n",
    "from IPython.display import JSON\n",
    "from IPython.display import Video\n",
    "from IPython.display import Pretty\n",
    "from IPython.display import Image as DisplayImage\n",
    "from lib.frames import VideoFrames\n",
    "from lib.shots import Shots\n",
    "from lib.scenes import Scenes\n",
    "from lib.transcript import Transcript\n",
    "import base64\n",
    "from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth\n",
    "import subprocess\n",
    "import numpy as np\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e300a9a",
   "metadata": {},
   "source": [
    "## Summarize video content from transcript\n",
    "\n",
    "We use **Large Language Model (LLM)** with [Amazon Bedrock](https://aws.amazon.com/bedrock/) to summarize the content of the video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6113bbd-6571-42c3-a0cc-cf9bca4e3410",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bedrock_client = boto3.client(service_name=\"bedrock-runtime\")\n",
    "accept = \"application/json\"\n",
    "content_type = \"application/json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24333345",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(video['transcript'].transcript_file, 'r') as file:\n",
    "    transcript_file = json.load(file)\n",
    "transcript = transcript_file['results']['transcripts'][0]['transcript']\n",
    "\n",
    "model_id = \"anthropic.claude-3-sonnet-20240229-v1:0\"\n",
    "\n",
    "prompt = f\"\"\"Summarize the key points from the following video content in chronological order:\n",
    "\n",
    "{transcript} \n",
    "\n",
    "\\n\\nThe summary should only contain information present in the video content. Do not include any new or unrelated information.\n",
    "\n",
    "Important: Start the summary immediately without any introductory phrases. Begin directly with the first key point.\"\"\"\n",
    "\n",
    "body = json.dumps(\n",
    "    {\n",
    "        \"messages\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\"type\": \"text\", \"text\": prompt},\n",
    "                ],\n",
    "            }\n",
    "        ],\n",
    "        \"anthropic_version\": \"bedrock-2023-05-31\",\n",
    "        \"max_tokens\": 1024,\n",
    "        \"temperature\": 0.25,\n",
    "        \"top_p\": 0.9,\n",
    "\n",
    "    }\n",
    ")\n",
    "response = bedrock_client.invoke_model(\n",
    "    body=body, modelId=model_id, accept=accept, contentType=content_type\n",
    ")\n",
    "response_body = json.loads(response[\"body\"].read())\n",
    "summarized_text = response_body[\"content\"][0][\"text\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00a94157",
   "metadata": {},
   "source": [
    "You can invoke the endpoint with different parameters defined in the payload to impact the text summarization. Two important parameters are `top_p` and `temperature`. While `top_p` is used to control the range of tokens considered by the model based on their cumulative probability, `temperature` controls the level of randomness in the output.\n",
    "\n",
    "Although there isn’t a one-size-fits-all combination of `top_p` and `temperature` for all use cases, in the previous example, we demonstrate sample values with high `top_p` and low `temperature` that leads to summaries focused on key information and avoid deviating from the original text but still introduce some creative variations to keep the output interesting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07258089",
   "metadata": {},
   "source": [
    "Let's check the summarized video content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f25fc218",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "summarized_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da241b73",
   "metadata": {},
   "source": [
    "## Generate metadata for voice narration\n",
    "\n",
    "The next step starts with [Amazon Polly](https://aws.amazon.com/polly/) to generate speech from the summarized text. The output of the Polly task is both MP3 files and documents marked up with [Speech Synthesis Markup Language (SSML)](https://docs.aws.amazon.com/polly/latest/dg/ssml.html). Within this SSML file, essential metadata is encapsulated, describing the duration of individual sentences vocalized by a specific Polly voice. With this audio duration information, we will be able to define the length of the video segments; in this case, a direct 1:1 correspondence is employed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a61487d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "polly_client = boto3.client(\"polly\")\n",
    "voice_id = \"Matthew\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9485e527-5e9c-489d-939b-98e5d1202f42",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = polly_client.synthesize_speech(\n",
    "    Engine=\"neural\",\n",
    "    OutputFormat=\"json\",\n",
    "    Text=summarized_text + \" This video is generated by Video Summarization Hub.\",\n",
    "    TextType=\"text\",\n",
    "    SpeechMarkTypes=[\"sentence\"],\n",
    "    VoiceId=voice_id,\n",
    ")\n",
    "\n",
    "stream_data = response['AudioStream'].read()\n",
    "polly_ssml = stream_data.decode('utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5643bba8",
   "metadata": {},
   "source": [
    "The following is the Amazon Polly synthesis speech output in SSML format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "345f9d0a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "polly_ssml = polly_ssml.split(\"\\n\")\n",
    "polly_ssml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69a0a09-886e-4bb6-9ad6-a6961f603da7",
   "metadata": {},
   "source": [
    "The SSML file provides both the video summary sentences and their speech durations, which represent the time Amazon Polly takes to vocalize each sentence. We will extract these values to align the synthesized speech with the video timeline in the next several steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1e7e41e-102a-45b9-bace-8f0e1a61e6d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "summarized_sentences = []\n",
    "speech_durations = []\n",
    "\n",
    "for i in range(len(polly_ssml) - 1):\n",
    "    curr = polly_ssml[i]\n",
    "    next = polly_ssml[i + 1]\n",
    "    if curr.strip() == \"\" or next.strip() == \"\":\n",
    "        continue\n",
    "    curr = json.loads(curr)\n",
    "    next = json.loads(next)\n",
    "    summarized_sentences.append(curr[\"value\"])\n",
    "    speech_durations.append(int(next[\"time\"]) - int(curr[\"time\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48aeeb3e",
   "metadata": {},
   "source": [
    "## Select most relevant video shots/scenes\n",
    "\n",
    "We need to select the most relevant video frame sequence to match with every sentence in the summarized content. Thus, we use text embedding to perform the sentence similarity task, which determines how similar two texts are.\n",
    "\n",
    "Sentence similarity models transform input texts into vectors (embeddings) that capture semantic information and calculate the proximity or similarity between them.\n",
    "\n",
    "In this step, we use **Text Embedding Model** with [Amazon Bedrock](https://aws.amazon.com/bedrock/) to create the embeddings for every sentence in the original subtitle and in the video summary.\n",
    "\n",
    "First, we get the original subtitle file and do some processings to break it down into sentences with start times and end times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41f0c51a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(video['transcript'].vtt_file, 'r', encoding='utf-8') as file:\n",
    "    subtitle = file.read()\n",
    "\n",
    "if subtitle.startswith(\"WEBVTT\"):\n",
    "    subtitle = subtitle[len(\"WEBVTT\"):].lstrip()\n",
    "\n",
    "print(subtitle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b5bd7bb-aeaa-4cbf-a26c-97f9c19e3091",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def srt_to_arrays(s):\n",
    "    \"\"\"\n",
    "    Converts the given transcription in SRT/WEBVTT format to list of sentences and their corresponding timecodes.\n",
    "    Args:\n",
    "       s - transcription in SRT/WEBVTT format.\n",
    "    Returns:\n",
    "       A list of dictionaries, where each dictionary represents a sentence and its corresponding start time and end time.\n",
    "    \"\"\"\n",
    "    sentences = [line.strip() for line in re.findall(r\"\\d+\\n.*?\\n(.*?)\\n\", s)]\n",
    "\n",
    "    def get_time(s):\n",
    "        return re.findall(r\"\\d{2}:\\d{2}:\\d{2}.\\d{3}\", s)\n",
    "\n",
    "    def time_to_ms(time_str):\n",
    "        match = re.match(r\"(\\d+):(\\d+):(\\d+)[.,](\\d+)\", time_str)\n",
    "        h, m, s, ms = match.groups()\n",
    "        return int(h) * 3600000 + int(m) * 60000 + int(s) * 1000 + int(ms)\n",
    "\n",
    "    startTimes = get_time(s)[::2]\n",
    "    endTimes = get_time(s)[1::2]\n",
    "    startTimes_ms = [time_to_ms(time) for time in startTimes]\n",
    "    endTimes_ms = [time_to_ms(time) for time in endTimes]\n",
    "\n",
    "    complete_sentences = []\n",
    "    complete_startTimes_ms = []\n",
    "    complete_endTimes_ms = []\n",
    "\n",
    "    startTime_ms = -1\n",
    "    endTime_ms = -1\n",
    "    sentence = \"\"\n",
    "    for i in range(len(sentences)):\n",
    "        if startTime_ms == -1:\n",
    "            startTime_ms = startTimes_ms[i]\n",
    "        sentence += \" \" + sentences[i]\n",
    "        if (\n",
    "            sentences[i].endswith(\".\")\n",
    "            or sentences[i].endswith(\"?\")\n",
    "            or sentences[i].endswith(\"!\")\n",
    "            or i == len(sentences) - 1\n",
    "        ):\n",
    "            endTime_ms = endTimes_ms[i]\n",
    "            complete_sentences.append(sentence)\n",
    "            complete_startTimes_ms.append(startTime_ms)\n",
    "            complete_endTimes_ms.append(endTime_ms)\n",
    "            startTime_ms = -1\n",
    "            endTime_ms = -1\n",
    "            sentence = \"\"\n",
    "    processed_transcript = []\n",
    "    for i in range(len(complete_sentences)):\n",
    "        processed_transcript.append(\n",
    "            {\n",
    "                \"sentence_startTime\": complete_startTimes_ms[i],\n",
    "                \"sentence_endTime\": complete_endTimes_ms[i],\n",
    "                \"sentence\": complete_sentences[i],\n",
    "            }\n",
    "        )\n",
    "    return processed_transcript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d95579d-84c3-4e12-a3dc-174610a0e6eb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "processed_transcript = srt_to_arrays(subtitle)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7efb3f31-538d-401a-ae82-a827b279ddfb",
   "metadata": {},
   "source": [
    "Let's visualize some sentences from the video's original transcript:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "140ebd7f-e9f7-4b0d-a502-a4530b6d31f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "original_sentences = [item['sentence'] for item in processed_transcript]\n",
    "original_sentences[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2d932e",
   "metadata": {},
   "source": [
    "Next, we create the text embeddings for every sentence in the original subtitle and in the video summary. The following code gives an example of how text embedding using Amazon Bedrock API works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3de19961",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def find_matching_sentences(original_sentences, summarized_sentences):\n",
    "    \"\"\"\n",
    "    Calculates the similarity between the given original sentences and the summarized sentences.\n",
    "    Args:\n",
    "       original_sentences - sentences extacted from the original video\n",
    "       summarized_sentences - sentences extacted from the video summary\n",
    "    Return:\n",
    "       best_matching_indices - list of indices indicating which original sentence best matches each summarized sentence\n",
    "       similarity_matrix - sentences similarity matrix\n",
    "    \n",
    "    \"\"\"\n",
    "    def np_cosine_similarity(original_embeddings, summarized_embeddings):\n",
    "        \"\"\"\n",
    "        We use `Cosine similarity` to measure similarities between two vectors.\n",
    "        \"\"\"\n",
    "        dot_products = np.dot(summarized_embeddings, original_embeddings.T)\n",
    "        summarized_norms = np.linalg.norm(summarized_embeddings, axis=1)\n",
    "        original_norms = np.linalg.norm(original_embeddings, axis=1)\n",
    "        similarity_matrix = dot_products / summarized_norms[:, None] / original_norms[None, :]\n",
    "        return similarity_matrix\n",
    "        \n",
    "    model_id = \"amazon.titan-embed-image-v1\"\n",
    "    accept = \"application/json\"\n",
    "    content_type = \"application/json\"\n",
    "    original_embeddings = []\n",
    "    for str in original_sentences:\n",
    "        body = json.dumps({\"inputText\": str})\n",
    "        response = bedrock_client.invoke_model(\n",
    "            body=body, modelId=model_id, accept=accept, contentType=content_type\n",
    "        )\n",
    "        response_body = json.loads(response[\"body\"].read())\n",
    "        original_embeddings.append(response_body.get(\"embedding\"))\n",
    "    original_embeddings = np.array(original_embeddings)\n",
    "\n",
    "    summarized_embeddings = []\n",
    "    for str in summarized_sentences:\n",
    "        body = json.dumps({\"inputText\": str})\n",
    "        response = bedrock_client.invoke_model(\n",
    "            body=body, modelId=model_id, accept=accept, contentType=content_type\n",
    "        )\n",
    "        response_body = json.loads(response[\"body\"].read())\n",
    "        summarized_embeddings.append(response_body.get(\"embedding\"))\n",
    "    summarized_embeddings = np.array(summarized_embeddings)\n",
    "\n",
    "    similarity_matrix = np_cosine_similarity(original_embeddings, summarized_embeddings)\n",
    "    best_matching_indices = []\n",
    "    len_summarized_sentences = len(summarized_sentences)\n",
    "    len_original_sentences = len(original_sentences)\n",
    "\n",
    "    # Find the best matching sentences.\n",
    "    dp = np.zeros([len_summarized_sentences, len_original_sentences], dtype=float)\n",
    "    for i in range(0, len_summarized_sentences):\n",
    "        for j in range(0, len_original_sentences):\n",
    "            if i == 0:\n",
    "                dp[i][j] = similarity_matrix[i][j]\n",
    "            else:\n",
    "                max_score = -1\n",
    "                for k in range(0, j):\n",
    "                    if similarity_matrix[i][j] > 0 and dp[i - 1][k] > 0:\n",
    "                        max_score = max(\n",
    "                            max_score, similarity_matrix[i][j] + dp[i - 1][k]\n",
    "                        )\n",
    "                dp[i][j] = max_score\n",
    "\n",
    "    j = len_original_sentences\n",
    "\n",
    "    for i in range(len_summarized_sentences - 1, -1, -1):\n",
    "        arr = dp[i][:j]\n",
    "        idx = np.argmax(arr)\n",
    "        best_matching_indices.append(idx)\n",
    "        j = idx\n",
    "    best_matching_indices.reverse()\n",
    "\n",
    "    return best_matching_indices, similarity_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a97b08a-4ffa-4a51-a5f4-329210421681",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "best_matching_indices, similarity_matrix = find_matching_sentences(original_sentences, summarized_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ef055d4",
   "metadata": {},
   "source": [
    "This will return the similarity matrix result as follow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ed777de-070a-448c-90ee-3aaaff5dbbc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3f486a9",
   "metadata": {},
   "source": [
    "You can interpret the prior result as: the first row of the matrix corresponds to the first sentence in the summarized content and all the columns show its similarity scores to the sentences in the original text. Similarity values typically range between -1 and 1, where 1 indicates that the vectors are identical or very similar; 0 indicates that the vectors are orthogonal (not correlated) and have no similarity; -1 indicates that the vectors are diametrically opposed or very dissimilar.\n",
    "\n",
    "From the similarity matrix, we identify the top-k highest similarity scores for each sentence in the summarized content, thereby aligning them with the most similar sentences in the original text. Each sentence in the original text also has its corresponding timestamp (i.e. startTime, endTime) stored in the original subtitle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f8f9d01-2b2c-40bf-b798-0655f50a25aa",
   "metadata": {},
   "source": [
    "By incorporating both the duration of Polly audio for each summarized sentence and the timestamps from the original subtitle file, we can then select the timestamp sequence for the most relevant frames corresponding to each summarized sentence. The length of each selected video segment for a summarized sentence will be aligned with the length of its narration audio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b2c77b-bb75-42b0-9d37-4e38ee24524b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_timecodes(best_matching_indices, idx, endTimes, duration, timecodes):\n",
    "    \"\"\"\n",
    "    Calculate the best start and end time for each summarized sentence aligned with the timecode from the original sentences\n",
    "    Args:\n",
    "      best_matching_indices - the indices from the original sentence that is most similar with the summarized sentences.\n",
    "      idx - index from the summarized sentences to process\n",
    "      endTimes - the endtime from the original sentences\n",
    "      duration - speech duration for the synthesized sentences from the summarized text\n",
    "      timecodes - timecode used for calculating the best placement for the summarized text within the video.\n",
    "    Return:\n",
    "      \n",
    "    \"\"\"\n",
    "    best_matching_idx = best_matching_indices[idx]\n",
    "    startTime = int(endTimes[best_matching_idx]) - duration\n",
    "    carry = max(0, timecodes[len(timecodes) - 1][1] - startTime)\n",
    "    startTime += carry\n",
    "    endTime = int(endTimes[best_matching_idx]) + carry\n",
    "    return startTime, endTime\n",
    "\n",
    "def ms_to_timecode(ms, drop_frame=False):\n",
    "    \"\"\"\n",
    "    Convert milliseconds to SMPTE timecode\n",
    "    Args:\n",
    "        ms: milliseconds\n",
    "        drop_frame: Boolean, True for drop frame, False for non-drop frame\n",
    "    Return:\n",
    "        string in HH:MM:SS:FF or HH:MM:SS;FF format\n",
    "    \"\"\"\n",
    "    total_frames = int(ms * (29.97 if drop_frame else 30) / 1000)\n",
    "    frames = total_frames % 30\n",
    "    \n",
    "    total_seconds = total_frames // 30\n",
    "    seconds = total_seconds % 60\n",
    "    \n",
    "    total_minutes = total_seconds // 60\n",
    "    minutes = total_minutes % 60\n",
    "    \n",
    "    hours = total_minutes // 60\n",
    "    separator = ';' if drop_frame else ':'    \n",
    "    return f\"{hours:02d}:{minutes:02d}:{seconds:02d}{separator}{frames:02d}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6690d90e-9cfb-412d-9098-43811db8b53f",
   "metadata": {},
   "outputs": [],
   "source": [
    "intro_time = float(transcript_file[\"results\"][\"items\"][0][\"start_time\"]) * 1000\n",
    "\n",
    "timecodes = [[0, intro_time]]\n",
    "for i in range(len(summarized_sentences)):\n",
    "    startTime, endTime = get_timecodes(\n",
    "        best_matching_indices,\n",
    "        i,\n",
    "        [item['sentence_endTime'] for item in processed_transcript],\n",
    "        speech_durations[i],\n",
    "        timecodes,\n",
    "    )\n",
    "    timecodes.append([startTime, endTime])\n",
    "creditTime = endTime + 3500\n",
    "timecodes.append([endTime, creditTime])\n",
    "timecodes_text = \"\"\n",
    "for timecode in timecodes:\n",
    "    timecodes_text += (\n",
    "        ms_to_timecode(timecode[0], True)\n",
    "        + \",\"\n",
    "        + ms_to_timecode(timecode[1], True)\n",
    "        + \"\\n\"\n",
    "    )\n",
    "to_json = lambda s: [\n",
    "    {\"StartTimecode\": t1, \"EndTimecode\": t2}\n",
    "    for t1, t2 in (line.split(\",\") for line in s.split(\"\\n\") if line.strip())\n",
    "]\n",
    "timecodes = to_json(timecodes_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "030714ef-a2bd-46e8-b895-a5d29785cfac",
   "metadata": {},
   "source": [
    "Here are the generated timecodes that will be used for AWS Elemental MediaConvert input clipping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44a603b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "timecodes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74ad03bf-b48b-4e0e-9f35-bdd629f12468",
   "metadata": {},
   "source": [
    "You can now generate the audio narration from the video summary in MP3 format using Amazon Polly. Keep in mind to escape special characters in the summarized text for SSML compatibility as well as create SSML markup with appropriate breaks for intro timing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70e7972d-f9d5-4bdf-8a18-be1e7cea87d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "escaped_summarized_text = (\n",
    "        summarized_text.replace(\"&\", \"&amp;\")\n",
    "        .replace('\"', \"&quot;\")\n",
    "        .replace(\"'\", \"&apos;\")\n",
    "        .replace(\"<\", \"&lt;\")\n",
    "        .replace(\">\", \"&gt;\")\n",
    "    )\n",
    "ssml = \"<speak>\\n\"\n",
    "break_time = intro_time\n",
    "\n",
    "while break_time > 10000:  # maximum break time in Polly is 10s\n",
    "    ssml += '<break time = \"' + str(break_time) + 'ms\"/>'\n",
    "    break_time -= 10000\n",
    "ssml += '<break time = \"' + str(break_time) + 'ms\"/>'\n",
    "ssml += escaped_summarized_text\n",
    "ssml += \"</speak>\"\n",
    "\n",
    "response = polly_client.synthesize_speech(\n",
    "    Engine=\"neural\",\n",
    "    OutputFormat=\"mp3\",\n",
    "    Text=ssml,\n",
    "    TextType=\"ssml\",\n",
    "    VoiceId=voice_id,\n",
    ")\n",
    "\n",
    "if \"AudioStream\" in response:\n",
    "    with response[\"AudioStream\"] as stream:\n",
    "        audio_narration = stream.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf16ae98-439f-4211-9ff8-550f6bf77d06",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ssml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff4f72f3-bbad-4d89-ad4f-afa62e06fa83",
   "metadata": {},
   "source": [
    "We upload the audio narration into Amazon S3 bucket ready for the video transcoding step with AWS Elemental MediaConvert."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccab1d74-942d-419a-a9e8-e7c9facf3013",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_client = boto3.client(\"s3\")\n",
    "s3_bucket = session[\"bucket\"]\n",
    "audio_narration_filename = os.path.splitext(os.path.basename(video['path']))[0] + \".mp3\"\n",
    "s3_client.put_object(\n",
    "    Body=audio_narration, Bucket=s3_bucket, Key=audio_narration_filename, ContentType=\"audio/mpeg\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4046c3b2",
   "metadata": {},
   "source": [
    "## Create MediaConvert assembly workflows\n",
    "\n",
    "We use the sequence of the timecodes as parameters to create AWS Elemental MediaConvert assembly workflows to performs basic input clipping.\n",
    "\n",
    "By combining it with the MP3 audio from Amazon Polly and along with the possibility of incorporating background music of your preference, you can ultimately achieve the final video summarization output."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e622269-11c1-42ce-9318-7d9c2f7b0c5c",
   "metadata": {
    "tags": []
   },
   "source": [
    "Let's start the assembly workflow from the original video input. An assembly workflow is a MediaConvert job that performs basic input clipping and stitching to assemble output assets from one or different sources without requiring separate editing software."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac99a505-65b2-4fbe-a932-07f5d3bdb631",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "iam_role = session[\"MediaConvertRole\"]\n",
    "input_video_path = video[\"url\"]\n",
    "output_video_path = f\"s3://{s3_bucket}/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce6014b0-ce58-49bd-8c37-28c8b8918e20",
   "metadata": {},
   "outputs": [],
   "source": [
    "video[\"url\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f08f8db3-2022-4aae-8baa-a25243105a5c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "media_convert = boto3.client(\"mediaconvert\")\n",
    "response = media_convert.create_job(\n",
    "    Queue=\"Default\",\n",
    "    UserMetadata={},\n",
    "    Role=iam_role,\n",
    "    Settings={\n",
    "        \"TimecodeConfig\": {\"Source\": \"ZEROBASED\"},\n",
    "        \"OutputGroups\": [\n",
    "            {\n",
    "                \"Name\": \"File Group\",\n",
    "                \"Outputs\": [\n",
    "                    {\n",
    "                        \"ContainerSettings\": {\n",
    "                            \"Container\": \"MP4\",\n",
    "                            \"Mp4Settings\": {},\n",
    "                        },\n",
    "                        \"VideoDescription\": {\n",
    "                            \"CodecSettings\": {\n",
    "                                \"Codec\": \"H_264\",\n",
    "                                \"H264Settings\": {\n",
    "                                    \"MaxBitrate\": 40000000,\n",
    "                                    \"RateControlMode\": \"QVBR\",\n",
    "                                    \"SceneChangeDetect\": \"TRANSITION_DETECTION\",\n",
    "                                },\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                ],\n",
    "                \"OutputGroupSettings\": {\n",
    "                    \"Type\": \"FILE_GROUP_SETTINGS\",\n",
    "                    \"FileGroupSettings\": {\"Destination\": output_video_path},\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "        \"Inputs\": [\n",
    "            {\n",
    "                \"VideoSelector\": {},\n",
    "                \"TimecodeSource\": \"ZEROBASED\",\n",
    "                \"FileInput\": input_video_path,\n",
    "                \"InputClippings\": timecodes,\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    AccelerationSettings={\"Mode\": \"DISABLED\"},\n",
    "    StatusUpdateInterval=\"SECONDS_60\",\n",
    "    Priority=0,\n",
    ")\n",
    "\n",
    "job_complete = False\n",
    "\n",
    "while not job_complete:\n",
    "    job_response = media_convert.get_job(Id=response[\"Job\"][\"Id\"])\n",
    "    \n",
    "    job_status = job_response['Job']['Status']\n",
    "    print(f\"MediaConvert job status: {job_status}\")\n",
    "    \n",
    "    if job_status == 'COMPLETE':\n",
    "        print(\"Job is complete!\")\n",
    "        job_complete = True\n",
    "    elif job_status == 'ERROR':\n",
    "        print(\"Job has failed.\")\n",
    "        job_complete = True\n",
    "    else:\n",
    "        time.sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a672bd8-f338-49d2-a4d1-7387aca9ef83",
   "metadata": {},
   "source": [
    "Finally, you create audio tracks in the output and associate a single audio selector with each output track. In addition, you could also add a subtitle into the final video ouput. You could generate a subtitle for the video summary as follow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8cf7378-b0a4-429d-a2f3-b29bb2ec6ac2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "video_summary_subtitle = \"\"\n",
    "start = intro_time\n",
    "\n",
    "def split_long_lines(text, max_line_length):\n",
    "    words = text.split()\n",
    "    lines = []\n",
    "    current_line = []\n",
    "    current_length = 0\n",
    "\n",
    "    for word in words:\n",
    "        if current_length + len(word) + len(current_line) > max_line_length:\n",
    "            lines.append(\" \".join(current_line))\n",
    "            current_line = []\n",
    "            current_length = 0\n",
    "        current_line.append(word)\n",
    "        current_length += len(word) + 1\n",
    "\n",
    "    if current_line:\n",
    "        lines.append(\" \".join(current_line))\n",
    "\n",
    "    return lines\n",
    "\n",
    "def milliseconds_to_subtitleTimeFormat(ms):\n",
    "    return \"{:02d}:{:02d}:{:02d},{:03d}\".format(\n",
    "        int((ms // 3600000) % 24),  # hours\n",
    "        int((ms // 60000) % 60),  # minutes\n",
    "        int((ms // 1000) % 60),  # seconds\n",
    "        int(ms % 1000),  # milliseconds\n",
    "    )\n",
    "\n",
    "for i in range(len(summarized_sentences)):\n",
    "    end = start + speech_durations[i]\n",
    "    video_summary_subtitle += f\"{i+1}\\n\"\n",
    "    video_summary_subtitle += f\"{milliseconds_to_subtitleTimeFormat(start)} --> {milliseconds_to_subtitleTimeFormat(end)}\\n\"\n",
    "    sentence_lines = split_long_lines(summarized_sentences[i], 90)\n",
    "    for line in sentence_lines:\n",
    "        video_summary_subtitle += f\"{line}\\n\"\n",
    "    video_summary_subtitle += \"\\n\"\n",
    "    start = end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2cee4cf-cc78-4948-8c82-decdc68994cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "video_summary_subtitle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52990b82-c585-4aed-87c9-dfd8026146cc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "subtitle_filename = os.path.splitext(os.path.basename(video['path']))[0] + \".srt\"\n",
    "s3_client.put_object(\n",
    "    Body=video_summary_subtitle, Bucket=s3_bucket, Key=subtitle_filename\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab15bb2-d99c-4c68-8b24-8b3bc32d3fe1",
   "metadata": {},
   "source": [
    "Finally, you create a MediaConvert job for the final video ouput."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27cc265f-19d2-4e79-88bc-00d0c19877a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "input_video_path = f\"s3://{s3_bucket}/{video['path']}\"\n",
    "audio_file_path = f\"s3://{s3_bucket}/{audio_narration_filename}\"\n",
    "subtitle_file_path = f\"s3://{s3_bucket}/{subtitle_filename}\"\n",
    "output_video_path = f\"s3://{s3_bucket}/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba312ef-46a2-4166-ba5b-61a613ba94f3",
   "metadata": {},
   "source": [
    "In the following step, we are using a [AWS Elemental MediaConvert](https://aws.amazon.com/mediaconvert/) job to apply the narrated voice and the subtitles on the original video. The output is written to S3 bucket for downstream consumption. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c8ec775-5ad9-4ab8-a95b-5b3090d40b55",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = media_convert.create_job(\n",
    "    Queue=\"Default\",\n",
    "    UserMetadata={},\n",
    "    Role=iam_role,\n",
    "    Settings={\n",
    "        \"TimecodeConfig\": {\"Source\": \"ZEROBASED\"},\n",
    "        \"OutputGroups\": [\n",
    "            {\n",
    "                \"Name\": \"File Group\",\n",
    "                \"Outputs\": [\n",
    "                    {\n",
    "                        \"ContainerSettings\": {\n",
    "                            \"Container\": \"MP4\",\n",
    "                            \"Mp4Settings\": {},\n",
    "                        },\n",
    "                        \"VideoDescription\": {\n",
    "                            \"CodecSettings\": {\n",
    "                                \"Codec\": \"H_264\",\n",
    "                                \"H264Settings\": {\n",
    "                                    \"MaxBitrate\": 40000000,\n",
    "                                    \"RateControlMode\": \"QVBR\",\n",
    "                                    \"SceneChangeDetect\": \"TRANSITION_DETECTION\",\n",
    "                                },\n",
    "                            }\n",
    "                        },\n",
    "                        \"NameModifier\": \"_summary\",\n",
    "                        \"AudioDescriptions\": [\n",
    "                            {\n",
    "                                \"AudioSourceName\": \"Audio Selector Group 1\",\n",
    "                                \"CodecSettings\": {\n",
    "                                    \"Codec\": \"AAC\",\n",
    "                                    \"AacSettings\": {\n",
    "                                        \"Bitrate\": 96000,\n",
    "                                        \"CodingMode\": \"CODING_MODE_2_0\",\n",
    "                                        \"SampleRate\": 48000,\n",
    "                                    },\n",
    "                                },\n",
    "                            }\n",
    "                        ],\n",
    "                        \"CaptionDescriptions\": [\n",
    "                            {\n",
    "                                \"CaptionSelectorName\": \"Captions Selector 1\",\n",
    "                                \"DestinationSettings\": {\n",
    "                                    \"DestinationType\": \"BURN_IN\",\n",
    "                                    \"BurninDestinationSettings\": {\n",
    "                                        \"BackgroundOpacity\": 100,\n",
    "                                        \"FontSize\": 18,\n",
    "                                        \"FontColor\": \"WHITE\",\n",
    "                                        \"ApplyFontColor\": \"ALL_TEXT\",\n",
    "                                        \"BackgroundColor\": \"BLACK\",\n",
    "                                    },\n",
    "                                },\n",
    "                            }\n",
    "                        ],\n",
    "                    }\n",
    "                ],\n",
    "                \"OutputGroupSettings\": {\n",
    "                    \"Type\": \"FILE_GROUP_SETTINGS\",\n",
    "                    \"FileGroupSettings\": {\"Destination\": output_video_path},\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "        \"Inputs\": [\n",
    "            {\n",
    "                \"AudioSelectors\": {\n",
    "                    \"Audio Selector 1\": {\n",
    "                        \"DefaultSelection\": \"NOT_DEFAULT\",\n",
    "                        \"ExternalAudioFileInput\": audio_file_path,\n",
    "                    },\n",
    "                },\n",
    "                \"AudioSelectorGroups\": {\n",
    "                    \"Audio Selector Group 1\": {\n",
    "                        \"AudioSelectorNames\": [\"Audio Selector 1\"]\n",
    "                    }\n",
    "                },\n",
    "                \"VideoSelector\": {},\n",
    "                \"TimecodeSource\": \"ZEROBASED\",\n",
    "                \"CaptionSelectors\": {\n",
    "                    \"Captions Selector 1\": {\n",
    "                        \"SourceSettings\": {\n",
    "                            \"SourceType\": \"SRT\",\n",
    "                            \"FileSourceSettings\": {\"SourceFile\": subtitle_file_path},\n",
    "                        }\n",
    "                    }\n",
    "                },\n",
    "                \"FileInput\": input_video_path,\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    AccelerationSettings={\"Mode\": \"DISABLED\"},\n",
    "    StatusUpdateInterval=\"SECONDS_60\",\n",
    "    Priority=0,\n",
    ")\n",
    "\n",
    "job_complete = False\n",
    "\n",
    "while not job_complete:\n",
    "    job_response = media_convert.get_job(Id=response[\"Job\"][\"Id\"])\n",
    "    \n",
    "    job_status = job_response['Job']['Status']\n",
    "    print(f\"MediaConvert job status: {job_status}\")\n",
    "    \n",
    "    if job_status == 'COMPLETE':\n",
    "        print(\"Job is complete!\")\n",
    "        job_complete = True\n",
    "    elif job_status == 'ERROR':\n",
    "        print(\"Job has failed.\")\n",
    "        job_complete = True\n",
    "    else:\n",
    "        time.sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8189efa",
   "metadata": {},
   "source": [
    "## Short-form video output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46b130c0-dcf5-478b-b244-e9688f8410e1",
   "metadata": {},
   "source": [
    "Here is the final video output generated from our summarization process. We preserve the original video's intro before starting our generated narration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "637926aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "video_summary = os.path.splitext(os.path.basename(video['path']))[0] + \"_summary.mp4\"\n",
    "s3_client.download_file(s3_bucket, video_summary, video_summary)\n",
    "display(Video(url=video_summary, width=640, height=360, html_attributes=\"controls muted autoplay\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480292d0-51a9-4737-b1c3-b7b7b76b9250",
   "metadata": {},
   "source": [
    "## Video summarization with visual and audio understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be8b5e4-949f-4656-98f8-4482a3ae49a8",
   "metadata": {},
   "source": [
    "In the previous section, we created a summarized video based solely on the video's transcription by:\n",
    "1. Extracting and summarizing the original video transcript\n",
    "2. Finding the best matching video segments by semantically comparing the summary with the original transcript\n",
    "3. Generating the final video output\n",
    "\n",
    "Now, let's enhance our approach by combining both video visual understanding and transcript analysis to create a more comprehensive summarized video."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a22321c3-fdcc-4423-a559-2363690983bd",
   "metadata": {},
   "source": [
    "To understand the visual content of the video, we'll first use [Amazon Rekognition](https://aws.amazon.com/rekognition/)'s segment detection to identify shot. A shot is a series of interrelated consecutive pictures taken contiguously by a single camera and representing a continuous action in time and space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebe5bd3d-5b00-4543-8fc0-1bebe972d5af",
   "metadata": {},
   "outputs": [],
   "source": [
    "rek_client = boto3.client(\"rekognition\")\n",
    "\n",
    "def start_segment_detection(bucket_videos, video_name):\n",
    "    \"\"\"\n",
    "    Start an Amazon Rekognition segment detection job to detect shots in a video\n",
    "    Args:\n",
    "        bucket_videos - S3 bucket name containing the video\n",
    "        video_name - Name/path of the video file in the S3 bucket\n",
    "        \n",
    "    Returns:\n",
    "        rek_jobId - Job ID of the Rekognition segment detection task\n",
    "    \"\"\"\n",
    "    min_Technical_Cue_Confidence = 80.0\n",
    "    min_Shot_Confidence = 80.0\n",
    "    max_pixel_threshold = 0.1\n",
    "    min_coverage_percentage = 60\n",
    "\n",
    "    response = rek_client.start_segment_detection(\n",
    "        Video={\"S3Object\": {\"Bucket\": bucket_videos, \"Name\": video_name}},\n",
    "        SegmentTypes=[\"SHOT\"],\n",
    "        Filters={\n",
    "            \"ShotFilter\": {\"MinSegmentConfidence\": min_Shot_Confidence},\n",
    "        },\n",
    "    )\n",
    "\n",
    "    rek_jobId = response[\"JobId\"]\n",
    "    return rek_jobId"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac95735-13b7-4b67-8390-22b6292d678d",
   "metadata": {},
   "outputs": [],
   "source": [
    "rek_jobId = start_segment_detection(s3_bucket, f\"{video['output_dir']}/{video['path']}\")\n",
    "while rek_client.get_segment_detection(JobId=rek_jobId)['JobStatus'] == 'IN_PROGRESS':\n",
    "    print(f\"Job {rek_jobId} is still in progress.\")\n",
    "    time.sleep(10)\n",
    "print(\"Completed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b188cdc6-9b98-44c7-9159-c7a32e1d3d2e",
   "metadata": {},
   "source": [
    "Now we'll retrieve the result from Amazon Rekognition's shot detection that includes a list of shots, theirs start time, end time and associated frames.\n",
    "\n",
    "Using ffmpeg, we'll convert these specific video frames into individual PNG images that we can analyze further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d87bb354-077c-4ea3-82f6-424bfd73eb09",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_shot_detection_results(jobId):\n",
    "    \"\"\" Retrieve and process the results from Amazon Rekognition's shot detection job\n",
    "    Args:\n",
    "        jobId - Job ID from the started Rekognition segment detection task\n",
    "    Returns:\n",
    "        shots - List of dictionaries containing shot information including:\n",
    "                - shot_id: unique identifier for the shot\n",
    "                - shot_startTime: start time of the shot in milliseconds\n",
    "                - shot_endTime: end time of the shot in milliseconds\n",
    "                - frames: list of frame extracted from this shot\n",
    "        frames - List of frames extracted from the video\n",
    "    \"\"\"\n",
    "    maxResults = 1000\n",
    "    paginationToken = \"\"\n",
    "    finished = False\n",
    "    firstTime = True\n",
    "\n",
    "    response = rek_client.get_segment_detection(\n",
    "        JobId=jobId, MaxResults=maxResults, NextToken=paginationToken\n",
    "    )\n",
    "\n",
    "    frames = []\n",
    "    shots = []\n",
    "    delta = response[\"Segments\"][0][\"StartTimestampMillis\"]\n",
    "\n",
    "    def get_frames(shot, N):\n",
    "        start_frame = shot[\"StartFrameNumber\"]\n",
    "        end_frame = (\n",
    "            shot[\"EndFrameNumber\"] - 1\n",
    "        )  # frame - 1 to avoid bug not getting the last frame of the video\n",
    "        step = (end_frame - start_frame) / (N - 1)\n",
    "        frames = [int(start_frame + i * step) for i in range(N)]\n",
    "        return frames\n",
    "\n",
    "    for shot in response[\"Segments\"]:\n",
    "        shot_frames = get_frames(shot, 4)\n",
    "        frames.extend(shot_frames)\n",
    "        frames.append(shot[\"StartFrameNumber\"])\n",
    "\n",
    "        shot_startTime = shot[\"StartTimestampMillis\"] - delta\n",
    "        shot_endTime = shot[\"EndTimestampMillis\"] - delta\n",
    "\n",
    "        shots.append(\n",
    "            {\n",
    "                \"shot_id\": f\"{shot_startTime}-{shot_endTime}\",\n",
    "                \"shot_startTime\": shot_startTime,\n",
    "                \"shot_endTime\": shot_endTime,\n",
    "                \"frames\": [{\"frameId\": frame} for frame in shot_frames],\n",
    "            }\n",
    "        )\n",
    "    return shots, frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4a21bd2-0cdc-4363-b36e-d186bd110af5",
   "metadata": {},
   "outputs": [],
   "source": [
    "video_shots, frames = get_shot_detection_results(rek_jobId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3360e3db-c410-425c-a2ed-02873bc24fe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Generating frame images for: {video['path']} ...\")\n",
    "tmp_frames_dir = \"video-summarization/\"\n",
    "os.makedirs(tmp_frames_dir, exist_ok=True)\n",
    "ffmpeg_path = \"/usr/bin/ffmpeg\"\n",
    "\n",
    "frame_list = \"+\".join([f\"eq(n,{frame})\" for frame in frames])\n",
    "output_pattern = f\"{tmp_frames_dir}%d.png\"\n",
    "with open(os.devnull, 'wb') as devnull:\n",
    "    subprocess.run(\n",
    "        [\n",
    "            ffmpeg_path,\n",
    "            \"-i\",\n",
    "            video['path'],\n",
    "            \"-vf\",\n",
    "            f\"select='{frame_list}'\",\n",
    "            \"-vsync\",\n",
    "            \"0\",\n",
    "            \"-frame_pts\",\n",
    "            \"1\",\n",
    "            output_pattern,\n",
    "        ],\n",
    "        stdout=devnull,\n",
    "        stderr=devnull\n",
    "    )\n",
    "print(\"Completed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a43cccf-110e-48f2-88ed-dbe775f3c400",
   "metadata": {},
   "source": [
    "Let's analyze each detected shot in our video. For each shot, we will generate a description of the visual content using Large Language Model in Amazon Bedrock. We will also match the corresponding transcript of what was said during this shot segment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad4a04db-3148-404a-8778-be7014458561",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_RETRIES = 50\n",
    "INITIAL_BACKOFF = 5\n",
    "\n",
    "def invoke_model_with_retry(body, modelId, accept, contentType):\n",
    "    retries = 0\n",
    "    backoff = INITIAL_BACKOFF\n",
    "\n",
    "    while retries < MAX_RETRIES:\n",
    "        try:\n",
    "            response = bedrock_client.invoke_model(\n",
    "                body=body, modelId=modelId, accept=accept, contentType=contentType\n",
    "            )\n",
    "            return response\n",
    "        except ClientError as e:\n",
    "            error_code = e.response['Error']['Code']\n",
    "            print(f\"Error: {error_code}. Retrying in {backoff} seconds...\")\n",
    "            time.sleep(backoff)\n",
    "            retries += 1\n",
    "            backoff += 1\n",
    "    \n",
    "    raise Exception(\"Max retries reached. Unable to invoke model.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be62146f-ba8d-4fd8-9862-b5f19271acd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_shot_description(shot):\n",
    "    \"\"\"\n",
    "    Generate a natural language description of a video shot using LLM in Amazon Bedrock\n",
    "    Args:\n",
    "        shot - Dictionary containing shot information including:\n",
    "                - shot_id: unique identifier for the shot\n",
    "                - shot_startTime: start time of the shot in milliseconds\n",
    "                - shot_endTime: end time of the shot in milliseconds\n",
    "                - frames: list of frame extracted from this shot\n",
    "              \n",
    "    Returns:\n",
    "        response_body - String containing the generated description of the visual content in the shot based on the analyzed frames\n",
    "    \"\"\"\n",
    "    prompt = f\"\"\"Provide a concise description of a video shot based on the given frame images. Focus on creating a cohesive narrative of the entire shot rather than describing each frame individually.\n",
    "        Skip the preamble; go straight into the description.\"\"\"\n",
    "        \n",
    "    body = {\n",
    "        \"messages\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\"type\": \"text\", \"text\": prompt},\n",
    "                ],\n",
    "            }\n",
    "        ],\n",
    "        \"anthropic_version\": \"bedrock-2023-05-31\",\n",
    "        \"max_tokens\": 512,\n",
    "    }\n",
    "\n",
    "    for index, value in enumerate(shot[\"frames\"]):\n",
    "        with open(f\"video-summarization/{value['frameId']}.png\", \"rb\") as image_file:\n",
    "            file_content = image_file.read()\n",
    "            base64_image_string = base64.b64encode(file_content).decode()\n",
    "            body[\"messages\"][0][\"content\"].append({\n",
    "                \"type\": \"image\",\n",
    "                \"source\": {\n",
    "                    \"type\": \"base64\",\n",
    "                    \"media_type\": \"image/png\",\n",
    "                    \"data\": base64_image_string,\n",
    "                },\n",
    "            })\n",
    "    response = invoke_model_with_retry(\n",
    "        body=json.dumps(body), modelId=model_id, accept=accept, contentType=content_type\n",
    "    )\n",
    "    response_body = json.loads(response[\"body\"].read())\n",
    "    response_body = response_body[\"content\"][0][\"text\"]\n",
    "\n",
    "    return response_body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fef0b651-b599-4917-b0bd-08742e9df600",
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_shot_transcript(shot_startTime, shot_endTime, transcript):\n",
    "    \"\"\"\n",
    "    Extract relevant transcript that corresponds to a specific video shot's time range\n",
    "    Args:\n",
    "        shot_startTime - Start time of the shot in milliseconds\n",
    "        shot_endTime - End time of the shot in milliseconds\n",
    "        transcript - List of dictionaries containing sentence information including:\n",
    "                    - sentence_startTime: start time of the sentence\n",
    "                    - sentence_endTime: end time of the sentence\n",
    "                    - sentence: the transcript text\n",
    "                    \n",
    "    Returns:\n",
    "        relevant_transcript - String containing concatenated sentences that overlap with the shot's time range by at least 1 second\n",
    "    \"\"\"\n",
    "    relevant_transcript = \"\"\n",
    "    for item in transcript:\n",
    "        if item[\"sentence_startTime\"] >= shot_endTime:\n",
    "            break\n",
    "        if item[\"sentence_endTime\"] <= shot_startTime:\n",
    "            continue\n",
    "        delta_start = max(item[\"sentence_startTime\"], shot_startTime)\n",
    "        delta_end = min(item[\"sentence_endTime\"], shot_endTime)\n",
    "        if delta_end - delta_start >= 1000:\n",
    "            relevant_transcript += item[\"sentence\"] + \"; \"\n",
    "    return relevant_transcript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb535b2f-6549-4729-aa77-a20b45dc09cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "for counter, shot in enumerate(video_shots, start=1):\n",
    "    print(f'\\nSHOT {counter}/{len(video_shots)}: from {shot[\"shot_startTime\"] }ms to {shot[\"shot_endTime\"] }ms =======\\n')\n",
    "    for frame in shot[\"frames\"]:\n",
    "        display(DisplayImage(f\"video-summarization/{frame['frameId']}.png\", width=90, height=70))\n",
    "    \n",
    "    shot['shot_description'] = generate_shot_description(shot)\n",
    "    shot['shot_transcript'] = add_shot_transcript(shot['shot_startTime'], shot['shot_endTime'], processed_transcript)\n",
    "    \n",
    "    print(f'Shot description: {shot[\"shot_description\"]}\\n')\n",
    "    print(f'Shot transcript: {shot[\"shot_transcript\"]}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0707fd29-fdf1-4848-b8ec-d83a40d987dc",
   "metadata": {},
   "source": [
    "Now as we have a description and transcript for each shot, let's store them in [Amazon OpenSearch Serverless](https://aws.amazon.com/opensearch-service/features/serverless/) vector database for semantic search capabilities.\n",
    "\n",
    "Firstly, we will create an OpenSearch index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59a875b6-1d2a-462d-8d87-56e7e8aed73c",
   "metadata": {},
   "outputs": [],
   "source": [
    "region = sagemaker_resources[\"region\"]\n",
    "aoss_host = session[\"AOSSCollectionEndpoint\"]\n",
    "aoss_index = \"video-summarization-index\"\n",
    "text_embedding_model = \"amazon.titan-embed-text-v2:0\"\n",
    "text_embedding_dimension = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b71ec38-0586-4688-8c9c-cd9011639856",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_opensearch_index(host, region, index, len_embedding):\n",
    "    \"\"\"\n",
    "    Create an OpenSearch Serverless index with vector search capabilities\n",
    "    Args:\n",
    "        host - OpenSearch domain endpoint URL\n",
    "        region - AWS region where the OpenSearch domain is hosted\n",
    "        index - Name of the index to create\n",
    "        len_embedding - Dimension size of the vector embeddings\n",
    "    Returns:\n",
    "        client - Configured OpenSearch client object\n",
    "    \"\"\"\n",
    "    host = host.split(\"://\")[1] if \"://\" in host else host\n",
    "    credentials = boto3.Session().get_credentials()\n",
    "    auth = AWSV4SignerAuth(credentials, region, \"aoss\")\n",
    "\n",
    "    client = OpenSearch(\n",
    "        hosts=[{\"host\": host, \"port\": 443}],\n",
    "        http_auth=auth,\n",
    "        use_ssl=True,\n",
    "        verify_certs=True,\n",
    "        connection_class=RequestsHttpConnection,\n",
    "        pool_maxsize=20,\n",
    "    )\n",
    "\n",
    "    exist = client.indices.exists(index)\n",
    "    if not exist:\n",
    "        print(\"Creating index\")\n",
    "        index_body = {\n",
    "            \"mappings\": {\n",
    "                \"properties\": {\n",
    "                    \"video_name\": {\"type\": \"text\"},\n",
    "                    \"shot_id\": {\"type\": \"text\"},\n",
    "                    \"shot_startTime\": {\"type\": \"text\"},\n",
    "                    \"shot_endTime\": {\"type\": \"text\"},\n",
    "                    \"shot_description\": {\"type\": \"text\"},\n",
    "                    \"shot_transcript\": {\"type\": \"text\"},\n",
    "                    \"shot_desc_vector\": {\n",
    "                        \"type\": \"knn_vector\",\n",
    "                        \"dimension\": len_embedding,\n",
    "                        \"method\": {\n",
    "                            \"engine\": \"nmslib\",\n",
    "                            \"space_type\": \"cosinesimil\",\n",
    "                            \"name\": \"hnsw\",\n",
    "                            \"parameters\": {\"ef_construction\": 512, \"m\": 16},\n",
    "                        },\n",
    "                    },\n",
    "                    \"shot_transcript_vector\": {\n",
    "                        \"type\": \"knn_vector\",\n",
    "                        \"dimension\": len_embedding,\n",
    "                        \"method\": {\n",
    "                            \"engine\": \"nmslib\",\n",
    "                            \"space_type\": \"cosinesimil\",\n",
    "                            \"name\": \"hnsw\",\n",
    "                            \"parameters\": {\"ef_construction\": 512, \"m\": 16},\n",
    "                        },\n",
    "                    }\n",
    "                }\n",
    "            },\n",
    "            \"settings\": {\n",
    "                \"index\": {\n",
    "                    \"number_of_shards\": 2,\n",
    "                    \"knn.algo_param\": {\"ef_search\": 512},\n",
    "                    \"knn\": True,\n",
    "                }\n",
    "            },\n",
    "        }\n",
    "\n",
    "        response = client.indices.create(index, body=index_body)\n",
    "\n",
    "    print(\"Completed!\")\n",
    "    return client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "def1f202-5d1f-422e-a251-dff8db175b0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "aoss_client = create_opensearch_index(aoss_host, region, aoss_index, text_embedding_dimension)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a86d5107-6138-4d74-a515-5c63b6625e63",
   "metadata": {},
   "source": [
    "Using text embedding model in Amazon Bedrock, we generate text embeddings for shot descriptions and transcripts before inserting these data into OpenSearch index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86bb8141-f760-4b03-866e-be882f84118f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_text_embedding(text_embedding_model, text):\n",
    "    \"\"\"\n",
    "    Generate vector embeddings for text using Amazon Bedrock's embedding model\n",
    "    Args:\n",
    "        text_embedding_model - Model id of the Bedrock embedding model\n",
    "        text - Input text to generate embeddings\n",
    "        \n",
    "    Returns:\n",
    "        embedding - Result text's vector embedding\n",
    "    \"\"\"\n",
    "    if not text.strip():\n",
    "        text = \"No transcript\"\n",
    "    body = json.dumps({\"inputText\": text, \"dimensions\": 1024, \"normalize\": True})\n",
    "    response = bedrock_client.invoke_model(\n",
    "        body=body, modelId=text_embedding_model, accept=accept, contentType=content_type\n",
    "    )\n",
    "    response_body = json.loads(response[\"body\"].read())\n",
    "    return response_body.get(\"embedding\")\n",
    "        \n",
    "print(f\"Insert embeddings to AOSS index ...\")\n",
    "for counter, shot in enumerate(video_shots, start=1):\n",
    "    shot_desc_embedding = get_text_embedding(text_embedding_model, shot[\"shot_description\"])\n",
    "    shot_transcript_embedding = get_text_embedding(text_embedding_model, shot[\"shot_transcript\"])\n",
    "    embedding_request_body = json.dumps(\n",
    "        {\n",
    "            \"video_name\": video[\"path\"],\n",
    "            \"shot_id\": shot[\"shot_id\"],\n",
    "            \"shot_startTime\": shot[\"shot_startTime\"],\n",
    "            \"shot_endTime\": shot[\"shot_endTime\"],\n",
    "            \"shot_description\": shot[\"shot_description\"],\n",
    "            \"shot_transcript\": shot[\"shot_transcript\"],\n",
    "            \"shot_desc_vector\": shot_desc_embedding,\n",
    "            \"shot_transcript_vector\": shot_transcript_embedding\n",
    "        }\n",
    "    )\n",
    "\n",
    "    response = aoss_client.index(\n",
    "        index=aoss_index,\n",
    "        body=embedding_request_body,\n",
    "        params={\"timeout\": 60},\n",
    "    )\n",
    "\n",
    "print(\"Completed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f22eeaf-a930-4286-97ee-0c64009d74b2",
   "metadata": {},
   "source": [
    "For each sentence in the video summary, we will search for relevant shots in the vector database using shots' description embeddings and shots' transcription embeddings. The total duration of the selected shots will also need to match with the speech duration for each sentence.\n",
    "\n",
    "But we will first make sure the inserted data in OpenSearch is ready to be searched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13610bd3-40fc-4772-b5d7-76596a2ef94c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Waiting for the recent inserted data to be searchable in OpenSearch...\")\n",
    "\n",
    "while True:\n",
    "    try:\n",
    "        result = aoss_client.search(index=aoss_index, body={\"query\": {\"match_all\": {}}})\n",
    "        if result['hits']['total']['value'] == len(video_shots):\n",
    "            print(\"\\nData is now available for search!\")\n",
    "            break\n",
    "    except Exception as e:\n",
    "        print(\".\", end=\"\", flush=True)\n",
    "        time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9033ee6b-8107-4d2f-9574-da5f9555e927",
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_by_text(aoss_index, client, user_query):\n",
    "    \"\"\"\n",
    "    Search for relevant video shots using semantic similarity with user's text query\n",
    "    Args:\n",
    "        aoss_index - Name of the OpenSearch index\n",
    "        client - Configured OpenSearch client object\n",
    "        user_query - Text query from the user\n",
    "        \n",
    "    Returns:\n",
    "        response - List of dictionaries containing matching shots, where each dictionary includes:\n",
    "                  - shot_id: unique identifier for the shot\n",
    "                  - shot_startTime: start time of the shot\n",
    "                  - shot_endTime: end time of the shot\n",
    "                  - shot_description: visual description of the shot\n",
    "                  - shot_transcript: transcript text from the shot\n",
    "                  - score: similarity score of the match\n",
    "    \"\"\"\n",
    "    text_embedding = get_text_embedding(text_embedding_model, user_query)\n",
    "\n",
    "    aoss_query = {\n",
    "        \"size\": 100,\n",
    "        \"query\": {\n",
    "            \"bool\": {\n",
    "                \"should\": [\n",
    "                    {\n",
    "                        \"script_score\": {\n",
    "                            \"query\": {\"match_all\": {}},\n",
    "                            \"script\": {\n",
    "                                \"lang\": \"knn\",\n",
    "                                \"source\": \"knn_score\",\n",
    "                                \"params\": {\n",
    "                                    \"field\": \"shot_desc_vector\",\n",
    "                                    \"query_value\": text_embedding,\n",
    "                                    \"space_type\": \"cosinesimil\",\n",
    "                                },\n",
    "                            },\n",
    "                            \"boost\": 3.0\n",
    "                        }\n",
    "                    },\n",
    "                    {\n",
    "                        \"script_score\": {\n",
    "                            \"query\": {\"match_all\": {}},\n",
    "                            \"script\": {\n",
    "                                \"lang\": \"knn\",\n",
    "                                \"source\": \"knn_score\",\n",
    "                                \"params\": {\n",
    "                                    \"field\": \"shot_transcript_vector\",\n",
    "                                    \"query_value\": text_embedding,\n",
    "                                    \"space_type\": \"cosinesimil\",\n",
    "                                },\n",
    "                            },\n",
    "                            \"boost\": 1.0\n",
    "                        }\n",
    "                    }\n",
    "                ],\n",
    "                \"minimum_should_match\": 1\n",
    "            }\n",
    "        },\n",
    "        \"_source\": [\n",
    "            \"shot_id\",\n",
    "            \"shot_startTime\",\n",
    "            \"shot_endTime\",\n",
    "            \"shot_description\",\n",
    "            \"shot_transcript\",\n",
    "        ],\n",
    "    }\n",
    "\n",
    "    response = client.search(body=aoss_query, index=aoss_index)\n",
    "    hits = response[\"hits\"][\"hits\"]\n",
    "    response = []\n",
    "    for hit in hits:\n",
    "        if hit[\"_score\"] >= 0:  # Set score threshold\n",
    "            response.append(\n",
    "                {\n",
    "                    \"shot_id\": hit[\"_source\"][\"shot_id\"],\n",
    "                    \"shot_startTime\": hit[\"_source\"][\"shot_startTime\"],\n",
    "                    \"shot_endTime\": hit[\"_source\"][\"shot_endTime\"],\n",
    "                    \"shot_description\": hit[\"_source\"][\"shot_description\"],\n",
    "                    \"shot_transcript\": hit[\"_source\"][\"shot_transcript\"],\n",
    "                    \"score\": hit[\"_score\"],\n",
    "                }\n",
    "            )\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c02548fc-4b9e-493e-99ae-f5220aaa5a56",
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_shots(timecodes, sentence, duration):\n",
    "    \"\"\"\n",
    "    Find and select video shots that match a sentence, considering timing constraints\n",
    "    Args:\n",
    "        timecodes - List to store selected shot timecodes [[start_time, end_time], ...]\n",
    "        sentence - Text to search for matching video shots\n",
    "        duration - Required duration for the shots\n",
    "    \"\"\"\n",
    "    relevant_shots = search_by_text(aoss_index, aoss_client, sentence)\n",
    "    if duration is None: # intro\n",
    "        timecodes.append([relevant_shots[0][\"shot_startTime\"], relevant_shots[0][\"shot_endTime\"]])\n",
    "        shot_ids.add(relevant_shots[0][\"shot_id\"])\n",
    "        intro_time = relevant_shots[0][\"shot_endTime\"] - relevant_shots[0][\"shot_startTime\"]\n",
    "    else:\n",
    "        i = 0\n",
    "        while i < len(relevant_shots) and duration > 0:\n",
    "            if relevant_shots[i][\"shot_id\"] in shot_ids:\n",
    "                i += 1\n",
    "                continue\n",
    "            shot_duration = relevant_shots[i][\"shot_endTime\"] - relevant_shots[i][\"shot_startTime\"]\n",
    "            # timecodes.append([relevant_shots[i][\"shot_startTime\"], relevant_shots[i][\"shot_startTime\"] + min(shot_duration, duration)])\n",
    "            timecodes.append([relevant_shots[i][\"shot_endTime\"] - min(shot_duration, duration), relevant_shots[i][\"shot_endTime\"]])\n",
    "            shot_ids.add(relevant_shots[i][\"shot_id\"])\n",
    "            duration -= shot_duration\n",
    "            i += 1\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c913d89-0a8f-41d1-b251-151c8039596c",
   "metadata": {},
   "outputs": [],
   "source": [
    "shot_ids = set()\n",
    "timecodes = []\n",
    "\n",
    "find_shots(timecodes, \"Meridian\", None) # Intro\n",
    "for i in range(len(summarized_sentences)):\n",
    "    find_shots(timecodes, summarized_sentences[i], speech_durations[i])\n",
    "\n",
    "# creditTime = timecodes[-1][1] + 1000\n",
    "# timecodes.append([timecodes[-1][1], creditTime])\n",
    "timecodes_text = \"\"\n",
    "for timecode in timecodes:\n",
    "    timecodes_text += (\n",
    "        ms_to_timecode(timecode[0], True)\n",
    "        + \",\"\n",
    "        + ms_to_timecode(timecode[1], True)\n",
    "        + \"\\n\"\n",
    "    )\n",
    "intro_time = timecodes[0][1] - timecodes[0][0]\n",
    "to_json = lambda s: [\n",
    "    {\"StartTimecode\": t1, \"EndTimecode\": t2}\n",
    "    for t1, t2 in (line.split(\",\") for line in s.split(\"\\n\") if line.strip())\n",
    "]\n",
    "timecodes = to_json(timecodes_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c838304-5037-4e20-8407-192e6a3a2033",
   "metadata": {},
   "source": [
    "The result is a list of timecodes that defines our final video segments.\n",
    "    \n",
    "Here are the generated timecodes that will be used for AWS Elemental MediaConvert input clipping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d7f8230-d50d-4e5c-9890-ee7dcd0bd2d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "timecodes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decfc5d0-e0ba-4a55-814f-cbfa7d3d1358",
   "metadata": {},
   "source": [
    "Now that we have our timecodes, we can follow the same steps as in the previous section to:\n",
    "1. Create input clips using AWS Elemental MediaConvert\n",
    "2. Insert audio narration and subtitle to create a final short-form video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "020b1312-b378-4aa6-a46a-e3a42853e951",
   "metadata": {},
   "outputs": [],
   "source": [
    "escaped_summarized_text = (\n",
    "        summarized_text.replace(\"&\", \"&amp;\")\n",
    "        .replace('\"', \"&quot;\")\n",
    "        .replace(\"'\", \"&apos;\")\n",
    "        .replace(\"<\", \"&lt;\")\n",
    "        .replace(\">\", \"&gt;\")\n",
    "    )\n",
    "ssml = \"<speak>\\n\"\n",
    "break_time = intro_time\n",
    "\n",
    "while break_time > 10000:  # maximum break time in Polly is 10s\n",
    "    ssml += '<break time = \"' + str(break_time) + 'ms\"/>'\n",
    "    break_time -= 10000\n",
    "ssml += '<break time = \"' + str(break_time) + 'ms\"/>'\n",
    "ssml += escaped_summarized_text\n",
    "ssml += \"</speak>\"\n",
    "\n",
    "response = polly_client.synthesize_speech(\n",
    "    Engine=\"neural\",\n",
    "    OutputFormat=\"mp3\",\n",
    "    Text=ssml,\n",
    "    TextType=\"ssml\",\n",
    "    VoiceId=voice_id,\n",
    ")\n",
    "\n",
    "if \"AudioStream\" in response:\n",
    "    with response[\"AudioStream\"] as stream:\n",
    "        audio_narration = stream.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e548633-5f0b-40d1-b240-8cc8037d3b4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_client.put_object(\n",
    "    Body=audio_narration, Bucket=s3_bucket, Key=audio_narration_filename, ContentType=\"audio/mpeg\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6faae7b-bc4e-457f-863a-7d84be6cd82a",
   "metadata": {},
   "outputs": [],
   "source": [
    "video_summary_subtitle = \"\"\n",
    "start = intro_time\n",
    "for i in range(len(summarized_sentences)):\n",
    "    end = start + speech_durations[i]\n",
    "    video_summary_subtitle += f\"{i+1}\\n\"\n",
    "    video_summary_subtitle += f\"{milliseconds_to_subtitleTimeFormat(start)} --> {milliseconds_to_subtitleTimeFormat(end)}\\n\"\n",
    "    sentence_lines = split_long_lines(summarized_sentences[i], 90)\n",
    "    for line in sentence_lines:\n",
    "        video_summary_subtitle += f\"{line}\\n\"\n",
    "    video_summary_subtitle += \"\\n\"\n",
    "    start = end\n",
    "s3_client.put_object(\n",
    "    Body=video_summary_subtitle, Bucket=s3_bucket, Key=subtitle_filename\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecad5d2e-0bc4-4c5d-ae4d-493c94f4f00a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_single_clip(media_convert, iam_role, input_video_path, output_video_path, timecode, clip_index):\n",
    "    \"\"\"\n",
    "    Create a MediaConvert job to process a single video clip (due to multiple input clipping need be processed in chronological order)\n",
    "    Args:\n",
    "        media_convert - MediaConvert client\n",
    "        iam_role - IAM role ARN for MediaConvert\n",
    "        input_video_path - S3 path for input video\n",
    "        output_video_path - S3 path for output video\n",
    "        timecode - Dictionary containing start and end timecodes for the clip\n",
    "        clip_index - Index number for the clip\n",
    "        \n",
    "    Returns:\n",
    "        - job_id: MediaConvert job Id\n",
    "        - clip_output: S3 path of the output clip\n",
    "    \"\"\"\n",
    "    clip_output = f\"{output_video_path}{video['output_dir']}_{clip_index}\"\n",
    "    \n",
    "    response = media_convert.create_job(\n",
    "        Queue=\"Default\",\n",
    "        UserMetadata={},\n",
    "        Role=iam_role,\n",
    "        Settings={\n",
    "            \"TimecodeConfig\": {\"Source\": \"ZEROBASED\"},\n",
    "            \"OutputGroups\": [\n",
    "                {\n",
    "                    \"Name\": \"File Group\",\n",
    "                    \"Outputs\": [\n",
    "                        {\n",
    "                            \"ContainerSettings\": {\n",
    "                                \"Container\": \"MP4\",\n",
    "                                \"Mp4Settings\": {},\n",
    "                            },\n",
    "                            \"VideoDescription\": {\n",
    "                                \"CodecSettings\": {\n",
    "                                    \"Codec\": \"H_264\",\n",
    "                                    \"H264Settings\": {\n",
    "                                        \"MaxBitrate\": 40000000,\n",
    "                                        \"RateControlMode\": \"QVBR\",\n",
    "                                        \"SceneChangeDetect\": \"TRANSITION_DETECTION\",\n",
    "                                    },\n",
    "                                }\n",
    "                            }\n",
    "                        }\n",
    "                    ],\n",
    "                    \"OutputGroupSettings\": {\n",
    "                        \"Type\": \"FILE_GROUP_SETTINGS\",\n",
    "                        \"FileGroupSettings\": {\"Destination\": clip_output},\n",
    "                    },\n",
    "                }\n",
    "            ],\n",
    "            \"Inputs\": [\n",
    "                {\n",
    "                    \"VideoSelector\": {},\n",
    "                    \"TimecodeSource\": \"ZEROBASED\",\n",
    "                    \"FileInput\": video[\"url\"],\n",
    "                    \"InputClippings\": [timecode],\n",
    "                }\n",
    "            ],\n",
    "        },\n",
    "        AccelerationSettings={\"Mode\": \"DISABLED\"},\n",
    "        StatusUpdateInterval=\"SECONDS_60\",\n",
    "        Priority=0,\n",
    "    )\n",
    "    \n",
    "    return response[\"Job\"][\"Id\"], clip_output\n",
    "\n",
    "def wait_for_job(media_convert, job_id):\n",
    "    \"\"\"\n",
    "    Wait for a MediaConvert job to complete\n",
    "    Args:\n",
    "        media_convert - MediaConvert client\n",
    "        job_id - MediaConvert job Id\n",
    "        \n",
    "    Returns:\n",
    "        bool - True if job completed successfully, False if error occurred\n",
    "    \"\"\"\n",
    "    job_complete = False\n",
    "    while not job_complete:\n",
    "        job_response = media_convert.get_job(Id=job_id)\n",
    "        job_status = job_response['Job']['Status']\n",
    "        print(f\"MediaConvert job status: {job_status}\")\n",
    "        \n",
    "        if job_status in ['COMPLETE', 'ERROR']:\n",
    "            return job_status == 'COMPLETE'\n",
    "        time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95ba5225-ab9b-4513-be26-bb0ada9e6c10",
   "metadata": {},
   "outputs": [],
   "source": [
    "clip_paths = []\n",
    "    \n",
    "for i, timecode in enumerate(timecodes):\n",
    "    print(f\"Processing clip {i+1}/{len(timecodes)}\")\n",
    "    print(timecode)\n",
    "    \n",
    "    job_id, clip_output = process_single_clip(\n",
    "        media_convert, \n",
    "        iam_role, \n",
    "        input_video_path, \n",
    "        output_video_path, \n",
    "        timecode, \n",
    "        i\n",
    "    )\n",
    "\n",
    "    if wait_for_job(media_convert, job_id):\n",
    "        clip_paths.append(clip_output)\n",
    "    else:\n",
    "        print(f\"Failed to process clip {i+1}\")\n",
    "        continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71e7ccf7-5d67-4324-8a59-65f40c9ca0c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_clips(media_convert, iam_role, clip_paths, output_video_path):\n",
    "    \"\"\"\n",
    "    Merge multiple video clips into a single video\n",
    "    Args:\n",
    "        media_convert - MediaConvert client\n",
    "        iam_role - IAM role ARN for MediaConvert\n",
    "        clip_paths - List of S3 paths of video clips to merge\n",
    "        output_video_path - S3 path for video output\n",
    "        \n",
    "    Returns:\n",
    "        - job_id: MediaConvert job Id\n",
    "        - merged_output: S3 path of the video output\n",
    "    \"\"\"\n",
    "    inputs = []\n",
    "    \n",
    "    for clip_path in clip_paths:\n",
    "        inputs.append({\n",
    "            \"VideoSelector\": {},\n",
    "            \"TimecodeSource\": \"ZEROBASED\",\n",
    "            \"FileInput\": clip_path + \".mp4\",\n",
    "        })\n",
    "\n",
    "    merged_output = f\"{output_video_path}{video['output_dir']}\"\n",
    "    \n",
    "    response = media_convert.create_job(\n",
    "        Queue=\"Default\",\n",
    "        UserMetadata={},\n",
    "        Role=iam_role,\n",
    "        Settings={\n",
    "            \"TimecodeConfig\": {\"Source\": \"ZEROBASED\"},\n",
    "            \"OutputGroups\": [\n",
    "                {\n",
    "                    \"Name\": \"File Group\",\n",
    "                    \"Outputs\": [\n",
    "                        {\n",
    "                            \"ContainerSettings\": {\n",
    "                                \"Container\": \"MP4\",\n",
    "                                \"Mp4Settings\": {},\n",
    "                            },\n",
    "                            \"VideoDescription\": {\n",
    "                                \"CodecSettings\": {\n",
    "                                    \"Codec\": \"H_264\",\n",
    "                                    \"H264Settings\": {\n",
    "                                        \"MaxBitrate\": 40000000,\n",
    "                                        \"RateControlMode\": \"QVBR\",\n",
    "                                        \"SceneChangeDetect\": \"TRANSITION_DETECTION\",\n",
    "                                    },\n",
    "                                }\n",
    "                            }\n",
    "                        }\n",
    "                    ],\n",
    "                    \"OutputGroupSettings\": {\n",
    "                        \"Type\": \"FILE_GROUP_SETTINGS\",\n",
    "                        \"FileGroupSettings\": {\"Destination\": merged_output},\n",
    "                    },\n",
    "                }\n",
    "            ],\n",
    "            \"Inputs\": inputs,\n",
    "        },\n",
    "        AccelerationSettings={\"Mode\": \"DISABLED\"},\n",
    "        StatusUpdateInterval=\"SECONDS_60\",\n",
    "        Priority=0,\n",
    "    )\n",
    "    \n",
    "    return response[\"Job\"][\"Id\"], merged_output\n",
    "\n",
    "def add_audio_subtitles(media_convert, iam_role, input_video_path, audio_file_path, subtitle_file_path, final_output_path):\n",
    "    \"\"\"\n",
    "    Add audio narration and subtitle into the video\n",
    "    Args:\n",
    "        media_convert - MediaConvert client\n",
    "        iam_role - IAM role ARN for MediaConvert\n",
    "        input_video_path - S3 path for input video\n",
    "        audio_file_path - S3 path for audio narration file\n",
    "        subtitle_file_path - S3 path for SRT subtitle file\n",
    "        final_output_path - S3 path for video output\n",
    "        \n",
    "    Returns:\n",
    "        job_id: MediaConvert job Id\n",
    "    \"\"\"\n",
    "    \n",
    "    response = media_convert.create_job(\n",
    "        Queue=\"Default\",\n",
    "        UserMetadata={},\n",
    "        Role=iam_role,\n",
    "        Settings={\n",
    "            \"TimecodeConfig\": {\"Source\": \"ZEROBASED\"},\n",
    "            \"OutputGroups\": [\n",
    "                {\n",
    "                    \"Name\": \"File Group\",\n",
    "                    \"Outputs\": [\n",
    "                        {\n",
    "                            \"ContainerSettings\": {\n",
    "                                \"Container\": \"MP4\",\n",
    "                                \"Mp4Settings\": {},\n",
    "                            },\n",
    "                            \"VideoDescription\": {\n",
    "                                \"CodecSettings\": {\n",
    "                                    \"Codec\": \"H_264\",\n",
    "                                    \"H264Settings\": {\n",
    "                                        \"MaxBitrate\": 40000000,\n",
    "                                        \"RateControlMode\": \"QVBR\",\n",
    "                                        \"SceneChangeDetect\": \"TRANSITION_DETECTION\",\n",
    "                                    },\n",
    "                                }\n",
    "                            },\n",
    "                            \"NameModifier\": \"_summary_v2\",\n",
    "                            \"AudioDescriptions\": [\n",
    "                                {\n",
    "                                    \"AudioSourceName\": \"Audio Selector Group 1\",\n",
    "                                    \"CodecSettings\": {\n",
    "                                        \"Codec\": \"AAC\",\n",
    "                                        \"AacSettings\": {\n",
    "                                            \"Bitrate\": 96000,\n",
    "                                            \"CodingMode\": \"CODING_MODE_2_0\",\n",
    "                                            \"SampleRate\": 48000,\n",
    "                                        },\n",
    "                                    },\n",
    "                                }\n",
    "                            ],\n",
    "                            \"CaptionDescriptions\": [\n",
    "                                {\n",
    "                                    \"CaptionSelectorName\": \"Captions Selector 1\",\n",
    "                                    \"DestinationSettings\": {\n",
    "                                        \"DestinationType\": \"BURN_IN\",\n",
    "                                        \"BurninDestinationSettings\": {\n",
    "                                            \"BackgroundOpacity\": 100,\n",
    "                                            \"FontSize\": 18,\n",
    "                                            \"FontColor\": \"WHITE\",\n",
    "                                            \"ApplyFontColor\": \"ALL_TEXT\",\n",
    "                                            \"BackgroundColor\": \"BLACK\",\n",
    "                                        },\n",
    "                                    },\n",
    "                                }\n",
    "                            ],\n",
    "                        }\n",
    "                    ],\n",
    "                    \"OutputGroupSettings\": {\n",
    "                        \"Type\": \"FILE_GROUP_SETTINGS\",\n",
    "                        \"FileGroupSettings\": {\"Destination\": final_output_path},\n",
    "                    },\n",
    "                }\n",
    "            ],\n",
    "            \"Inputs\": [\n",
    "                {\n",
    "                    \"VideoSelector\": {},\n",
    "                    \"TimecodeSource\": \"ZEROBASED\",\n",
    "                    \"FileInput\": input_video_path,\n",
    "                    \"AudioSelectors\": {\n",
    "                        \"Audio Selector 1\": {\n",
    "                            \"DefaultSelection\": \"NOT_DEFAULT\",\n",
    "                            \"ExternalAudioFileInput\": audio_file_path,\n",
    "                        },\n",
    "                    },\n",
    "                    \"AudioSelectorGroups\": {\n",
    "                        \"Audio Selector Group 1\": {\n",
    "                            \"AudioSelectorNames\": [\"Audio Selector 1\"]\n",
    "                        }\n",
    "                    },\n",
    "                    \"CaptionSelectors\": {\n",
    "                        \"Captions Selector 1\": {\n",
    "                            \"SourceSettings\": {\n",
    "                                \"SourceType\": \"SRT\",\n",
    "                                \"FileSourceSettings\": {\"SourceFile\": subtitle_file_path},\n",
    "                            }\n",
    "                        }\n",
    "                    },\n",
    "                }\n",
    "            ],\n",
    "        },\n",
    "        AccelerationSettings={\"Mode\": \"DISABLED\"},\n",
    "        StatusUpdateInterval=\"SECONDS_60\",\n",
    "        Priority=0,\n",
    "    )\n",
    "    \n",
    "    return response[\"Job\"][\"Id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9ed1493-a464-4027-b330-b9cfb614feb0",
   "metadata": {},
   "source": [
    "Merge all clips with audio and subtitles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fa1ae31-cff6-4432-944a-12bea0e42a1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "merge_job_id, merged_output = merge_clips(\n",
    "    media_convert, \n",
    "    iam_role, \n",
    "    clip_paths, \n",
    "    output_video_path\n",
    ")\n",
    "\n",
    "if wait_for_job(media_convert, merge_job_id):\n",
    "    print(\"Successfully merged video clips\")\n",
    "else:\n",
    "    print(\"Failed to merge video clips\")\n",
    "\n",
    "final_job_id = add_audio_subtitles(\n",
    "    media_convert,\n",
    "    iam_role,\n",
    "    merged_output + \".mp4\",\n",
    "    audio_file_path,\n",
    "    subtitle_file_path,\n",
    "    output_video_path\n",
    ")\n",
    "\n",
    "if wait_for_job(media_convert, final_job_id):\n",
    "    print(\"Successfully created final video with audio and subtitle\")\n",
    "else:\n",
    "    print(\"Failed to add audio and subtitle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a26fda34-e421-4349-84cd-b1352407d2a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "video_summary_v2 = os.path.splitext(os.path.basename(video['path']))[0] + \"_summary_v2.mp4\"\n",
    "s3_client.download_file(s3_bucket, video_summary_v2, video_summary_v2)\n",
    "print(\"Short-form video with video and audio understanding\\n\")\n",
    "print(\"=========================================\\n\")\n",
    "display(Video(url=video_summary_v2, width=640, height=360, html_attributes=\"controls muted autoplay\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43d6dc22-8908-4e26-ab86-390debed5539",
   "metadata": {},
   "source": [
    "Here is our final video summary incorporating both visual and audio understanding. Compare it with our earlier version (based only on audio narration) to see the differences in the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab9602a9-3ff7-430e-8cb6-938ffe3a1f08",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Short-form video with audio understanding\\n\")\n",
    "print(\"=========================================\\n\")\n",
    "display(Video(url=video_summary, width=640, height=360, html_attributes=\"controls muted autoplay\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be69f6e3-d26e-4aac-b09f-cecaee7c9b99",
   "metadata": {},
   "source": [
    "## Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2189d57-fe0e-4ce5-85b6-95ee9a0a82f7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# s3_client.delete_object(Bucket=s3_bucket, Key=audio_narration_filename)\n",
    "# s3_client.delete_object(Bucket=s3_bucket, Key=video['path'])\n",
    "# s3_client.delete_object(Bucket=s3_bucket, Key=subtitle_filename)\n",
    "# s3_client.delete_object(Bucket=s3_bucket, Key=video_summary)\n",
    "# s3_client.delete_object(Bucket=s3_bucket, Key=video_summary_v2)\n",
    "# aoss_client.indices.delete(aoss_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e43a50b-8b30-4d54-bf01-e4c6bd98c332",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd1c3914-a7a4-4cf5-b968-f520c8cc9262",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}