speech/use-cases/storytelling/storytelling.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Narrate a Multi-character Story with Text-to-Speech and Gemini\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/speech/use-cases/storytelling/storytelling.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fspeech%2Fuse-cases%2Fstorytelling%2Fstorytelling.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/speech/use-cases/storytelling/storytelling.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/speech/use-cases/storytelling/storytelling.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "---\n",
        "\n",
        "* Author: Holt Skinner\n",
        "* Created: Jan 2024\n",
        "\n",
        "---\n",
        "\n",
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to use the [Text-to-Speech API](https://cloud.google.com/text-to-speech) to read a story with each character having a distinct voice."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "This tutorial uses the following Google Cloud AI services and resources:\n",
        "\n",
        "- [Cloud Text-to-Speech API](https://cloud.google.com/text-to-speech/docs)\n",
        "- Cloud Storage\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Parse the input story text in play script format. (`Character: Lines`)\n",
        "- Assign each character to a voice.\n",
        "- Synthesize each line based on character voice.\n",
        "- Combine audio files into one MP3 file.\n",
        "\n",
        "Planned expansions:\n",
        "\n",
        "- Upload audio to Cloud Storage\n",
        "- Read in story text using [Document AI OCR](https://cloud.google.com/document-ai)\n",
        "- Convert story to play script format using Gemini.\n",
        "  - Possibility: Use Gemini to generate [SSML](https://cloud.google.com/text-to-speech/docs/ssml) directly from book text.\n",
        "- Create alternative implementation using LangChain.\n",
        "- Add [Journey voices](https://cloud.google.com/text-to-speech/docs/voice-types#journey_voices) once more voices are supported.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Text-to-Speech\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Text-to-Speech pricing](https://cloud.google.com/text-to-speech/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Getting Started\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NNSWiCNPjh_p"
      },
      "source": [
        "### Install Vertex AI SDK, other packages and their dependencies\n",
        "\n",
        "Install the following packages required to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "# Install the packages\n",
        "%pip install --user --upgrade -q google-cloud-aiplatform google-cloud-texttospeech pydub pandas tqdm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J3a6juReciKR"
      },
      "source": [
        "If you're running on a Mac, you will need to install [FFmpeg](https://ffmpeg.org/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BzCUF4oqciKR"
      },
      "outputs": [],
      "source": [
        "import platform\n",
        "\n",
        "# Check if the system is macOS\n",
        "if platform.system() == \"Darwin\":\n",
        "    # Install using Homebrew\n",
        "    !brew install ffmpeg"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gbWwuHK8j1xm"
      },
      "source": [
        "### Run the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GTdx_NRxkD5M"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YbMFqPZ3tnwz"
      },
      "source": [
        "Set the project and region.\n",
        "\n",
        "* Please note the **available regions** for Text-to-Speech, see [documentation](https://cloud.google.com/text-to-speech/docs/endpoints)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GjSsu6cmUdEx"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"YOUR_PROJECT_ID\"  # @param {type:\"string\"}\n",
        "\n",
        "TTS_LOCATION = \"us\"  # @param {type:\"string\"}\n",
        "VERTEXAI_LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opUxT_k5TdgP"
      },
      "source": [
        "### Authenticating your notebook environment\n",
        "\n",
        "* If you are using **Colab** to run this notebook, run the cell below and continue.\n",
        "* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vbNgv4q1T2Mi"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Additional authentication is required for Google Colab\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()\n",
        "\n",
        "    ! gcloud config set project {PROJECT_ID}\n",
        "    ! gcloud auth application-default login -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c93f8e645f6e"
      },
      "source": [
        "Initialize the [Vertex AI SDK](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "baf18b29f8a8"
      },
      "outputs": [],
      "source": [
        "import vertexai\n",
        "\n",
        "# Initialize Vertex AI\n",
        "vertexai.init(project=PROJECT_ID, location=VERTEXAI_LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Download source texts from Google Cloud Storage\n",
        "\n",
        "This public bucket contains some stories generated by PaLM."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil cp gs://github-repo/speech/storytelling/*.txt ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import os\n",
        "from pathlib import Path\n",
        "\n",
        "from IPython.display import Audio\n",
        "from google.api_core.client_options import ClientOptions\n",
        "from google.cloud import texttospeech_v1beta1 as texttospeech\n",
        "import pandas as pd\n",
        "from pydub import AudioSegment\n",
        "from tqdm import tqdm\n",
        "from vertexai.preview.generative_models import GenerationConfig, GenerativeModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v4gUI8WqciKS"
      },
      "source": [
        "### Define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Pl3un_YciKS"
      },
      "outputs": [],
      "source": [
        "DEFAULT_LANGUAGE = \"en\"\n",
        "# Voice used for narration, scene details, etc.\n",
        "DEFAULT_VOICE = \"en-GB-Neural2-B\"\n",
        "\n",
        "tts_client = texttospeech.TextToSpeechClient(\n",
        "    client_options=ClientOptions(\n",
        "        api_endpoint=f\"{TTS_LOCATION}-texttospeech.googleapis.com\"\n",
        "    )\n",
        ")\n",
        "model = GenerativeModel(\"gemini-pro\")\n",
        "\n",
        "SILENCE_LENGTH = 200  # In Milliseconds\n",
        "TXT_EXTENSION = \".txt\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5CEc4-Wrjk2"
      },
      "source": [
        "### Helper functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kYx2wwhjrmD6"
      },
      "outputs": [],
      "source": [
        "def list_voices(\n",
        "    language_code: str = DEFAULT_LANGUAGE, voice_type: str = \"Neural2\"\n",
        ") -> list[dict]:\n",
        "    response = tts_client.list_voices(language_code=language_code)\n",
        "\n",
        "    return [\n",
        "        {\n",
        "            \"name\": voice.name,\n",
        "            \"gender\": texttospeech.SsmlVoiceGender(voice.ssml_gender).name.lower(),\n",
        "        }\n",
        "        for voice in response.voices\n",
        "        if voice_type in voice.name and voice.name != DEFAULT_VOICE\n",
        "    ]\n",
        "\n",
        "\n",
        "def create_character_map(characters: list[str], voices: list[str]) -> dict[str, str]:\n",
        "    responses = model.generate_content(\n",
        "        f\"\"\"Your job is to uniquely and appropriately match character names to voices available with Google Cloud Text to Speech.\n",
        "\n",
        "The following is a list of available voices for Google Cloud Text to Speech in a JSON list.\n",
        "\n",
        "{voices}\n",
        "\n",
        "The following is a list of character names in a JSON list:\n",
        "\n",
        "{characters}\n",
        "\n",
        "Output a JSON formatted object mapping Character Names to Voice Names:\n",
        "\"\"\",\n",
        "        generation_config=GenerationConfig(\n",
        "            max_output_tokens=2048, temperature=0.9, top_p=1\n",
        "        ),\n",
        "        safety_settings=[],\n",
        "        stream=True,\n",
        "    )\n",
        "\n",
        "    for response in responses:\n",
        "        json_string = response.text.replace(\"`\", \"\").replace(\"json\", \"\")\n",
        "        return json.loads(json_string)\n",
        "\n",
        "\n",
        "def synthesize_text(\n",
        "    text: str, output: str, voice_name: str, language_code: str = DEFAULT_LANGUAGE\n",
        "):\n",
        "    response = tts_client.synthesize_speech(\n",
        "        input=texttospeech.SynthesisInput(text=text),\n",
        "        voice=texttospeech.VoiceSelectionParams(\n",
        "            language_code=language_code,\n",
        "            name=voice_name,\n",
        "        ),\n",
        "        audio_config=texttospeech.AudioConfig(\n",
        "            audio_encoding=texttospeech.AudioEncoding.MP3\n",
        "        ),\n",
        "    )\n",
        "\n",
        "    # The response's audio_content is binary.\n",
        "    with open(output, \"wb\") as f:\n",
        "        f.write(response.audio_content)\n",
        "\n",
        "\n",
        "def combine_audio_files(audio_files: list[str], filename: str) -> str:\n",
        "    full_audio = AudioSegment.silent(duration=SILENCE_LENGTH)\n",
        "\n",
        "    for file in audio_files:\n",
        "        sound = AudioSegment.from_mp3(file)\n",
        "        silence = AudioSegment.silent(duration=SILENCE_LENGTH)\n",
        "        full_audio += sound + silence\n",
        "        os.remove(file)\n",
        "\n",
        "    outfile_name = f\"{Path(filename).stem}-complete.mp3\"\n",
        "    full_audio.export(outfile_name, format=\"mp3\")\n",
        "    return outfile_name\n",
        "\n",
        "\n",
        "def get_characters(input_file: str) -> list[str]:\n",
        "    character_list = []\n",
        "    with open(input_file) as f:\n",
        "        lines = f.readlines()\n",
        "\n",
        "    start_line = lines.index(\"Characters:\\n\")\n",
        "\n",
        "    for i in range(start_line + 2, len(lines)):\n",
        "        if lines[i] == \"\\n\":\n",
        "            break\n",
        "        character_list.append(lines[i].strip())\n",
        "    return character_list\n",
        "\n",
        "\n",
        "def parse_file(\n",
        "    input_file: str, character_to_voice: dict[str, tuple[str, str]]\n",
        ") -> list[str]:\n",
        "    with open(input_file) as f:\n",
        "        lines = f.readlines()\n",
        "\n",
        "    line_number = 1\n",
        "    output_files = []\n",
        "    filename = Path(input_file).stem\n",
        "\n",
        "    for line in tqdm(lines, \"Parsing input file\"):\n",
        "        split_line = line.strip().split(\": \", 1)\n",
        "\n",
        "        character = split_line[0]\n",
        "        if not character:  # Skip blank lines\n",
        "            continue\n",
        "\n",
        "        voice = character_to_voice.get(character, DEFAULT_VOICE)\n",
        "\n",
        "        if len(split_line) <= 1:\n",
        "            dialogue = split_line[0]\n",
        "        elif \"Scene\" in split_line[0]:\n",
        "            dialogue = f\"{split_line[0]}, {split_line[1]}\"\n",
        "        else:\n",
        "            dialogue = split_line[1]\n",
        "\n",
        "        output_file = f\"{filename}-{line_number}.mp3\"\n",
        "        output_files.append(output_file)\n",
        "        synthesize_text(dialogue, output_file, voice[0], voice[1])\n",
        "        line_number += 1\n",
        "\n",
        "    return output_files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5eaHokbL2nS9"
      },
      "source": [
        "## Call the Text-to-Speech API with script content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F4NR_30rfuhA"
      },
      "source": [
        "### Get available voices"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0KqGXuRVuBDf"
      },
      "outputs": [],
      "source": [
        "all_voices = list_voices()\n",
        "print(pd.DataFrame(all_voices))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LKMhnJ6mf6bb"
      },
      "source": [
        "### List all characters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j1DahJu-f5L3"
      },
      "outputs": [],
      "source": [
        "input_file = \"Macbeth.txt\"  # @param {type:\"string\"}\n",
        "\n",
        "character_list = get_characters(input_file)\n",
        "\n",
        "if len(character_list) > len(all_voices):\n",
        "    print(f\"Too many characters {len(character_list)}. Max {len(all_voices)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y081kurHgGoK"
      },
      "source": [
        "### Map Characters to Voices"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "otMX_-lfgIG1"
      },
      "outputs": [],
      "source": [
        "character_to_voice = create_character_map(character_list, all_voices)\n",
        "\n",
        "print(pd.DataFrame(character_to_voice))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p-laS59bgKo5"
      },
      "source": [
        "### Parse input text and output each line as audio"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L6rYY43xgLwh"
      },
      "outputs": [],
      "source": [
        "output_files = parse_file(input_file, character_to_voice)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tAMk2RbgO-C"
      },
      "source": [
        "### Combine audio files into a single file\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xl0ZnHStgRg3"
      },
      "outputs": [],
      "source": [
        "outfile_name = combine_audio_files(output_files, input_file)\n",
        "print(f\"Audio content written to file {outfile_name}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ce553e1efe43"
      },
      "source": [
        "### Listen to the audio"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d_ax6LxzciKT"
      },
      "outputs": [],
      "source": [
        "Audio(outfile_name)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "storytelling.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}