OpenAI_Whisper.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iS7W1Pi7Q-9P"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NahuelCostaCortez/AI_notebooks/blob/main/OpenAI_Whisper.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5hvo8QWN-a9"
      },
      "source": [
        "🗣️ **Whisper**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V84aU8RHQ-9R"
      },
      "source": [
        "[Whisper](https://github.com/openai/whisper) is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-BcD0b8hwdA"
      },
      "source": [
        "🚨 Make sure you are using a GPU. Go to `\"Runtime\" > \"Change runtime type\"` and in `\"Hardware accelerator\"` select `\"GPU\"`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Js9B_oAvQ-9S"
      },
      "source": [
        "## Install necessary libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SJl7HJOeo0-P",
        "outputId": "60b3f988-ea0b-4b65-9a73-35d160f77a8e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/openai/whisper.git\n",
            "  Cloning https://github.com/openai/whisper.git to /tmp/pip-req-build-8svp21xn\n",
            "  Running command git clone -q https://github.com/openai/whisper.git /tmp/pip-req-build-8svp21xn\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from whisper==1.0) (1.21.6)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from whisper==1.0) (1.12.1+cu113)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from whisper==1.0) (4.64.1)\n",
            "Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/dist-packages (from whisper==1.0) (9.0.0)\n",
            "Collecting transformers>=4.19.0\n",
            "  Downloading transformers-4.24.0-py3-none-any.whl (5.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 5.5 MB 16.8 MB/s \n",
            "\u001b[?25hCollecting ffmpeg-python==0.2.0\n",
            "  Downloading ffmpeg_python-0.2.0-py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from ffmpeg-python==0.2.0->whisper==1.0) (0.16.0)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers>=4.19.0->whisper==1.0) (4.13.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.19.0->whisper==1.0) (6.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers>=4.19.0->whisper==1.0) (3.8.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.19.0->whisper==1.0) (2022.6.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.19.0->whisper==1.0) (21.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers>=4.19.0->whisper==1.0) (2.23.0)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
            "  Downloading tokenizers-0.13.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 7.6 MB 44.6 MB/s \n",
            "\u001b[?25hCollecting huggingface-hub<1.0,>=0.10.0\n",
            "  Downloading huggingface_hub-0.11.0-py3-none-any.whl (182 kB)\n",
            "\u001b[K     |████████████████████████████████| 182 kB 65.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers>=4.19.0->whisper==1.0) (4.1.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers>=4.19.0->whisper==1.0) (3.0.9)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers>=4.19.0->whisper==1.0) (3.10.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.19.0->whisper==1.0) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.19.0->whisper==1.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.19.0->whisper==1.0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.19.0->whisper==1.0) (2022.9.24)\n",
            "Building wheels for collected packages: whisper\n",
            "  Building wheel for whisper (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for whisper: filename=whisper-1.0-py3-none-any.whl size=1175239 sha256=e5310d9fd2852e3111ed5c11159a260f7d0c967e551a1e96a541f00ef740b295\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-wpg9k1my/wheels/16/15/89/1c7bb31bd0006793a95549d04785121a8a36daad9158e1e43a\n",
            "Successfully built whisper\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers, ffmpeg-python, whisper\n",
            "Successfully installed ffmpeg-python-0.2.0 huggingface-hub-0.11.0 tokenizers-0.13.2 transformers-4.24.0 whisper-1.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting jiwer\n",
            "  Downloading jiwer-2.5.1-py3-none-any.whl (15 kB)\n",
            "Collecting levenshtein==0.20.2\n",
            "  Downloading Levenshtein-0.20.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.4 MB 34.4 MB/s \n",
            "\u001b[?25hCollecting rapidfuzz<3.0.0,>=2.3.0\n",
            "  Downloading rapidfuzz-2.13.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.2 MB 50.9 MB/s \n",
            "\u001b[?25hInstalling collected packages: rapidfuzz, levenshtein, jiwer\n",
            "Successfully installed jiwer-2.5.1 levenshtein-0.20.2 rapidfuzz-2.13.2\n"
          ]
        }
      ],
      "source": [
        "#@title ⚙️ Execute this cell to install the necessary libraries.\n",
        "\"\"\"\n",
        "To write this piece of code I took inspiration/code from a lot of places.\n",
        "It was late night, so I'm not sure how much I created or just copied o.O\n",
        "Here are some of the possible references:\n",
        "https://blog.addpipe.com/recording-audio-in-the-browser-using-pure-html5-and-minimal-javascript/\n",
        "https://stackoverflow.com/a/18650249\n",
        "https://hacks.mozilla.org/2014/06/easy-audio-capture-with-the-mediarecorder-api/\n",
        "https://air.ghost.io/recording-to-an-audio-file-using-html5-and-js/\n",
        "https://stackoverflow.com/a/49019356\n",
        "\"\"\"\n",
        "\n",
        "!pip install git+https://github.com/openai/whisper.git\n",
        "!pip install jiwer\n",
        "\n",
        "from IPython.display import HTML, Audio\n",
        "from google.colab.output import eval_js\n",
        "from base64 import b64decode\n",
        "import numpy as np\n",
        "from scipy.io.wavfile import read as wav_read\n",
        "import io\n",
        "import ffmpeg\n",
        "\n",
        "AUDIO_HTML = \"\"\"\n",
        "<script>\n",
        "var my_div = document.createElement(\"DIV\");\n",
        "var my_p = document.createElement(\"P\");\n",
        "var my_btn = document.createElement(\"BUTTON\");\n",
        "var t = document.createTextNode(\"Press to start recording\");\n",
        "\n",
        "my_btn.appendChild(t);\n",
        "//my_p.appendChild(my_btn);\n",
        "my_div.appendChild(my_btn);\n",
        "document.body.appendChild(my_div);\n",
        "\n",
        "var base64data = 0;\n",
        "var reader;\n",
        "var recorder, gumStream;\n",
        "var recordButton = my_btn;\n",
        "\n",
        "var handleSuccess = function(stream) {\n",
        "  gumStream = stream;\n",
        "  var options = {\n",
        "    //bitsPerSecond: 8000, //chrome seems to ignore, always 48k\n",
        "    mimeType : 'audio/webm;codecs=opus'\n",
        "    //mimeType : 'audio/webm;codecs=pcm'\n",
        "  };            \n",
        "  //recorder = new MediaRecorder(stream, options);\n",
        "  recorder = new MediaRecorder(stream);\n",
        "  recorder.ondataavailable = function(e) {            \n",
        "    var url = URL.createObjectURL(e.data);\n",
        "    var preview = document.createElement('audio');\n",
        "    preview.controls = true;\n",
        "    preview.src = url;\n",
        "    document.body.appendChild(preview);\n",
        "\n",
        "    reader = new FileReader();\n",
        "    reader.readAsDataURL(e.data); \n",
        "    reader.onloadend = function() {\n",
        "      base64data = reader.result;\n",
        "      //console.log(\"Inside FileReader:\" + base64data);\n",
        "    }\n",
        "  };\n",
        "  recorder.start();\n",
        "  };\n",
        "\n",
        "recordButton.innerText = \"Recording... press to stop\";\n",
        "\n",
        "navigator.mediaDevices.getUserMedia({audio: true}).then(handleSuccess);\n",
        "\n",
        "\n",
        "function toggleRecording() {\n",
        "  if (recorder && recorder.state == \"recording\") {\n",
        "      recorder.stop();\n",
        "      gumStream.getAudioTracks()[0].stop();\n",
        "      recordButton.innerText = \"Saving the recording... pls wait!\"\n",
        "  }\n",
        "}\n",
        "\n",
        "// https://stackoverflow.com/a/951057\n",
        "function sleep(ms) {\n",
        "  return new Promise(resolve => setTimeout(resolve, ms));\n",
        "}\n",
        "\n",
        "var data = new Promise(resolve=>{\n",
        "//recordButton.addEventListener(\"click\", toggleRecording);\n",
        "recordButton.onclick = ()=>{\n",
        "toggleRecording()\n",
        "\n",
        "sleep(2000).then(() => {\n",
        "  // wait 2000ms for the data to be available...\n",
        "  // ideally this should use something like await...\n",
        "  //console.log(\"Inside data:\" + base64data)\n",
        "  resolve(base64data.toString())\n",
        "\n",
        "});\n",
        "\n",
        "}\n",
        "});\n",
        "      \n",
        "</script>\n",
        "\"\"\"\n",
        "\n",
        "def get_audio():\n",
        "  display(HTML(AUDIO_HTML))\n",
        "  data = eval_js(\"data\")\n",
        "  binary = b64decode(data.split(',')[1])\n",
        "  \n",
        "  process = (ffmpeg\n",
        "    .input('pipe:0')\n",
        "    .output('pipe:1', format='wav')\n",
        "    .run_async(pipe_stdin=True, pipe_stdout=True, pipe_stderr=True, quiet=True, overwrite_output=True)\n",
        "  )\n",
        "  output, err = process.communicate(input=binary)\n",
        "  \n",
        "  riff_chunk_size = len(output) - 8\n",
        "  # Break up the chunk size into four bytes, held in b.\n",
        "  q = riff_chunk_size\n",
        "  b = []\n",
        "  for i in range(4):\n",
        "      q, r = divmod(q, 256)\n",
        "      b.append(r)\n",
        "\n",
        "  # Replace bytes 4:8 in proc.stdout with the actual size of the RIFF chunk.\n",
        "  riff = output[:4] + bytes(b) + output[8:]\n",
        "\n",
        "  sr, audio = wav_read(io.BytesIO(riff))\n",
        "\n",
        "  return audio, sr"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-9_I0W3tqTjr"
      },
      "source": [
        "## 🎙️ **Record and transcribe** [Choose the task]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "opNkn_Lgpat4",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "Task = \"Translate to English\" #@param [\"Transcript to Language\", \"Translate to English\"]\n",
        "import numpy as np\n",
        "import whisper\n",
        "from scipy.io.wavfile import write\n",
        "from IPython.display import clear_output\n",
        "\n",
        "sel_task = \"translate\" if Task == \"Translate to English\" else \"transcribe\" \n",
        "\n",
        "audio, sr = get_audio()\n",
        "write('record.wav', sr, audio)\n",
        "\n",
        "!whisper \"record.wav\" --task {sel_task} --model medium --verbose False\n",
        "\n",
        "clear_output()\n",
        "if sel_task == \"translate\":\n",
        "  print(\"-- TRADUCCIÓN A INGLÉS --\\n\")\n",
        "else:\n",
        "  print(\"-- TRANSCRIPCIÓN A ESPAÑOL --\\n\")\n",
        "  \n",
        "print(open('record.wav.txt').read())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXZooJ-Wy9EL"
      },
      "source": [
        "## 👉  Only **transcribe** audio to the original language. (X ► X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PEkqEwI4Q-9U"
      },
      "source": [
        "Just upload the audio you want to work with to the \"Files\" tab on the left. Then, run the cell below to transcribe the audio."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Aejy9Haxy81U",
        "outputId": "ea472b90-8ce4-48a5-ff03-7e4d37b71072"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tcmalloc: large alloc 1528012800 bytes == 0xa0b6000 @  0x7fc6c24061e7 0x4b2590 0x5ad01c 0x5dcfef 0x58f92b 0x590c33 0x5e48ac 0x4d20fa 0x51041f 0x58fd37 0x50c4fc 0x5b4ee6 0x58ff2e 0x50d482 0x58fd37 0x50c4fc 0x5b4ee6 0x6005a3 0x607796 0x60785c 0x60a436 0x64db82 0x64dd2e 0x7fc6c2003c87 0x5b636a\n",
            "Detected language: Spanish\n",
            "100% 338796/338796 [13:50<00:00, 407.85frames/s]\n"
          ]
        }
      ],
      "source": [
        "!whisper \"/content/audio.mp3\" --task transcribe --model medium --verbose False --output_dir audio_transcription"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wfQIccnbcO1v"
      },
      "source": [
        "It can also be used in code, instead of from the terminal:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "import whisper\n",
        "\n",
        "model = whisper.load_model(\"medium\")\n",
        "result = model.transcribe(\"audio.mp3\")\n",
        "print(result[\"text\"])\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOdSeKWnzJSV"
      },
      "source": [
        "## 👉  Only **translate** audio from the original language to English. (X ► English)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U2oxL7wHQ-9V"
      },
      "source": [
        "Just upload the audio you want to work with to the \"Files\" tab on the left. Then, run the cell below to transcribe the audio."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UlVRSbPC7IRJ"
      },
      "outputs": [],
      "source": [
        "!whisper \"/content/audio.mp3\" --task translate --model medium"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "Js9B_oAvQ-9S",
        "-9_I0W3tqTjr",
        "XXZooJ-Wy9EL",
        "wOdSeKWnzJSV"
      ]
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.7.11 ('base')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.11"
    },
    "vscode": {
      "interpreter": {
        "hash": "f076537bb276b42a17a68ee1ac7ae5d287449b2c7612a80803578e6d23289952"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}