main.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Kaldi_OpenDevLibrary.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tYqEk4EZgURz",
        "colab_type": "text"
      },
      "source": [
        "# **Speech Censor Bot**\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "A simple bot for censoring audio-visual clips. The notebook is inspired by the OpenVINO v2020.1 docs for [Offline Speech Recognition Demo](https://docs.openvinotoolkit.org/latest/_inference_engine_samples_speech_libs_and_demos_Offline_speech_recognition_demo.html) and [another](https://docs.openvinotoolkit.org/latest/_inference_engine_samples_speech_libs_and_demos_Speech_libs_and_demos.html) documentation on inference engine samples. \n",
        "The first link explains how the custom *KALDI* models can be used in the application. For demonstration purpose, we use the pre-trained model provided in the OpenVINO package itself.\n",
        "\n",
        "Users can test/use the notebook to run custom files by editing the **TODO** sections in the code cells. \n",
        "\n",
        "**Note: The version for OpenVINO matters as it has been observed that different versions have slighly different information**\n",
        "\n",
        "To understand the KALDI files, see [this](https://stackoverflow.com/questions/54428601/kaldis-objects-explained-in-laymans-term)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QY2NYaGtBDFm",
        "colab_type": "text"
      },
      "source": [
        "#**Section 1 : For Audio + Video**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PNNs504vyZoD",
        "colab_type": "text"
      },
      "source": [
        "##**Install packages and dependencies**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t_eL16xQgdKx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#For audio preprocessing and audio-video manipulation\n",
        "import wave\n",
        "!apt-get install libsox-fmt-all libsox-dev sox\n",
        "!pip install ffmpeg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QIdgwxSYHxE",
        "colab_type": "text"
      },
      "source": [
        "##**Install OpenVINO toolkit and dependencies**\n",
        "We used a portable version of OpenVINO toolkit for LINUX-based systems, maintained and open-sourced by my friend [Muhammad Ali](https://github.com/alihussainia/OpenDevLibrary). Look into his repository for more insight into the below cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m5pa8HAQSelr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget \"https://raw.githubusercontent.com/alihussainia/OpenDevLibrary/master/openvino_initialization_script.py\"\n",
        "!python openvino_initialization_script.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcET4FGcd27T",
        "colab_type": "text"
      },
      "source": [
        "## **Initialize the OpenVINO environment and download related files**\n",
        "The bash script will\n",
        "- Download pre-trained Intel Models\n",
        "- Create configuration file (needed for making inference on speech)\n",
        "- Required pre-requisites for LibriSpeech Model (graph file, etc)\n",
        "- Test **Online** and **Offline** demos to validate if all pre-requisites were installed properly. \n",
        "\n",
        "**Note: If working on Google Collab, the online demo may not work and hence the execution may seem to get entrapped into a never ending cycle. To avoid this you must replace the \"demo_speech_recognition.sh\" file with a custom one.**\n",
        "\n",
        "So, we tackle this issue by un-commenting the call below. This will replace the original file with a modified one, maintained in the repository.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JUKqLJ1PdHiW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!wget -P \"/content/\" \"https://raw.githubusercontent.com/PrashantDandriyal/Speech-Censor-Bot/master/Demo_Backup_Files/demo_speech_recognition.sh\"\n",
        "#!rm \"/opt/intel/openvino_2020.1.023/deployment_tools/demo/demo_speech_recognition.sh\" \n",
        "#!cp -f \"/content/demo_speech_recognition.sh\" \"/opt/intel/openvino_2020.1.023/deployment_tools/demo/demo_speech_recognition.sh\" "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M2JX3Z0vYAey",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!bash /opt/intel/openvino_2020.1.023/deployment_tools/demo/demo_speech_recognition.sh\n",
        "# Note: Simply running the bash script using \"!\" gives permission issues. The issue is being tackled."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wth8Takp4gHe",
        "colab_type": "text"
      },
      "source": [
        "## **Running Offline DEMO**\n",
        "*(To use it for custom WAV file, edit the \"run_demo.sh\" file and add the path to your file)*\n",
        "\n",
        "The output generated by the *run_demo.sh* is similar to :\n",
        "\n",
        "\n",
        ">[ INFO ] Using feature transformation /root/openvino_models/ir/intel/lspeech_s5_ext/FP32/lspeech_s5_ext.feature_transform        \n",
        "[ INFO ] InferenceEngine API ver. 2.1 (build: 37988)        \n",
        "[ INFO ] Device info:        \n",
        "[ INFO ] \tCPU: MKLDNNPlugin ver. 2.1        \n",
        "[ INFO ] Batch size: 8        \n",
        "[ INFO ] Model loading time: 49.93 ms        \n",
        "Recognition result:        \n",
        "**HOW ARE YOU DOING**\n",
        "\n",
        "We extract this output (in a naive way) by simply asking *sed* method to filter the console output as we wish to use only the text generated from the speech.\n",
        "\n",
        "Next, we save this output to a txt file for the next step: forced alignment.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m3ZTBcwZ0DON",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "# TODO:\n",
        "#Add the path to your WAV file\n",
        "!wget \"https://raw.githubusercontent.com/PrashantDandriyal/Speech-Censor-Bot/master/negan.wav\"\n",
        "wav_path = \"/content/negan.wav\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dsR0jFzXQ55V",
        "colab_type": "text"
      },
      "source": [
        "##**Preprocessing audio file**\n",
        "As per the OpenVINO v2020.1 docs [here](https://docs.openvinotoolkit.org/latest/_inference_engine_samples_speech_libs_and_demos_Offline_speech_recognition_demo.html), WAV file needs to be in following format: RIFF WAVE PCM 16bit, 16kHz, 1 channel i.e.,\n",
        "\n",
        ">Sample size : 16bit    \n",
        "Sampling Rate : 16kHz    \n",
        "Number of channels : 1        \n",
        "\n",
        "We preprocess audio and convert it if needed and replace the old file with new."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XdRhp_A2JWWV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocess(org_aud_path):\n",
        "  tx = wave.open(org_aud_path, 'r')\n",
        "  print (\"Initial Parameters:\")\n",
        "  !sox --i \"$org_aud_path\"\n",
        "  if(tx.getnchannels() > 1):\n",
        "    #Convert stereo to mono\n",
        "    #and replace the original with new\n",
        "    !sox \"$org_aud_path\" processed.wav channels 1\n",
        "    !rm -r \"$org_aud_path\"\n",
        "    !mv \"processed.wav\" \"$org_aud_path\"\n",
        "    print(\"Converted Stereo to Mono\")\n",
        "\n",
        "  if(tx.getframerate() != 16000):\n",
        "    #Downsample (if > 16k) and Upsample (if < 16k)\n",
        "    #and replace the original with new\n",
        "    !sox \"$org_aud_path\" processed.wav rate 16000\n",
        "    !rm -r \"$org_aud_path\"\n",
        "    !mv \"processed.wav\" \"$org_aud_path\"\n",
        "    print(\"Changed sample rate to 16k\")\n",
        "\n",
        "    print(\"Processed file into the same path with name 'processed.wav' \")\n",
        "\n",
        "preprocess(wav_path)\n",
        "print(\"Update file parameters\")\n",
        "!sox --i \"$wav_path\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mdibAT1W4HIA",
        "colab_type": "text"
      },
      "source": [
        "The demo uses the \"how_are_you_doing.wav\" audio file stored in the location         \n",
        "```/opt/intel/openvino/deployment_tools/demo/how_are_you_doing.wav```\n",
        "This file is fed to the inference engine using the bash file ```run_demo.sh```. Instead of editing another bash file or creating a new one, we rename our WAV file to **how_are_you_doing.wav`** and replace the original file with ours."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "btGtRO5KdOh4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd \"/content/\"\n",
        "\n",
        "#Rename file here OR edit the bash file\n",
        "!mv \"$wav_path\" \"how_are_you_doing.wav\"\n",
        "\n",
        "#Replace the file for test on custom file by removing it first \n",
        "!rm -r \"/opt/intel/openvino/deployment_tools/demo/how_are_you_doing.wav\"\n",
        "!cp \"/content/how_are_you_doing.wav\" \"/opt/intel/openvino/deployment_tools/demo/\"\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y7g3KAVYpEWU",
        "colab_type": "text"
      },
      "source": [
        "##**Perform Inference**\n",
        "The OpenVINO dependencies have successfully been installed and the environment has also been initialized. Its time to make the inference ! Run the cell to make inference. As the shell script echoes the result onto the terminal, we use ```sed``` piping to publish our results onto a text file. Another instance of the same command but without this pipe is run, to provide status of the inference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yHXnrFW-3a2H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!/opt/intel/openvino/data_processing/audio/speech_recognition/demos/offline_speech_recognition_demo/run_demo.sh \n",
        "#Running again to save the output\n",
        "!/opt/intel/openvino/data_processing/audio/speech_recognition/demos/offline_speech_recognition_demo/run_demo.sh | sed '1,/Recognition result/d' > /content/out_text.txt \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e24_xOZvpjaz",
        "colab_type": "text"
      },
      "source": [
        "##Important filepaths\n",
        "####WAV file path: \n",
        "**\"/opt/intel/openvino/deployment_tools/demo/how_are_you_doing.wav\"**\n",
        "\n",
        "####Configuration file path: \n",
        "**\"/root/openvino_models/ir/intel/lspeech_s5_ext/FP32/speech_lib.cfg\"**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3MIlKEWWeJxH",
        "colab_type": "text"
      },
      "source": [
        "##**Use *gentle* to obtain syncmap**\n",
        "[Gentle](httpshttps://github.com/lowerquality/gentle://) is a forced aligner built on KALDI that automatically generates a synchronization map between a list of text fragments and an audio file containing the narration of the text. In computer science this task is known as forced alignment.\n",
        "\n",
        "Due to some difficulties, we will be manually obtaining the syncmap files (a json + csv file) from the [site](https://lowerquality.com/gentle/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rgQoh0nwQy8j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#TODO: Add path to your json file\n",
        "json_path = \"/content/align.json\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6R1-d5TNP2Ud",
        "colab_type": "text"
      },
      "source": [
        "##**Time to censor** \n",
        "\n",
        "\n",
        "1.   Get the list of profanity words from [here](https://raw.githubusercontent.com/PrashantDandriyal/Google-profanity-words/master/list.txt)\n",
        "2.   Detect any such word and mute the uttrance using *ffmpeg*\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rahan50yQ-ma",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/\n",
        "!wget \"https://raw.githubusercontent.com/PrashantDandriyal/Google-profanity-words/master/list.txt\"\n",
        "profane_text = \"/content/list.txt\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ou4FwO8AYs6n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Make a list out of all the words in the profane_text txt file. \n",
        "#This is to avoid repetitive file searching \n",
        "with open(profane_text) as f:\n",
        "  cuss_list = [i.strip() for i in f]\n",
        "f.close()\n",
        "\n",
        "#TODO: Add any such word that was supposed to be detected as a profanic word\n",
        "#The word \"F*Ck\" has been recognised as \"For\", so censoring it for the WAV file \"negan.wav\" \n",
        "cuss_list.append(\"for\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ezvrN9BdZUwW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Parsing the json content to a easily-accesible dictionary\n",
        "import json\n",
        "import pandas as pd\n",
        "\n",
        "with open(json_path, 'r') as f:\n",
        "    handle = json.load(f)\n",
        "\n",
        "df = pd.DataFrame(columns=['word', 'start', 'end'])\n",
        "rows_list = []\n",
        "\n",
        "for i in handle['words']:\n",
        "  #print(i)\n",
        "  dict1 = {}\n",
        "  if((i['word']).lower() in cuss_list):\n",
        "    try:\n",
        "      dict1.update({\"word\":i['word'], \n",
        "                \"start\":i['start'],\n",
        "                \"end\":i['end']})   \n",
        "      rows_list.append(dict1) \n",
        "    except KeyError:    #Sometimes the word is not properly detected and the entry is \"'case': 'not-found-in-audio'\"\n",
        "      pass\n",
        "\n",
        "df = pd.DataFrame(rows_list)\n",
        "print(df.head())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EotPSxYpUkpg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Traverse the entire transcript(text converted from speech) and look for any of such cuss word\n",
        "#If any such word occurs, save its start and stop time (timestamp) t\n",
        "test_file = open(text_path,\"r\")\n",
        "\n",
        "dff = pd.DataFrame(columns=['swear', 'start', 'end'])\n",
        "swear_dict_list = []\n",
        "\n",
        "with open(text_path, 'r') as f:\n",
        "    for line in f:  \n",
        "      for word in line.split():\n",
        "        if(word.upper() in cuss_list):    #Convert the word to Upper case and seach int cuss word list\n",
        "          d = {}\n",
        "          d.update({\"swear\":word, \n",
        "                \"start\":df.loc[word].start,\n",
        "                \"end\":df.loc[word].end})\n",
        "          swear_dict_list.append(d)\n",
        "dff = pd.DataFrame(swear_dict_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n8OtW0_2ic38",
        "colab_type": "text"
      },
      "source": [
        "As we have obtained the durations of all the profane words in the speech, we will now suppress them by muting/fading the slice of audio."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1TUWKqSbglav",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "wav_path = \"/opt/intel/openvino/deployment_tools/demo/how_are_you_doing.wav\"\n",
        "#Format of SOX command to fade in. Refer to(https://stackoverflow.com/questions/20127095/using-sox-to-change-the-volume-level-of-a-range-of-time-in-an-audio-file)\n",
        "'''\n",
        "fade [type] fade-in-length [stop-position(=) [fade-out-length]]\n",
        "\n",
        "sox -m\n",
        "    -t wav \"|sox -V1 inputfile.wav -t wav - fade t 0 2.2 0.4\" \n",
        "    -t wav \"|sox -V1 inputfile.wav -t wav - trim 1.8 fade t 0.4 3.4 0.4 gain -6 pad 1.8\"\n",
        "    -t wav \"|sox -V1 inputfile.wav -t wav - trim 4.8 fade t 0.4 0 0 pad 4.8\"\n",
        "    outputfile.wav gain 9.542\n",
        "'''\n",
        "for i in df.index: \n",
        "  start = df['start'][i]\n",
        "  end = df['end'][i]\n",
        "  duration = end-start\n",
        "  #The duration of transition period when a fades in and fades out\n",
        "  trans = 0.3\n",
        "  fade_start = start-(trans/2)\n",
        "  act_fade_start = start+(trans/2)\n",
        "  #fade_start -> act_fade_start -> fade_end -> act_fade_end\n",
        "  fade_end = end-(trans/2)\n",
        "  act_fade_end = end+(trans/2)\n",
        "\n",
        "  fade_duration = duration + trans\n",
        "  #Note: Don't forget to add the $ before variable names\n",
        "  print(\"For \",start,\", \", end)\n",
        "  !sox -m -t wav \"|sox -V1 $wav_path -t wav - fade t 0 $act_fade_start $trans\" -t wav \"|sox -V1 $wav_path -t wav - trim $fade_start fade t $trans $fade_duration $trans gain -40 pad $fade_start\" -t wav \"|sox -V1 $wav_path -t wav - trim $fade_end fade t $trans 0 0 pad $fade_end\" outputfile.wav gain 9.542\n",
        "  #Replacing the input file with the outputfile\n",
        "  !rm -r \"$wav_path\"\n",
        "  !cp outputfile.wav \"$wav_path\"\n",
        "  \n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yi50k-7Vs2dA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Copy the final censored audio file to working directory\n",
        "!cp \"$wav_path\" \"/content/final_audio.wav\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "txJdq6eL1MmD",
        "colab_type": "text"
      },
      "source": [
        "               \n",
        "\n",
        "\n",
        "        \n",
        "---\n",
        "The above section marks an end to the audio censoring. To generated video outputs, get the link to the video file and proceed below.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xec5EAF70nwi",
        "colab_type": "text"
      },
      "source": [
        "#**Section 2 : Only for Video**\n",
        "Replace the audio of the video with the censored audio by using ffmpeg"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "emjTmAW60vVm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### TODO : Get the video file\n",
        "!wget -P \"/content/\" \"https://raw.githubusercontent.com/PrashantDandriyal/Speech-Censor-Bot/master/DocsResources/negan_clip_original.mp4\"\n",
        "vid_path = \"/content/negan_clip_original.mp4\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ooPj8YVf1yRX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ffmpeg -i \"$vid_path\" -i \"$wav_path\" -c:v copy -map 0:v:0 -map 1:a:0 censored_vid.mp4\n",
        "#The output is generated in the working directory with the name \"censored_vid.mp4\""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}