DeepVoice3_multi_speaker_TTS_en_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Za124iWvdMsZ"
   },
   "source": [
    "# DeepVoice3: Multi-speaker text-to-speech demo\n",
    "\n",
    "In this notebook, you can try DeepVoice3-based multi-speaker text-to-speech (en) using a model trained on [VCTK dataset](http://homepages.inf.ed.ac.uk/jyamagis/page3/page58/page58.html). The notebook is supposed to be executed on [Google colab](https://colab.research.google.com) so you don't have to setup your machines locally.\n",
    "\n",
    "**Estimated time to complete**: 5 miniutes.\n",
    "\n",
    "- Code: https://github.com/r9y9/deepvoice3_pytorch\n",
    "- Audio samples: https://r9y9.github.io/deepvoice3_pytorch/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ml6wOhwqhGiI"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QjindPTItq75"
   },
   "source": [
    "### Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kemMMs6pg9Rv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'deepvoice3_pytorch'...\n",
      "remote: Enumerating objects: 1091, done.\u001b[K\n",
      "remote: Total 1091 (delta 0), reused 0 (delta 0), pack-reused 1091\u001b[K\n",
      "Receiving objects: 100% (1091/1091), 6.91 MiB | 412.00 KiB/s, done.\n",
      "Resolving deltas: 100% (606/606), done.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from os.path import exists, join, expanduser\n",
    "\n",
    "# Clone\n",
    "name = \"deepvoice3_pytorch\"\n",
    "if not exists(name):\n",
    "  ! git clone https://github.com/r9y9/$name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ntBxf7b6DCqT"
   },
   "outputs": [],
   "source": [
    "# Change working directory to the project dir \n",
    "os.chdir(name)\n",
    "\n",
    "# Use pytorch v0.3.1\n",
    "!pip install -q torch==0.3.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X6VFmDe-ideo"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "! pip install -q librosa nltk\n",
    "\n",
    "import torch\n",
    "import numpy as np\n",
    "import librosa\n",
    "import librosa.display\n",
    "import IPython\n",
    "from IPython.display import Audio\n",
    "# need this for English text processing frontend\n",
    "import nltk\n",
    "! python -m nltk.downloader cmudict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_l1Gd2SStt0E"
   },
   "source": [
    "### Download a pre-trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "42Zwjr4UjNn_"
   },
   "outputs": [],
   "source": [
    "checkpoint_path = \"20171222_deepvoice3_vctk108_checkpoint_step000300000.pth\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "45Wrp8INj6Xu"
   },
   "outputs": [],
   "source": [
    "if not exists(checkpoint_path):\n",
    "  !curl -O -L \"https://www.dropbox.com/s/uzmtzgcedyu531k/20171222_deepvoice3_vctk108_checkpoint_step000300000.pth\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pbN0Kuo43G4U"
   },
   "source": [
    "### git checkout to the working commit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Pqu6dICruu56"
   },
   "outputs": [],
   "source": [
    "# Copy preset file (json) from master\n",
    "# The preset file describes hyper parameters\n",
    "! git checkout master --quiet\n",
    "preset = \"./presets/deepvoice3_vctk.json\"\n",
    "! cp -v $preset .\n",
    "preset = \"./deepvoice3_vctk.json\"\n",
    "\n",
    "# And then git checkout to the working commit\n",
    "# This is due to the model was trained a few months ago and it's not compatible\n",
    "# with the current master. \n",
    "! git checkout 0421749 --quiet\n",
    "! pip install -q -e '.[train]'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_yJ90ESZiT_S"
   },
   "source": [
    "## Synthesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FUyhiJg03dj6"
   },
   "source": [
    "### Setup hyper parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "E9sLuYgcnbZb"
   },
   "outputs": [],
   "source": [
    "import hparams\n",
    "import json\n",
    "\n",
    "# Newly added params. Need to inject dummy values\n",
    "for dummy, v in [(\"fmin\", 0), (\"fmax\", 0), (\"rescaling\", False),\n",
    "                 (\"rescaling_max\", 0.999), \n",
    "                 (\"allow_clipping_in_normalization\", False)]:\n",
    "  if hparams.hparams.get(dummy) is None:\n",
    "    hparams.hparams.add_hparam(dummy, v)\n",
    "    \n",
    "# Load parameters from preset\n",
    "with open(preset) as f:\n",
    "  hparams.hparams.parse_json(f.read())\n",
    "\n",
    "# Tell we are using multi-speaker DeepVoice3\n",
    "hparams.hparams.builder = \"deepvoice3_multispeaker\"\n",
    "  \n",
    "# Inject frontend text processor\n",
    "import synthesis\n",
    "import train\n",
    "from deepvoice3_pytorch import frontend\n",
    "synthesis._frontend = getattr(frontend, \"en\")\n",
    "train._frontend =  getattr(frontend, \"en\")\n",
    "\n",
    "# alises\n",
    "fs = hparams.hparams.sample_rate\n",
    "hop_length = hparams.hparams.hop_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C4NOldY83wG1"
   },
   "source": [
    "### Define utility functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xRbelGLjiSfA"
   },
   "outputs": [],
   "source": [
    "def tts(model, text, p=0, speaker_id=0, fast=True, figures=True):\n",
    "  from synthesis import tts as _tts\n",
    "  waveform, alignment, spectrogram, mel = _tts(model, text, p, speaker_id, fast)\n",
    "  if figures:\n",
    "      visualize(alignment, spectrogram)\n",
    "  IPython.display.display(Audio(waveform, rate=fs))\n",
    "  \n",
    "def visualize(alignment, spectrogram):\n",
    "  label_fontsize = 16\n",
    "  figure(figsize=(16,16))\n",
    "\n",
    "  subplot(2,1,1)\n",
    "  imshow(alignment.T, aspect=\"auto\", origin=\"lower\", interpolation=None)\n",
    "  xlabel(\"Decoder timestamp\", fontsize=label_fontsize)\n",
    "  ylabel(\"Encoder timestamp\", fontsize=label_fontsize)\n",
    "  colorbar()\n",
    "\n",
    "  subplot(2,1,2)\n",
    "  librosa.display.specshow(spectrogram.T, sr=fs, \n",
    "                           hop_length=hop_length, x_axis=\"time\", y_axis=\"linear\")\n",
    "  xlabel(\"Time\", fontsize=label_fontsize)\n",
    "  ylabel(\"Hz\", fontsize=label_fontsize)\n",
    "  tight_layout()\n",
    "  colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m2jmbSD430Ws"
   },
   "source": [
    "### Load the model checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lr8pgqtYhvav"
   },
   "outputs": [],
   "source": [
    "from train import build_model\n",
    "from train import restore_parts, load_checkpoint\n",
    "\n",
    "model = build_model()\n",
    "model = load_checkpoint(checkpoint_path, model, None, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DOJ3miW63ywA"
   },
   "source": [
    "### Generate speech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GR1XRy-ykbz_"
   },
   "outputs": [],
   "source": [
    "# Try your favorite senteneces:)\n",
    "text = \"Some have accepted this as a miracle without any physical explanation\"\n",
    "N = 15\n",
    "print(\"Synthesizing \\\"{}\\\" with {} different speakers\".format(text, N))\n",
    "for speaker_id in range(N):\n",
    "  print(speaker_id)\n",
    "  tts(model, text, speaker_id=speaker_id, figures=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nirMEf2J5Roy"
   },
   "outputs": [],
   "source": [
    "# With attention plot\n",
    "tts(model, text, speaker_id=0, figures=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hjQXwF1GIF7A"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ArQspYbs5Aoo"
   },
   "source": [
    "For details, please visit https://github.com/r9y9/deepvoice3_pytorch"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "DeepVoice3 multi-speaker TTS en demo.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}