{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6LWsNd3_M3MP"
   },
   "source": [
    "# Mozilla TTS on CPU Real-Time Speech Synthesis "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FAqrSIWgLyP0"
   },
   "source": [
    "We use Tacotron2 and MultiBand-Melgan models and Baker dataset (chinese mandarin).\n",
    "\n",
    "Tacotron2 is trained using [Double Decoder Consistency](https://erogol.com/solving-attention-problems-of-tts-models-with-double-decoder-consistency/) (DDC) only for 126K steps (3 days) with a single GPU.\n",
    "\n",
    "MultiBand-Melgan is trained 1.45M steps with real spectrograms.\n",
    "\n",
    "Note that both model performances can be improved with more training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ku-dA4DKoeXk"
   },
   "source": [
    "### Download Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zlgi8fPdpRF0"
   },
   "source": [
    "### Define TTS function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZksegYQepkFg"
   },
   "source": [
    "### Load Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oVa0kOamprgj"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "import IPython\n",
    "\n",
    "from TTS.utils.synthesizer import Synthesizer\n",
    "from TTS.utils.manage import ModelManager\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EY-sHVO8IFSH"
   },
   "outputs": [],
   "source": [
    "# runtime settings\n",
    "use_cuda = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tts and vocoder name\n",
    "TTS_NAME = \"tts_models/zh-CN/baker/tacotron2-DDC-GST\"\n",
    "VOCODER_NAME = \"vocoder_models/en/ljspeech/multiband-melgan\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "manager = ModelManager(\"../TTS/.models.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > tts_models/zh-CN/baker/tacotron2-DDC-GST is already downloaded.\n",
      " > vocoder_models/en/ljspeech/multiband-melgan is already downloaded.\n"
     ]
    }
   ],
   "source": [
    "tts_checkpoint_file, tts_config_file, tts_json_dict = manager.download_model(TTS_NAME)\n",
    "vocoder_checkpoint_file, vocoder_config_file, vocoder_json_dict = manager.download_model(VOCODER_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Using model: tacotron2\n",
      " > Generator Model: multiband_melgan_generator\n"
     ]
    }
   ],
   "source": [
    "synthesizer = Synthesizer(tts_checkpoint_file, tts_config_file, vocoder_checkpoint_file, vocoder_config_file, use_cuda)\n",
    "sample_rate = synthesizer.tts_config.audio[\"sample_rate\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ws_YkPKsLgo-"
   },
   "source": [
    "## Run Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here some test sentences for you to play with :\n",
    "sentences= [\"我从来不会说很标准的中文。\",\n",
    "\"我喜欢听人工智能的博客。\",\n",
    "\"我来自一个法国郊区的地方。\",\n",
    "\"不比不知道，一比吓一跳！\",\n",
    "\"台湾是一个真的很好玩的地方！\",\n",
    "\"干一行，行一行，行行都行。\",\n",
    "\"我要盖被子，好尴尬！\",]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我从来不会说很标准的中文。']\n",
      " > Processing time: 1.6665124893188477\n",
      " > Real-time factor: 0.5583910829911347\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 175548 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 1.4052538871765137\n",
      " > Real-time factor: 0.5193391025114328\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 159164 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我来自一个法国郊区的地方。']\n",
      " > Processing time: 1.605910062789917\n",
      " > Real-time factor: 0.5785999490934259\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 163260 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['不比不知道，一比吓一跳！']\n",
      " > Processing time: 1.9105627536773682\n",
      " > Real-time factor: 0.6607262973429417\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 170088 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['台湾是一个真的很好玩的地方！']\n",
      " > Processing time: 1.3081049919128418\n",
      " > Real-time factor: 0.4218891158389621\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 182376 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['干一行，行一行，行行都行。']\n",
      " > Processing time: 2.0958540439605713\n",
      " > Real-time factor: 0.6709288860239634\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 183740 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我要盖被子，好尴尬！']\n",
      " > Processing time: 1.5188167095184326\n",
      " > Real-time factor: 0.6257456734843319\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 142780 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for sentence in sentences:\n",
    "    wav = synthesizer.tts(sentence)\n",
    "    IPython.display.display(IPython.display.Audio(wav, rate=sample_rate))  \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 2.114016056060791\n",
      " > Real-time factor: 0.643271887228699\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 193296 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# you can also play with Global Style Token (GST) by feeding a \n",
    "# ... wav_style parameter to the tts method\n",
    "\n",
    "style_wav = {\"2\": 0.2}\n",
    "\n",
    "wav = synthesizer.tts(sentences[1], style_wav=style_wav)\n",
    "IPython.display.display(IPython.display.Audio(wav, rate=sample_rate))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 1.5687272548675537\n",
      " > Real-time factor: 0.6401842606201799\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 144144 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 2.070594072341919\n",
      " > Real-time factor: 0.8067677285683367\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 150972 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 1.3769311904907227\n",
      " > Real-time factor: 0.5088718951180015\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 159164 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 2.024374485015869\n",
      " > Real-time factor: 0.6782983435843654\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 175548 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " > Text splitted to sentences.\n",
      "['我喜欢听人工智能的博客。']\n",
      " > Processing time: 2.4434399604797363\n",
      " > Real-time factor: 0.7435119663360867\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio  controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 193296 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# On this model specifically, we can observe that the GSToken \"2\" is responsible for speech speed\n",
    "# You can listen to these 5 different samples, the flow is slower and slower as the value is higher\n",
    "for value in [-0.2, -0.1, 0, 0.1, 0.2]:\n",
    "    style_wav = {\"2\": value}\n",
    "    wav = synthesizer.tts(sentences[1], style_wav=style_wav)\n",
    "    IPython.display.display(IPython.display.Audio(wav, rate=sample_rate))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "DDC-TTS_and_MultiBand-MelGAN_Example.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
