4.2 KiB

Raw Blame History

mimic2

This is a fork of keithito/tacotron with changes specific to Mimic 2 applied.

Background

Google published a paper, Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model, where they present a neural text-to-speech model that learns to synthesize speech directly from (text, audio) pairs. However, they didn't release their source code or training data. This is an attempt to provide an open-source implementation of the model described in their paper.

The quality isn't as good as Google's demo yet, but hopefully it will get there someday :-). Pull requests are welcome!

Quick Start

Installing dependencies

using docker (recommended)

make sure you have docker installed
Build Docker

the Dockerfile comes with a gpu option or cpu option. If you want to use the GPU in docker make sure you have nvidia-docker installed

gpu: docker build -t mycroft/mimic2:gpu -f gpu.Dockerfile .

cpu: docker build -t mycroft/mimic2:cpu -f cpu.Dockerfile .
Run Docker

gpu: nvidia-docker run -it -p 3000:3000 mycroft/mimic2:gpu

cpu: docker run -it -p 3000:3000 mycroft/mimic2:cpu

manually

Install Python 3.
Install the latest version of TensorFlow for your platform. For better performance, install with GPU support if it's available. This code has been tested on tensorflow 1.8.
Install requirements:
```
pip install -r requirements.txt
```

Training

Note: you need at least 40GB of free disk space to train a model.

Download a speech dataset.

The following are supported out of the box:
- LJ Speech (Public Domain)
- Blizzard 2012 (Creative Commons Attribution Share-Alike)
- M-ailabs
You can use other datasets if you convert them to the right format. See TRAINING_DATA.md for more info.

Unpack the dataset into ~/tacotron

After unpacking, your tree should look like this for LJ Speech:

tacotron
  |- LJSpeech-1.1
      |- metadata.csv
      |- wavs

or like this for Blizzard 2012:

tacotron
  |- Blizzard2012
      |- ATrampAbroad
      |   |- sentence_index.txt
      |   |- lab
      |   |- wav
      |- TheManThatCorruptedHadleyburg
          |- sentence_index.txt
          |- lab
          |- wav

For M-AILABS follow the directory structure from here

Preprocess the data
```
python3 preprocess.py --dataset ljspeech
```
- other datasets can be used i.e. --dataset blizzard for Blizzard data
- for the mailabs dataset, do preprocess.py --help for options. Also note that mailabs uses sample_size of 16000
Train a model
```
python3 train.py
```
Tunable hyperparameters are found in hparams.py. You can adjust these at the command line using the --hparams flag, for example --hparams="batch_size=16,outputs_per_step=2". Hyperparameters should generally be set to the same values at both training and eval time. I highly recommend setting the params in the hparams.py file to gurantee consistentcy during preprocessing, training, evaluating, and running the demo server.
Monitor with Tensorboard (optional)
```
tensorboard --logdir ~/tacotron/logs-tacotron
```
The trainer dumps audio and alignments every 1000 steps. You can find these in ~/tacotron/logs-tacotron.
Synthesize from a checkpoint
```
python3 demo_server.py --checkpoint ~/tacotron/logs-tacotron/model.ckpt-185000
```
Replace "185000" with the checkpoint number that you want to use, then open a browser to localhost:3000 and type what you want to speak. Alternately, you can run eval.py at the command line:
```
python3 eval.py --checkpoint ~/tacotron/logs-tacotron/model.ckpt-185000
```
If you set the --hparams flag when training, set the same value here.

4.2 KiB Raw Blame History