TTS Real-Time Voice Cloning (Translation)

The original author is Corentin Jemine, a master’s student at the University of Liège. This is an open-source GitHub project that mainly implements real-time voice cloning.

Abstract

Recent advances in deep learning have shown impressive results in the domain of text-to-speech. To this end, a deep neural network is usually trained using a corpus of several hours of professionally recorded speech from a single speaker. Giving a new voice to such a model is highly expensive, as it requires recording a new dataset and retraining the model. A recent research introduced a three-stage pipline that allows to clone a voice unseen during training from only a few seconds of reference speech, and without retraining the model. The authors share remarkably natural-sounding results, but provide no implementation. wo reproduce this framework and open-source the first public implementation of it. We adapt the framework with a newer vocoder model, so as to make it run in real-time.

Recently, developments in deep learning have achieved remarkable results in the field of TTS. To this end, a deep neural network is usually trained on several hours of a corpus professionally recorded by a single person. For such a model, producing a new voice is extremely expensive, because it requires recording a new dataset and retraining the model. A recent study introduced a three-stage pipeline that can clone a voice almost unseen during training from just a few seconds of reference speech, without needing to retrain the model. The authors shared strikingly natural-sounding results, but did not provide an implementation. We reproduce this framework and open-source a public implementation of it. We adapt the framework with a new vocoder model so that it can run in real time.

1 Introduction

Deep learning models have become predominant in many fields of applied machine learning. Text-to-speech(TTS)， the process of synthesizing artificial speech from a text prompt, is no exception. Deep models that would produce more natural-sounding speech than the traditional approaches begun appearing in 2016. Much of research focus has been since gathered around making these deep models more efficient，sound more natural, or training the in an end-to-end fashion. Inference has come from being hundreds of times slower than real-time on GPU(van den oord et at 2016) to possible in real-time on a mobile CPU. As for the quality of the generated speech, demonstrate near-human naturalness. Interestingly, speech naturalness is best rated with subjective metrics; and comparison with actural human speech leads to the conlusion that there might be such a thing as “speech more natural than human speech”. In fact ,some argue that the human naturalness threshold has already been crossed.

Deep learning models have come to dominate many areas of applied machine learning. Speech synthesis, the process of synthesizing artificial speech from a text prompt, is undoubtedly one of them. Compared with the traditional concatenative methods, deep models, which emerged in 2016, can produce much more natural speech. Much research has focused on making deep models more efficient, making the voice sound more natural, or training them in an end-to-end fashion. Inference has gone from being hundreds of times slower than real time on a GPU to being possible in real time on a mobile CPU. As for the quality of the generated speech, it shows near-human naturalness. Interestingly, speech naturalness is best evaluated with subjective metrics, and comparing it with actual human speech leads to the conclusion that there may be such a thing as “speech more natural than human speech”. In fact, some argue that the human naturalness threshold has already been surpassed.

Datasets of professionally recorded speech are a scare resource. Synthesizing a natural voice with a correct pronunciation, lively intonation and a minimum of background noise requires training data with the same qualities. Furthermore, data efficiency remains a core issue of deep learning. Training a common text-to-speech model such as Tacotron (Wang et al., 2017) typically requires hundreds of hours of speech. Yet the ability to generate speech with any voice is attractive for a range of applications, be they useful or merely a matter of customization. Research has led to frameworks for voice conversion and voice cloning. They differ in that voice conversion is a form of style transfer on a speech segment from a voice to another, whereas voice cloning consists in capturing the voice of a speaker to perform text-to- speech on arbitrary inputs.

Professionally recorded speech datasets are a scarce resource. Synthesizing a natural voice with correct pronunciation, lively intonation and minimal background noise requires training data of the same quality. Moreover, data efficiency remains a core problem of deep learning. Training a common text-to-speech model, such as Tacotron (Wang et al., 2017), usually requires hundreds of hours of speech. However, the ability to generate speech in any voice is attractive for a range of applications, whether they are useful or merely a matter of customization. Research has produced frameworks for voice conversion and voice cloning. They differ in that voice conversion is a form of converting a speech segment from one voice to another, whereas voice cloning captures a speaker’s voice in order to perform text-to-speech conversion on arbitrary inputs.

While the complete training of a single-speaker TTS model is technically a form of voice cloning, the interest rather lies in creating a fixed model able to incorporate newer voices with little data. The common approach is to condition a TTS model trained to generalize to new speakers on an embedding of the voice to clone (Arik et al., 2017, 2018; Jia et al., 2018). The embedding is low-dimensional and derived by a speaker encoder model that takes reference speech as input. This approach is typically more data efficient than training a separate TTS model for each speaker, in addition to being orders of magnitude faster and less computationally expensive. Interestingly, there is a large discrepancy between the duration of reference speech needed to clone a voice among the different methods, ranging from half an hour per speaker to only a few seconds. This factor is usually determining of the similarity of the generated voice with respect to the true voice of the speaker.

Although the complete training of a single-speaker TTS model is technically a form of voice cloning, the interest lies in creating a fixed model that can incorporate newer voices from very little data. The common approach is to condition a trained TTS model so that it generalizes to new speakers through an embedding of the voice to clone. The embedding is low-dimensional and is derived by a speaker encoder model that takes reference speech as input. This approach is more data efficient than training a separate TTS model for each speaker, and it is also faster and less computationally expensive. Interestingly, the duration of reference speech needed to clone a voice varies widely among the different methods, ranging from half an hour per speaker to a few seconds. This factor usually determines how similar the generated voice is to the speaker’s real voice.

Our objective is to achieve a powerful form of voice cloning. The resulting framework must be able to operate in a zero-shot setting, that is, for speakers unseen during training. It should incorporate a speaker’s voice with only a few seconds of reference speech. These desired results are shown to be fulfilled by (Jia et al., 2018).Their results are impressive1, but not backed by any public implementation. We reproduce their framework and make our implementation open-source . In addition, we integrate a model based on (Kalchbrenner et al., 2018) in the framework to make it run in real-time, i.e. to generate speech in a time shorter or equal to the duration of the produced speech.