TTS 实时语音克隆(翻译)

原作者是LIEGE University master Corentin Jemine，这是一个github开源项目，主要是实现实时语音克隆。

Abstract 摘要

Recent advances in deep learning have shown impressive results in the domain of text-to-speech. To this end, a deep neural network is usually trained using a corpus of several hours of professionally recorded speech from a single speaker. Giving a new voice to such a model is highly expensive, as it requires recording a new dataset and retraining the model. A recent research introduced a three-stage pipline that allows to clone a voice unseen during training from only a few seconds of reference speech, and without retraining the model. The authors share remarkably natural-sounding results, but provide no implementation. wo reproduce this framework and open-source the first public implementation of it. We adapt the framework with a newer vocoder model, so as to make it run in real-time.

最近，深度学习的发展在TTS领域已经取得了令人瞩目的成绩。为了达到这个目的，一个深度神经网络通常使用一个人专业录制好几个小时的语料来进行训练。对于那么一个模型来说，得出一个新的声音是十分昂贵的，因为它要求录制一个新的数据集和重新训练模型。一项最近的研究介绍了一种三级的流水线，这种三级流水线能够从几秒的参考语音中训练几乎看不见就能能复制一个声音，并且不需要重新训练模型。作者分享了惊人的自然的声音的结果，但是，并没有提供实现方式。我们再现了这个框架，并且开源了它的公用的实现方式。我们用一个新的声码器模型改编了这个框架，以便是他能够实时运行。

1 Introduction 介绍

Deep learning models have become predominant in many fields of applied machine learning. Text-to-speech(TTS)， the process of synthesizing artificial speech from a text prompt, is no exception. Deep models that would produce more natural-sounding speech than the traditional approaches begun appearing in 2016. Much of research focus has been since gathered around making these deep models more efficient，sound more natural, or training the in an end-to-end fashion. Inference has come from being hundreds of times slower than real-time on GPU(van den oord et at 2016) to possible in real-time on a mobile CPU. As for the quality of the generated speech, demonstrate near-human naturalness. Interestingly, speech naturalness is best rated with subjective metrics; and comparison with actural human speech leads to the conlusion that there might be such a thing as “speech more natural than human speech”. In fact ,some argue that the human naturalness threshold has already been crossed.

深度学习模型已经变得在许多机器学习应用领域中站主导地位。语音合成，从一段文本的提示信息合成人工的语音的过程，毫无疑问就是深度学习模型的一种。深度模型相对于2016年诞生的传统拼接方法将能产生出更加更加自然的语音。许多研究焦点在于聚集起来是深度模型更加高效，声音更加自然，或者以一种端到端的方式训练。从在GPU上比实时慢数百倍推断，在移动CPU上实时是可能的。在生成的语音质量上，表现出接近人类的自然。有趣的是，语言自然度最好用主观指标来评价。并与人类实际语言进行比较，得出可能存在“比人类语言更自然的语言”的结论。事实上，一些人认为，人类的自然性门槛已经被超越了。

Datasets of professionally recorded speech are a scare resource. Synthesizing a natural voice with a correct pronunciation, lively intonation and a minimum of background noise requires training data with the same qualities. Furthermore, data efficiency remains a core issue of deep learning. Training a common text-to-speech model such as Tacotron (Wang et al., 2017) typically requires hundreds of hours of speech. Yet the ability to generate speech with any voice is attractive for a range of applications, be they useful or merely a matter of customization. Research has led to frameworks for voice conversion and voice cloning. They differ in that voice conversion is a form of style transfer on a speech segment from a voice to another, whereas voice cloning consists in capturing the voice of a speaker to perform text-to- speech on arbitrary inputs.

专业录制的语音数据集是一种稀缺资源。要用正确的发音、生动的语调和最小的背景噪音来合成一个自然的声音，需要具有相同质量的训练数据。此外，数据效率仍然是深度学习的核心问题。训练一个常见的文本到语音模型，如Tacotron (Wang et al.， 2017)，通常需要数百小时的语音。然而，用任何声音生成语音的能力对一系列应用程序都很有吸引力，不管它们是有用的，还是仅仅是定制的问题。研究已经产生了语音转换和语音克隆的框架。它们的不同之处在于语音转换是在语音片段上从一个声音转换到另一个声音的一种形式，而语音克隆则是捕获说话者的声音，在任意输入上执行文本到语音的转换。

While the complete training of a single-speaker TTS model is technically a form of voice cloning, the interest rather lies in creating a fixed model able to incorporate newer voices with little data. The common approach is to condition a TTS model trained to generalize to new speakers on an embedding of the voice to clone (Arik et al., 2017, 2018; Jia et al., 2018). The embedding is low-dimensional and derived by a speaker encoder model that takes reference speech as input. This approach is typically more data efficient than training a separate TTS model for each speaker, in addition to being orders of magnitude faster and less computationally expensive. Interestingly, there is a large discrepancy between the duration of reference speech needed to clone a voice among the different methods, ranging from half an hour per speaker to only a few seconds. This factor is usually determining of the similarity of the generated voice with respect to the true voice of the speaker.

虽然单扬声器TTS模型的完整训练从技术上讲是语音克隆的一种形式，但其兴趣在于创建一个固定的模型，能够在很少的数据中包含较新的声音。常用的方法是对经过训练的TTS模型进行条件设置，以便通过嵌入的声音克隆来泛化到新的说话者。嵌入是低维的，并由以引用语音作为输入的扬声器编码器模型派生。这种方法比为每个演讲者训练一个单独的TTS模型更具有数据效率，而且速度更快，计算成本更低。有趣的是，在不同的方法中，克隆一个声音所需的参考讲话时间相差很大，从每个发言者半小时到几秒钟不等。这个因素通常决定产生的声音相对于说话者的真实声音的相似性。

Our objective is to achieve a powerful form of voice cloning. The resulting framework must be able to operate in a zero-shot setting, that is, for speakers unseen during training. It should incorporate a speaker’s voice with only a few seconds of reference speech. These desired results are shown to be fulfilled by (Jia et al., 2018).Their results are impressive1, but not backed by any public implementation. We reproduce their framework and make our implementation open-source . In addition, we integrate a model based on (Kalchbrenner et al., 2018) in the framework to make it run in real-time, i.e. to generate speech in a time shorter or equal to the duration of the produced speech.