零-shot 跨語言語音轉換技術用於 TTS

摘要

本文介紹了一個零-shot 聲音轉換（Voice Transfer, VT）模組，可以無縫地整合到多語言文本轉語音（Text-to-speech, TTS）系統中，跨語言轉換個人的聲音。我們提出的 VT 模組包括一個處理參考語音的說話者編碼器、一個瓶頸層和連接到現有 TTS 層的殘差適配器。我們比較了這些組件的各種配置的性能，並報告了跨語言的平均意見分數（Mean Opinion Score, MOS）和說話者相似度。使用每位說話者一個英語參考語音，我們在九種目標語言中實現了平均聲音轉換相似度得分為73%。聲音特徵對於個人身份的構建和感知有著重要影響。由於生理或神經狀況的原因而失去自己的聲音可能導致對核心身份的深刻失落感。作為一個案例研究，我們展示了我們的方法不僅可以轉換典型語音，還可以恢復患有言語遲緩症的個人的聲音，即使只有非典型的語音樣本可用 - 對於從未有過典型語音或存儲自己聲音的人來說，這是一個有價值的工具。跨語言典型音頻樣本，以及展示對言語遲緩症患者進行聲音恢復的視頻，可在此處找到（google.github.io/tacotron/publications/zero_shot_voice_transfer）。

English

In this paper, we introduce a zero-shot Voice Transfer (VT) module that can be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to transfer an individual's voice across languages. Our proposed VT module comprises a speaker-encoder that processes reference speech, a bottleneck layer, and residual adapters, connected to preexisting TTS layers. We compare the performance of various configurations of these components and report Mean Opinion Score (MOS) and Speaker Similarity across languages. Using a single English reference speech per speaker, we achieve an average voice transfer similarity score of 73% across nine target languages. Vocal characteristics contribute significantly to the construction and perception of individual identity. The loss of one's voice, due to physical or neurological conditions, can lead to a profound sense of loss, impacting one's core identity. As a case study, we demonstrate that our approach can not only transfer typical speech but also restore the voices of individuals with dysarthria, even when only atypical speech samples are available - a valuable utility for those who have never had typical speech or banked their voice. Cross-lingual typical audio samples, plus videos demonstrating voice restoration for dysarthric speakers are available here (google.github.io/tacotron/publications/zero_shot_voice_transfer).

零-shot 跨語言語音轉換技術用於 TTS

Zero-shot Cross-lingual Voice Transfer for TTS

摘要

Summary

Support

Support